Improved transfer learning using textural features conflation and dynamically fine-tuned layers

Research contributions

This study proposes an alternative method for improving the pre-trained model’s performance in two stages: selecting relevant target data points and dynamic layer selection using Kullback–Leibler divergence ( ${\mathrm{D}}_{\mathrm{K}\mathrm{L}})$. The method combines the pre-trained models’ performance improvement at the two stages giving an overall model performance.

The contributions of this study are:

• The pre-trained model on the target tasks achieves better transfer learning adaptation by selecting higher-quality data points in the target dataset through textural divergence measurements. The selected data points can be used in other model improvement tasks like augmentation. The data points selection builds better performance confidence in their application in machine learning algorithms.

• By selecting layers with higher positive weights, the pre-trained model can fine-tune much better, which improves its performance. These positive weights can give faster convergence, improving the adaptability of the target tasks.

• Pre-trained models can improve performance and user confidence in the target tasks during transfer learning processes by preparing the modelling pipeline using quality target dataset samples and adaptable-fine-tuned layers. This approach results in an effective and efficient pre-trained model selection procedure; from the previous trial and error selection of pre-trained models and fine-tuning layers.

Textural features of image data

In image processing, texture refers to an objective function representing the brightness and intensity variation of an image’s pixels (Tuceryan & Jain, 1993). Texture explains images’ smoothness, roughness, regularity and coarseness, as noted by Laleh & Shervan (2019). Texture gives the sequential illumination patterns of the pixels in an image and the image grey tones in the pixel’s neighbourhood (Dixit & Hegde, 2013). Textural features in an image can be classified into three: low-level, mid-level and high-level. The classification is based on pixel levels, image descriptors and image data representation (Bolón-Canedo & Remeseiro, 2019). The features in an image are analysed to understand the spatial arrangement of the pixels’ grey tones after their extraction. The extraction process can be categorised according to transformation, structure, model, graphical, statistical, entropy and learning views. The low-level image features are heavily used in image classification, utilising colour, texture and shape attributes. These attributes are passed through filters, quantified using statistical descriptors such as entropy and correlation, and ranked through relevance indices.

The two commonly used methods in textural analysis are the grey-level co-occurrence matrix (GLCM) and the local binary pattern (LBP) (Ershad, 2012). The GLCM was introduced by Haralick, Shanmugam & Dinstein (1973) to represent the pixels’ brightness levels using a matrix that combines the grey levels intervals, direction and amplitude change. The GLCM descriptor has 14 features, with interval distance and orientation being the most important (Andrearczyk & Whelan, 2016). This study evaluates three features: correlation, homogeneity and energy. The LBP uses the local textural patterns in an image and compares the pixel’s neighbouring grey levels. The comparison of the neighbours uses representative binary numbers described using histograms. The LBP is a robust textural descriptor used in edge detection and textural description (Zeebaree et al., 2020).

Textural features conflation of image data

Conflation refers to a merge of two or more probability distributions. The concept was introduced by Hill (2011). Given probability $P_1$,…, $P_n$, the conflation(&) of 1 to $n$ is expressed in Eq. (1) below, with Eqs. (2) and (3) giving the conflation for discrete distributions. Equation (4) shows the conflation of continuous distributions for textural features in the source and target domain data points.

(1) $$Q=\mathrm{\&}(P_1,\dots ,P_n)$$where $Q$ is the merged probability distribution.

(2) $$\mathrm{\&}(P_1,\dots ,P_n)={\displaystyle \frac{f_1\left(x\right)f_2\left(x\right),\dots ,f_n\left(x\right)}{{\sum }_y{f}_1\left(y\right)f_2\left(y\right),\dots ,f_n\left(y\right)}}$$where $f_1,\dots ,f_n$ refers to probability density functions of the textural features. This equation can be rewritten as;

(3) $$\mathrm{\&}(S{P}_1,\dots ,S{P}_n)={\displaystyle \frac{S{f}_1\left(x\right)S{f}_2\left(x\right),\dots ,S{f}_n\left(x\right)}{{\sum }_{y}S{f}_1\left(y\right)S{f}_2\left(y\right),\dots ,S{f}_n\left(y\right)}}$$

(4) $$\mathrm{\&}(T{P}_1,\dots ,T{P}_n)={\displaystyle \frac{T{f}_1\left(x\right)T{f}_2\left(x\right),\dots ,T{f}_n\left(x\right)}{{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}T{f}_1\left(y\right)T{f}_2\left(y\right),\dots ,T{f}_n\left(y\right)}}$$where $S{P}_1$,…, $S{P}_n$ refers to the probability distributions of the source domain samples. The probability distributions of the target domain samples can be represented using Eq. (4) by substituting S with T.

The conflation of features removes redundant features resulting in a balanced probability distribution (Hill, 2011).

Model layer fine-tuning in transfer learning

Fine-tuning is one of the methods of transfer learning. The process involves selecting training layers and freezing the weights of the pre-trained model on a target task. In most cases, the first layers of a model are chosen due to their ability to extract features as opposed to the last layers, which are mainly used for classification purposes (Coskun et al., 2017). Fine-tuning has been a manual process involving selecting the first or the initial layers (in most cases, the last three layers of the network), as reported in the literature (Deniz et al., 2018; Fan, Lee & Lee, 2021). Vrbančič & Podgorelec (2020) noted that models have specific architectures that sometimes make their layer selection inefficient and fine-tuning a trial-and-error process.

Proposed study’s approach

The proposed method comprises two parts: the selection of quality dataset samples and the dynamic selection of the pre-trained model’s fine-tunable layers, as shown in Fig. 1.

As shown in Fig. 1, the target and source domain data samples are compared based on their textural features resulting in a final target dataset as expressed in Eqs. (5) to (8). The pre-trained model layers are then selected for transfer learning to accomplish a target CNN task as shown in Eqs. (9) to (13).

Selection of quality image dataset

The selection of quality images involves extracting textural features from a target domain image, conflating its textural features’ probability distributions, and comparing the resultant distribution with the conflated distributions in the source domain images.

Definition 1. The conflation of a target image’s textural features. Given a target image $T_{i1}$, its textural features $T_{if1},\dots ,T_{ifn}$, the conflated probability distribution can be expressed in Eq. (5) as;

(5) $$\mathrm{\&}(T_{if1},\dots ,T_{ifn})={\displaystyle \frac{T_{if1}\left(x\right)T_{if2}\left(x\right),\dots ,T_{ifn}\left(x\right)}{{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{T}_{if1}\left(y\right)T_{if2}\left(y\right),\dots ,T_{ifn}\left(y\right)}}$$where $T_{if}$ represents a target image’s feature. We can proceed and equate the conflated value to $\mathrm{\&}{T}_{if}$ as expressed in Eq. (6);

(6) $$(\mathrm{\&}{T}_{if})=\mathrm{\&}(T_{if1},\dots ,T_{ifn})$$

Definition 2. The conflation of a source image’s textural features. Given a source image $S_{i1}$, its textural feature vectors $S_{if1},\dots ,S_{ifn}$ distributions can be conflated into $\mathrm{\&}{S}_{if}$ as expressed in Eq. (7) below;

(7) $$\mathrm{\&}(S_{if})=\frac{S_{if1}(x)S_{if2}(x),\dots ,S_{ifn}(x)}{{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{S}_{if1}(y)S_{if2}(y),\dots ,S_{ifn}(y)}$$where $S_{if1}$ represents a probability distribution of the first feature in a selected source image feature.

Once the source conflated distributions are identified, a vector of the source image conflated probability distribution is used to check its divergence from the target image conflated distribution, as shown in Eq. (8).

(8) $${\mathrm{D}}_{\mathrm{K}\mathrm{L}}\left(\left(\mathrm{\&}{T}_{if}\right)\parallel \left(S_{if}\right)\right)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}\left(\mathrm{\&}{T}_{if}\right)\left(x\right)log{\displaystyle \frac{\left(\mathrm{\&}{T}_{if}\right)\left(x\right)}{\mathrm{\&}\left(\mathrm{\&}{S}_{if}\right)\left(x\right)}dx}$$where ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ represents Kullback–Leibler divergence.

Finally, we select images whose ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ is lower than the average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ of the source image distributions.

Selection of fine-tunable layers in pre-trained models

The fine-tunable layers selection involves identifying convolutional layers in a pre-trained model, their positive and negative weights and selecting fine-tunable layers by utilising the weights’ divergence.

Definition 3. Identification of a convolutional layer. Given a pre-trained model $M$, a convolutional layer $C_l$ is expressed as follows;

(9) $$C_l=\{\begin{array}{c}{M}_l,if{M}_{ln}=“conv”\\ otherwise\end{array}$$where $M_l$ is the model’s layer, $M_{ln}$ represents the model’s layer name which identifies a convolutional layer if its name contains the keyword “conv”; otherwise, the layer is skipped.

Definition 4. Identification of positive and negative weights. Given a convolutional layer $C_l$, a weighting filter $C_{lw}$ is a vector of $n\times n$ kernel size. Given two filters, $x$ and $y$, we can reshape the tensors from $C_{lw}^x\in {\mathbb{R}}^{n\ast n}$ and $C_{lw}^y\in {\mathbb{R}}^{n\ast n}$ into $C_{lw}^{x^{\mathrm{\prime }}}$ and $C_{lw}^{y^{\mathrm{\prime }}}$, respectively. We can express the positive and negative weight units as $C_{lw}^{x^{\mathrm{\prime }}+ve}$ and $C_{lw}^{x^{\mathrm{\prime }}-ve}$. Therefore each weight filter becomes a single tensor of positive and negative weights, as expressed in Eq. (10).

(10) $$C_{lw}^{x^{\mathrm{\prime }}}=Tensor\left(C_{lw1}^{x^{\mathrm{\prime }}+ve},\dots ,C_{lwn}^{x^{\mathrm{\prime }}-ve}\right)$$where $C_{lw1}^{x^{\mathrm{\prime }}+ve}$ represents the first positive weight unit of filter $x$ for the convolutional layer $C_l$.

Definition 5. Divergence measure between layers. Given two single-dimensional layers $C_{lw}^{x^{\mathrm{\prime }}}$ and $C_{lw}^{y^{\mathrm{\prime }}}$, we can calculate their differences by converting the vectors into probability distributions based on their positive or negative weights vectors. Utilising $D_{KL}$, the divergence of the positive weight vectors is expressed in Eq. (11).

(11) $$D_{KL}\left(p(C_{lw}^{x^{\mathrm{\prime }}+ve})\parallel p(C_{lw}^{y^{\mathrm{\prime }}+ve})\right)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}p\left(C_{lw}^{x^{\mathrm{\prime }}+ve}\right)\left(x\right)log{\displaystyle \frac{p\left(C_{lw}^{x^{\mathrm{\prime }}+ve}\right)\left(x\right)}{p(C_{lw}^{y^{\mathrm{\prime }}+ve})\left(x\right)}dx}$$where $p$ refers to a probability distribution.

We can simplify this further by substituting $p(C_{lw}^{x^{\mathrm{\prime }}+ve})$ with $p\left(x\right)$ and $p(C_{lw}^{y^{\mathrm{\prime }}+ve})$ with $p\left(y\right)$, as shown in Eq. (12).

(12) $$D_{KL}\left(p\left(x\right)\parallel p\left(y\right)\right)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}p\left(x\right)\left(x\right)log{\displaystyle \frac{p\left(x\right)\left(x\right)}{p\left(y\right)\left(x\right)}dx}$$

The layers with the lower divergence measures are then selected for use in the fine-tuning process.

Algorithm

The following steps summarise the proposed approach:

1. Select an image $T_{i1}$ from a target dataset.

2. Extract $T_{i1}$ textural features using a pre-trained model $M$, giving vectors of features $T_{if1},\dots ,T_{ifn}$.

3. Convert the vectors of features into probability distributions $p(T_{if1}),\dots ,p(T_{ifn})$

4. Conflate the probability distributions $(\mathrm{\&}{T}_{if})$ in (iii) into one probability distribution $p(T_{if})$.

5. Repeat the procedure for the source images.

6. Calculate the average Kullback–Leibler divergence ( ${\mathrm{D}}_{\mathrm{K}\mathrm{L}})$ of the source images.

7. Compare the ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ of $T_{i1},\dots ,T_{in}$ to the average of the source selecting data points whose $D_{KL}$ is lower. These images form the final target dataset.

8. Identify the convolution layers in pre-trained model $M$ using the keyword “conv”.

9. Identify the positive and negative weights in each convolutional layer $C_l$.

10. Create probability distributions of the positive and negative weight filter units.

11. Calculate the $D_{KL}$ between positive or negative probability distributions between any two layers.

12. Repeat the procedure in (xi).

13. Select the layers with the lowest $D_{KL}$ as candidates for fine-tuning.

14. Perform transfer learning on the target task utilising the target dataset in step (vii) and the selected layers in step (xiii).

Divergence measures

Divergence is a measure of the difference between any two probability distributions. This proposed approach uses $D_{KL}$ and compares it to four other divergence measures: Jensen–Shannon, Bhattacharya, Hellinger and Wasserstein.

Kullback–Leibler: Theodoridis (2015) presents it as a measure between two probability distributions, as expressed in Eq. (12). The divergence gives a zero measure when the two distributions are equal. Adding the zero measure, we can rewrite Eqs. (12) to (13).

(13) $$D_{KL}\left(p\left(x\right)\parallel p\left(y\right)\right)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}p\left(x\right)\left(x\right)log{\displaystyle \frac{p\left(x\right)\left(x\right)}{p\left(y\right)\left(x\right)}dx}$$where $D_{KL}\left(p\left(x\right)\parallel p\left(y\right)\right)$ if and only if $x=y$.

Jensen–Shannon: A symmetrised version of the $D_{KL}$ that measures the distance between two probability distributions. Unlike the $D_{KL}$, it has high computational costs in search operations (Nielsen & Nock, 2021).

Hellinger distance: A measure of the difference between any two probability distributions in a shared space. Also known as Jeffrey’s distance but has a higher computational complexity than the $D_{KL}$, as noted by Greegar & Manohar (2015).

Wasserstein distance: A measure of the difference between any two probability distributions. It uses the concept of moving an amount of earth and the distance involved. The divergence has been used in optimal transportation theory, as Villani (2003) noted.

Bhattacharya: A measure between two probability distributions that gives the cosine angle to interpret the overlapping angle between them. Like the other mentioned divergence measures, it is highly complex despite its good performance.

The Kullback–Leibler divergence has been used to model the physical microstructure properties of steel (Lee et al., 2020), separation of multi-source speech sources (Togami et al., 2020), and extracting features in the development of an impulse-noise resistant LBP. Yuhong et al. (2019) have used $D_{KL}$ to develop a scale filter bank in a CNN model to create a down-sampled spectrum from two distributions.

Datasets

This study uses nine publicly available image datasets: CIFAR-10, CIFAR-100 (Krizhevsky, Nair & Hinton, 2014a, 2014b), MNIST (LeCun, Cortes & Burges, 1998), Fashion-MNIST (Xiao, Rasul & Vollgraf, 2017), Caltech 256 (Griffin, Holub & Perona, 2022), Stanford Dogs 120 (Aditya et al., 2011), MIT Indoor Scenes (Ariadna & Antonio, 2009), ISIC 2016 (Gutman et al., 2016) and ChestX-ray8 (Xiaosong et al., 2017) to examine the proposed approach. Other studies have used these datasets, making them suitable for performance comparison. The data from the ChestX-ray8 dataset is a subset of over 108,000 images. The smaller ChestX-ray8 and ISIC 2016 sets have been used to show the advantages of transfer learning in cases of inadequate data. Table 1 below shows the dataset sizes and sets.

Data preparation

Since the approach utilises selected images from the larger datasets, the images are input into the pre-trained models with 224 × 224 pixel dimensions. The images are converted into grayscale, and their features are extracted using the first convolutional layer of the selected model. The pre-trained model is the feature extractor since it is also used in the final transfer learning process. This attribute makes it ideal for preparing the final transfer learning environment. Once the feature selection is made through the proposed approach, the dataset is categorised as a training or test dataset.

Experimental setup and settings

This study uses five pre-trained models: ResNet50, DenseNet169, InceptionV3, VGG16, and MobileNetV2 (used to show the proposed approach performance on small networks). The inputs to the models have been scaled to 224 × 224 pixels with the InceptionV3 model using an upsampling layer and VGG16 taking 4,096 neurons in the last layer before classification. These pre-trained models have been trained on the ImageNet dataset. The pre-trained models and their parameters are listed in Table 2, with InceptionV3 having the most layers (Team, 2023). The study also uses a custom CNN model to show the proposed methods’ effects on a non-pre-trained model.

The experiments have been conducted using the TensorFlow framework using the Keras library on the PaperSpace platform (A4000, 45 GB RAM, 8 CPU with 16 GB GPU). The training of the models involved the selected datasets without data augmentation. However, during training, fine-tuned model regularisation was performed using Dropout and Batch normalisation.

Proposed approach methods

The proposed approach introduces four methods from image textural features and pre-trained model layer selection views.

Textural features view in image data

This view utilises two methods: Above average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ and below-average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$.

Above average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$: The method uses data points whose ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ is higher than the average for all the data points in their category.

Below average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$: The method uses data points whose ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ is lower than the average of other samples in their class.

Layer selection view in pre-trained models

This view uses two methods: Positive weights ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ and negative weights ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$.

Positive weight ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$: The method evaluates divergence between two distributions formed from positive weights of filters of a pre-trained model’s convolutional layers.

Negative weight ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$: The method compares two distributions formed from negative weight filters in the pre-trained model’s convolutional layers.

${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ is used and compared to four divergence measures in both views to determine the reasons behind its selection.

Commonly used transfer learning methods

Since the introduced methods in the proposed approach aim to improve the transfer learning process, they are compared with these four commonly used methods in transfer learning:

Standard fine-tuning: The method replaces the classification layer with a classification layer for the target task’s classes.

Last-k layer fine-tuning: It involves replacing the last-k layers in the network with other layers suitable for the target task. These layers can be 1, 2 or 3 in the pre-trained model.

Results on conflated textural features methods

The two textural methods: GLCM and LBP accuracy performance, are shown in Tables 3–7 for the various datasets and models.

GLCM’s energy and homogeneity give the best accuracy performance compared to the LBP and GLCM’s correlation. LBP gives the lowest performance compared to the GLCM properties, while ResNet50 performs lowest on CIFAR-100 among the datasets.

In the VGG16 model, correlation and energy perform best among the GLCM properties. The CIFAR-10 give the highest performance across all the properties, while CIFAR-100 still gives the least accuracy.

GLCM’s Energy and LBP perform better than the other properties on the InceptionV3 model. GLCM’s correlation and homogeneity give the least performance for the InceptionV3, and the CIFAR-10 dataset gives the best accuracy performance across the four textural properties among the datasets. Figure 2A shows the conflated performance of CIFAR-10 samples on the VGG16 pre-trained model.

Figure 2 shows that samples below the average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ perform better, illustrating the importance of selecting quality data points in a dataset.

GLCM’s energy and LBP give better results than the other properties on the DenseNet169 model. GLCM’s correlation gives most of the least accuracy performance values, and the CIFAR-100 dataset still gives the least performance.

GLCM’s energy and homogeneity give the best accuracies for the MobileNetV2 pre-trained model, while the GLCM’s correlation gives the least accuracy. GLCM’s energy gives more than half the best results of all four properties, followed by GLCM’s homogeneity, LBP and correlation. Figure 3 shows the energy property performance of Fashion-MNIST and MNIST on the MobileNetV2.

The dataset samples with below-average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ give high accuracy when using both MNIST and Fashion-MNIST, showing the divergence between the data points and their effect in adapting the pre-trained model in the target task. The relevance of these target data points to the source samples is essential to the adaptation.

Results on dynamically selected layer methods

The dynamic layer selection methods form the second part of the model. The results of the selected layers for the pre-trained model are presented in Tables 8–12.

Before using the selected conflated ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$, MNIST performs best when utilising selected dynamic layers, with the CIFAR-100 dataset giving the least performance. However, using conflated ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ samples of the CIFAR-100 dataset and the positive ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ dynamically selected layers improves the ResNet50 pre-trained model’s performance, as seen in Table 8. The negative ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ dynamically selected layers also result in minimal improvement due to the conflated dataset samples.

Using dynamically selected layers performs well compared to the standard methods (compared in Results on Methods against commonly used measures). However, an improvement is noted when employing samples with below-average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$, as seen in Table 9. The CIFAR-100 dataset gives the least accuracy, while the MNIST dataset performs best. Figure 4 shows the performance of the ISIC 2016 dataset on the VGG16 pre-trained model, with and without conflated selected images and dynamic layers.

The dataset with data points below average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ performs better on VGG16 pre-trained model than those with above-average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$. The performance improves as the dynamically selected layers fine-tune the pre-trained model. In Fig. 5, the selected layer (layer 1) with lower ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ is seen to have more excitatory weights (light ones) in the first channel of the layer than layer 9 of the VGG16 model. Its selection coincides with the first layers being able to extract features better.

A similar trend of improvement in fine-tuned dynamically selected layers’ model is noted in the InceptionV3 pre-trained model, as seen in Table 10. The MNIST datasets still perform well, probably owing to their size of data points and classes compared to the poorly performing CIFAR-100 dataset. The two smaller datasets: ChestX-ray8 and ISIC 2016, also perform well.

The trend of low performance continues with the CIFAR-100 dataset for the DenseNet169 pre-trained model, as seen in Table 11. The MNIST datasets continue to perform best by utilising below-average selected samples, improving the adaptation process’s performance in the target task.

The MobileNetV2 pre-trained model gives similar results to the other four pre-trained models, as shown in Table 12. The MNIST dataset gives the best accuracy among the datasets, and a performance improvement is noted when data points of below-average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ are used together with the positive ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ dynamically selected layers in the transfer learning process. The excellent performance of the positive ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ when using the selected samples is seen in the precision shown in Table 13.

The precision determines the repeatability of obtaining the models’ good performance, indicating how well the model can predict the correct class. Table 14 looks at the recall performance of using the positive samples to determine the model’s performance in correctly identifying the positives. Figure 6 shows a confusion matrix of the ISIC 2016 testing dataset on MobileNetV2, where the number of false negatives and false positives is lower in classifying benign or malignant conditions.

In Table 14, the recall values are lower than the accuracy and the precision by a slight margin. However, it proves that the model can identify a good proportion of the positives during classification.

Results of proposed methods against commonly used transfer learning methods

The introduced methods of the proposed approach have been compared to the three commonly used methods of the k-1, k-2 and k-3. Table 15 shows that the introduced methods outperform these regular methods.

The combination of the positive ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ in dynamic selected layers and the use of data points below conflated average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ give an average improvement of 0.87% across all the five pre-trained models, with the DenseNet169 model giving the best improvement of 1.57% and the VGG16 model giving the slightest improvement of 0.06% in comparison to the standard fine-tuning as seen in Table 15 for the ChestX-ray8 dataset. Among the last-k methods, the combination dramatically improves on the k-3. A similar performance is shown in Fig. 7.

Combining data points below the average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ and dynamically selected layers gives better accuracy than the commonly used and individually introduced methods. Utilising above-average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ samples without dynamically selected layers gives the least performance, as shown in Fig. 7. Using below-average samples gives the second-best method. Figure 8 shows the ISIC 2016 dataset sample with and without transfer learning in the VGG16 pre-trained model. When the same samples are used in transfer learning, it is noted that below-average ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ performs better and takes less time than a typical convolutional neural network (a 12-layer CNN listed in Table 2). The typical CNN takes 50 s longer to train the ISIC 2016 samples.

Results on computational complexities in proposed methods

The proposed approach complexity is evaluated against four divergence measures: Wasserstein, Hellinger, Jensen–Shannon and Bhattacharya. The results in Table 16 show that the ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ has a good balance of memory and time complexities.

The Wasserstein and the Jensen–Shannon have better time complexities but consume more computing resources than the ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$. Tables 16 and 17 show that it takes about 70.37 milliseconds (ms) to select a sample and pass it through a selected layer in ResNet50. The complexities of the ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ are better than Hellinger’s and Bhattacharyya’s, which take 99.63 and 239.14 ms, respectively. According to Yossi, Carlo & Leonidas (2000), the Wasserstein is reported to be more complex than Bhattacharyya, forming a reasonable basis for selecting ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ in this study. Further evaluation of the divergence measures is shown in Fig. 9.

The models’ accuracy performance when using ${\mathrm{D}}_{\mathrm{K}\mathrm{L}}$ conflated dataset samples is closest to Hellinger’s conflated dataset samples, as seen in Fig. 9, outranking the other divergences. However, as reported in Tables 16 and 17, Hellinger has a higher computational complexity.