Application of machine learning algorithms to predict the thyroid disease risk: an experimental comparative study

Decision tree

The decision tree technique is a subset of supervised machine learning that is based on a continuous data splitting mechanism across specified parameters (Rokach, 2016). Internal nodes reflect attributes, branches represent decision laws, and each leaf node in the recursive decision tree algorithm interprets the outcome. The data is categorized using the Top-Down approach, which involves sorting from the root to a leaf/terminal node. After obtaining the training data, it splits the dataset into small subsets using the Gini Index, Information Gain, Entropy, Gain Ratio, and Chi-Square. The Gini Index and Information Gain are measured in the majority of datasets using Eqs. (1) and (2) respectively. The process is repeated for each child tuple until all tuples belong to the same class and no additional attributes are needed. The Gini Index is used to increase precision and diagnostic accuracy. (1)${\displaystyle InformationGain=Entropy\left(S\right)\left[\left(WeightedAvg\right)\ast Entropy\left(eachfeature\right)\right].}$ (2)${\displaystyle GiniIndex=1-\sum _{j=1}^c\frac{}{}{\mathbf{P}}_j^2.}$

Here in Eq. (2), P_j is proportion of the samples that belongs to class c for a particular node.

This algorithm traverses the potential decision-making space using a top-down greedy search strategy, never retracing and reconsidering prior choices (Jin, De-Lin & Fen-Xiang, 2009). The metric used in this algorithm to evaluate the best attribute at each point of the expanding tree is precisely information gain. To begin, a predictive model must be trained using available data and then validated for accuracy and reliability using test data collected specifically for the purpose of predicting proven test outcomes (Song & Ying, 2015). To evaluate the optimal attribute for breaking records using Attribute Selection Measures (ASM), we may employ various methods, including the Gini index or information gain(Tangirala, 2020). This attribute then subdivides the dataset into smaller subsets for each child until there are no more attributes.

As shown in the diagram, the model is initially created using training data. The model’s accuracy is determined, and it is optimized by estimating the known outcome using the data. Finally, the model can be used to make predictions about future outcomes. Predictive ratings would be calculated after the computer program has constructed the forest. The predictive score is calculated as a percentage of the objective predictor in the eligible model’s terminal node (or leaf). We can use various strategies such as the Gini index or knowledge gain to determine the best attribute to break the records using Attribute Selection Measures (ASM). This attribute then subdivides the dataset into smaller subsets for each child until no more attributes remain. Figure 1 depicts the working mechanism of the decision tree.

Random forest

A random forest is composed of multiple independent decision trees that function in combination (Liaw et al., 2002). Each tree produces ad hoc data samples. Based on the votes, it determines the best prediction score. Additionally, it identifies significant elements within a dataset and provides a simple indicator of the feature’s significance. Feature selection is often used in classification studies to reconstruct the data and improve accuracy. The feature selection technique is applied in a number of ways, including filtering (Filter) and encapsulation(Wrapper). The filtering method’s feature selection function was independent of the classification algorithm and was based on a feature of the data set. The utility of a function contributes to its accuracy (Lebedev et al., 2014). A random forest can yield more reliable ensemble forecasts than a decision tree, which is the primary distinction between the two. The F value of a test statistic for a single function is calculated by the feature selection procedure, which follows the Equation (3). (3)${\displaystyle normf{i}_i=\frac{f{i}_i}{{{\Sigma }_i}_{j\in allfeatures}fi}.}$ Here in Eq. (3), normfi_i is the normalized importance of feature i, fi_i represents the importance of feature i.

Following that, the total number of trees is divided by the value assigned to each node’s feature’s importance utilizing Eq. (4). (4)${\displaystyle RFf{i}_i=\frac{{\Sigma }_{j\in alltrees}normf{i}_{ij}}{T}}$

Here in Equation (4), RFfi_i the importance of feature i calculated from all trees in the Random Forest model, normfi_ij represents the normalized feature importance for i in tree j, and T is the total number of trees.

This technique extracted data points from the training set and constructed trees for each data point. Each decision tree generates a prediction result during the training phase. When a new data point is acquired, the Random Forest classifier predicts the ultimate decision based on the majority of products (Ali et al., 2012). Recent computational biology work has witnessed a growth in the usage of Random Forest, owing to its inherent benefits in dealing with tiny sample numbers, a high-dimensional feature space, and complex data structures (Qi, 2012). Using the random forest has a good chance of improving accuracy and assisting significantly in diagnostic cases. Figure 2 depicts the working mechanism of the random forest technique.

LightGBM

LightGBM is a gradient boosting technology built on top of a tree-based learning algorithm. It performs at a rate six times faster than the XGBoost algorithm (Wang, Zhang & Zhao, 2017). The algorithm is called “light” due to its computational power and ability to produce faster results. Unlike other tree-based algorithms, the LightGBM tree grows vertically, or leaf by leaf (Ke et al., 2017). Due to the dispersion of high-dimensional data, it is necessary to create a virtually lossless path in order to reduce the number of features. As a result, the training phase can be completed more quickly while keeping the same level of accuracy. As an outcome, diagnostic accuracy would ultimately improve. The XGBoost algorithm, which is based on the GBDT technique and the preceding definition, incorporates the regular function to prevent the model from becoming too well-fit. The decision tree model splits each node at the most informative feature (with the largest information gain). For GBDT, the information gain is usually measured by the variance after splitting, which is defined as below (Ke et al., 2017). Equation (5) is widely utilized in machine learning at present. (5)${\displaystyle {{\vee }_{j\mid }}_{\mathcal{O}}\left(d\right)=\left(\frac{1}{n_o}\frac{{\left({\Sigma }_{x_i\in \mathcal{O}:x_{ij\le d}}{g}_i\right)}^2}{n_{{l|\mathcal{O}}^j}\left(d\right)}+\frac{{\left({\Sigma }_{x_i\in \mathcal{O}:x_{ij\succ d}}{g}_i\right)}^2}{n_{r|\mathcal{O}}^j\left(d\right)}\right).}$

In Eq. (5), $\mathcal{O}$ represents the training on a fixed node of the decision tree. ${{\vee }_{j\mid }}_{\mathcal{O}}\left(d\right)$ is the variance gain of splitting feature j at point d for this node (Ke et al., 2017). For feature j, the decision tree algorithm selects $d_j^{\ast }=argma{x}_d{V}_j\left(d\right)$ and calculates the largest gain $V_j\left(d_j^{\ast }\right)$. Then, the data are split according feature j^∗ at point $d_j^{\ast }$ into the left and right child nodes (Ke et al., 2017).

This Eq. (5) can be utilized to define the algorithm’s regular term. This is a more precise definition that the algorithm inventor arrived at after conducting numerous experiments, and researchers can use it directly (Ma et al., 2018). This algorithm runs on the dataset, followed by pre-processing the training data for pre-training. The pre-processed data is then classified according to the importance of the features. This analysis will aid in our comprehension of the data (Ju et al., 2019). Figure 3 depicts the working mechanism of the LightGBM technique.

XGBoost

XGBoost is an approach that utilizes ensembles. Ensemble learning is a method for combining the predictive abilities of numerous learners in a systematic manner. As a result, a consolidated model is built from the output of multiple models. The ensemble’s models, which are occasionally referred to as fundamental learners, may originate from the same or different learning algorithms (Li, Fu & Li, 2020). If a dataset has n instances, which corresponds to n rows, we use i to represent each instance. The XGboost algorithm, based on the GBDT technique and the preceding definition, introduces the regular function Ω(f) to control the model’s over-fitting. (6)${\displaystyle min\left\{obj\right\}=min\left\{L\left(F_m\left(X\right),Y\right)+\Omega \left(f\right)+C\right\}.}$

Here, L(F_m(X), Y) represents the Loss function. The difference between the final predicted variables after m times iteration and the actual ones, Ω(f) indicates the regularization function in the final prediction model after all iterations.

Equation (7) is a more precise form that the algorithm designer derived after conducting numerous trials, and researchers can immediately apply it Ma et al. (2018). (7)${\displaystyle \Omega \left(f_t\right)=\gamma T+\frac{1}{2}\lambda \sum _{j=1}^T{\omega }_j^2.}$

In Eq. (7) the regular item function is a function of leaf number T, and leaf node output result ω_j.Ω(f_t) represents the the function of regularization in the t wheel iteration.

It automatically handles missing data values and tracks structured databases when dealing with classification and regression predictive modeling issues. XGBoost predicts residuals or errors of previous models (Chen et al., 2015), combined in order to generate the last forecast. It also improves efficiency by reducing the over-fitting of data sets. Figure 4 depicts the working mechanism of the XGBoost technique.

GaussianNB

The technique of Gaussian Naive Bayes (GaussianNB) is predicated on the susceptibility to predictor independence (Rish et al., 2001). A classifier of GaussianNB, in general, means that a function is not linked to any other variable in a class. It is easy to make for large data sets and particularly useful. As each feature is independent, the inclusion of one feature does not affect the appearance of other features in the GaussianNB algorithm. The algorithm is based on the Bayes theorem, and the probability can be calculated utilizing Equation (8). (8)${\displaystyle P\left(A|B\right)=\frac{P\left(B|A\right)P\left(A\right)}{P\left(B\right)}}$

The probability P(A|B) that we are interested in computing is referred to as the posterior probability. P(A) is the probability of occurrence of event A. Similarly, P(B) is the probability of occurrence of event B.P(B|A) denotes the conditional probability of event B occurring if A occurs. Figure 5 depicts the working mechanism of the GaussianNB technique.

KNN

K-Nearest Neighbors (KNN) is a technique for inducing laziness in learning (Gou et al., 2012). That is to say, it makes no assumptions about the distribution of the data on which it is based. Frequently, the model is evaluated using the dataset. It is beneficial when working with real-world datasets. Additionally, there are no training data points needed for model generation. All training data is incorporated into the testing process. While this expedites planning, it delays testing and necessitates a great deal of time and memory. When building the model, the number of neighbors (K) must be specified in KNN. In this case, K acts as a controlling variable for the prediction model. When the number of classes is even, K is usually an odd number. It does not immediately learn from the training set; rather, it maintains the dataset and uses it for classification. Figure 6 depicts the working mechanism of the KNN technique.

ANN

Artificial Neural Networks (ANN) is designed to replicate human studies by modeling human brain structures (Dongare et al., 2012). There are input and output layers in neural networks, as well as a hidden layer that contains components that convert the input to a format that the output layer can use. Three interconnected layers comprise ANNs. It is also known as a Feed-Forward Neural Network (Benardos & Vosniakos, 2007).

The input neurons receive the data that we feed the ANN in the input layer. These neurons transmit data to the hidden layer, which performs the magic, and then transfer output neurons to the output layer, which stores the network’s final calculations for future use. The data will produce outputs with insufficient data after ANN training. It can also learn on its own and provide results. Figure 7 depicts the working mechanism of the ANN technique.

SVC

In the features that categorize data points, the Support Vector Classifier (SVC) algorithm functions as a hyperplane (Rueping, 2010). Maximizing the data point distance margin reinforces the need for future data point organization. The distinction between SVC and SVM is that LinearSVC is defined in terms of liblinear, whereas SVC is defined in terms of libsvm. That is why LinearSVC offers greater flexibility in terms of penalty and loss function selection. Additionally, it scales better to large sample sizes. It is a technique for determining the fittest separation hyperplane. The hyperplane has the greatest margin gap between the two groups’ nearest points. This approach ensures that the larger the margin, the lower the classifier’s generalization error. Figure 8 depicts the framework of SVC.

CatBoost

The name “CatBoost” is derived from the phrases “Category” and “Boosting”. CatBoost supports numerical, categorical, and text features but excels at categorical data handling (Dorogush, Ershov & Gulin, 2018). It is a publicly available machine learning algorithm. It is based on the gradient boosting library. Additionally, it is capable of producing outstanding outcomes with a minimal amount of data, as opposed to deep learning models, which require a vast amount of data to train from. In terms of performance, CatBoost vastly beats all other machine learning algorithms. Additionally, CatBoost can be used to convert categories to numbers without conducting any explicit pre-processing. Figure 9 depicts the working mechanism of the CatBoost technique.

Extra-Trees

The Extra-Trees classifier is a decision tree-based ensemble learning approach. Extra-Trees classifier randomly selects specific decisions and subsets of data in order to avoid overlearning and overfitting. The Extra-Trees classifier is a decision tree-based ensemble learning approach (Bhati & Rai, 2020). Extra-Trees classifier randomly selects specific decisions and subsets of data in order to avoid overlearning and overfitting. It creates several trees and separates nodes based on random feature subsets. Randomness is introduced into Extra-Trees not by data bootstrapping but through random splits of all observations. Figure 10 depicts the framework of the Extra-Trees classifier.