Improving prediction of maternal health risks using PCA features and TreeNet model

Dataset for experiments

The dataset used in this study was originally created by Marzia et al. from Daffodil International University in Dhaka, Bangladesh, and is publicly available (UCI Machine Learning Repository, 2021). The dataset has been collected using an IoT-based risk monitoring system implemented in various healthcare facilities, including hospitals, community clinics, and maternal health centers (Afreen & Bajwa, 2021). Additionally, a benchmark dataset with similar characteristics is available on Kaggle (2022). The dataset consists of seven features: Age, Bs, RiskLevel, SystolicBP, HeartRate, DiastolicBP, and Bodytemp, which are used as target classes. A comprehensive description of the maternal health dataset including details of its attributes is provided in Table 2. The dataset contains a total of 1014 samples of maternal health data, with 406 instances classified as low-risk, 336 as mid-risk, and 272 as high-risk.

Data preprocessing and feature selection

The data preprocessing includes the label encoding of the categorical attribute ‘RiskLevel’. The second step of preprocessing includes the feature selection technique employed to identify the most relevant features for training the machine learning models. These techniques involve extracting and combining selected features to create an efficient feature set. Feature selection plays a vital role in achieving a good fit for machine-learning models as each feature has its significance for the target class. Therefore, an approach that incorporates only the features that contribute significantly to the final class prediction is developed. This approach offers several advantages such as easier interpretation of learning models, reduction of model variances, and decreased training time and computational costs. To get the best feature solution, PCA is used as a feature selection approach in this study. By using PCA, the system’s complexity is decreased while classification accuracy and stability are improved. The most useful features for the machine learning model can then be chosen by using PCA to find the principle components that capture the most important variances in the data. A detailed description of the PCA is given below.

Principal component analysis

PCA is a commonly used technique for reducing the dimensionality of large datasets. This is accomplished by transforming a large set of features into a smaller set while retaining most of the relevant information from the original data. While reducing the number of features inherently sacrifices some accuracy, the key idea behind dimensionality reduction is to balance accuracy with simplicity. The dataset can be made easier to handle, explore, and visualize by simplifying it. Data processing is additionally sped up by machine learning algorithms, which can handle the data more effectively without a load of irrelevant features.

Machine learning models

This study employed supervised machine learning classifiers to analyze maternal health risk data. The classifiers are implemented in Python using the ’Sci-kit learn’ module. They are trained on a set of data samples dedicated for training purposes and evaluated on a separate test set that was unfamiliar to the classifiers. Several models, including RF, DT, ETC, LR, XGBoost, and SGD are individually utilized to construct the ensemble model. The optimal hyperparameter settings for these models are determined through a fine-tuning process. In this section, a brief overview of the ML classifiers used in this study is provided.

Proposed TreeNet model

The suggested model in this work combines two highly effective classifiers, the MLP, and ET classifiers, which are applied to the dataset for maternal health. The ET is an ensemble model based on trees, whereas the MLP is a neural model. These models are combined to create a strong hybrid model that takes advantage of both models. The reason for creating an ensemble of these two models is that they perform best among all other models individually. The soft voting criteria is used to combine the models, where the average probability for each class is determined by averaging the probabilities of each class predicted by each model. To arrive at a final prediction, this method considers the combined knowledge of the individual models. Figure 1 shows a graphical representation of the tree model, which helps to illustrate the ensemble model’s structure and decision-making process.

The predictions of various machine learning algorithms are combined in the ensemble model to increase prediction accuracy and robustness. Both the MLP and ET models are independently trained on the same dataset for the ET+MLP ensemble model. Predicted probabilities are produced by each of these models for the various classes of the target variable. These projected probabilities are pooled to create a final forecast for each observation in the dataset. Taking a weighted average of the predicted probabilities is a typical technique for combining predictions. The weights allocated to each model’s prediction are often decided based on how well they perform on a validation set or using methods like cross-validation. The ensemble model seeks to provide improved predictive performance and improve the overall accuracy and reliability of the forecasts by combining the advantages of the MLP, and ET models.

The proposed ensemble tree model uses the advantages of two different machine-learning methods to produce predictions that are more accurate. We can improve the model’s capacity for generalization and reduce overfitting by training multiple models on the maternal health dataset and combining their predictions. The suggested ensemble model’s operation is described by Algorithm 1 .

 
_______________________ 
Algorithm 1 Ensemble of ETC and MLP._______________________________________________ 
Input: input data (x,y)Ni=1 
METC = Trained_ETC 
MMLP  = Trained_MLP 
  1:  for i = 1 to M do 
 2:     if  METC ⁄= 0 & MMLP ⁄= 0 & training_set ⁄= 0 then 
 3:         ProbMLP − lowRisk = MMLP.probability(lowRisk − class) 
  4:         ProbMLP − midRisk = MMLP.probability(midRisk − class) 
  5:         ProbMLP − highRisk = MMLP.probability(highRisk − class) 
  6:         ProbETC − lowRisk = METC.probability(lowRisk − class) 
  7:         ProbETC − midRisk = METC.probability(midRisk − class) 
  8:         ProbETC − highRisk = METC.probability(highRisk − class) 
  9:         Decision function = max(      1_______ 
Nclassifier ∑ 
        classifier 
           (Avg(ProbETC−lowRisk,ProbMLP−lowRisk) 
              ,(Avg(ProbETC−midRisk,ProbMLP−midRisk) 
              ,(Avg(ProbETC−highRisk,ProbMLP−highRisk) 
10:     end if 
11:     Return final label ^ p 
12:  end for____________________________________________________________________________    

M: Total number of learning models in the voting (two in our case).

N: Total number of samples in the dataset for x-¿feature and y-¿target.

p-hat: Represents the predictive probabilities of each test sample.

n: Total test sample probabilities.

Both ${\sum }_i^{n}ET{C}_i$, ${\sum }_i^{n}ML{P}_i$ produce prediction probabilities for every test sample. After being aggregated for each test case, these probabilities are then subjected to the soft voting criterion, as shown in Fig. 2. The highest average probability among the classes is taken into account, and the projected probabilities from the two classifiers are combined, to determine the final class in the ensemble model. Using the class with the highest likelihood score as a starting point, the final prediction will be made. (3)${\displaystyle \widehat{p}=argmax\left\{\sum _i^{n}ET{C}_i,\sum _i^{n}ML{P}_i\right\}.}$

To elucidate the capabilities of the proposed approach, let’s consider an illustration. This approach entails passing a sample through the ETC and MLP components. Following this process, probability scores are assigned to each class. Specifically, for ETC, Class 1 (lowRisk), Class 2 (midRisk), and Class 3 (highRisk) have likelihood scores of 0.6, 0.7, and 0.8, respectively. Similarly, for MLP, Class 1 (lowRisk), Class 2 (midRisk), and Class 3 (highRisk) have probability scores of 0.4, 0.5, and 0.6, respectively.

In this scenario, let g(x) denote the probability score of x, where x belongs to the three classes in the dataset. The domain of x is confined to these three classes. Therefore, the probabilities for the three classes can be determined as follows:

P(lowRisk) = (0.6+0.4)/2 = 0.50

P(midRisk) = (0.7+0.5)/2 = 0.60

P(highRisk) = (0.8+0.6)/2 = 0.70

The final prediction will be highRisk, whose probability score is the largest, as shown below: (4)${\displaystyle VC\left(ETC+MLP\right)=argmax\left(g\left(x\right)\right)}$

The final class is determined by the VC(ETC+MLP) using the highest average probability among the classes, and it combines the projected probabilities from both classifiers.

Evaluation metrics

Several assessment criteria, such as accuracy, precision, recall, and F1 score are frequently employed to assess a model’s performance. The true positive (TP), true negative (TN), false positive (FP), and false negative (FN) values from a confusion matrix can be used to determine these parameters.

Accuracy gauges how accurately the model’s predictions are made overall and is computed as (5)${\displaystyle Accuracy=\frac{TP+TN}{TP+TN+FP+FN}}$

Precision measures how well the model can pick out positive cases from all those that are projected to be positive. It can be calculated using (6)${\displaystyle Precision=\frac{TP}{TP+FP}}$

The recall is sometimes referred to as sensitivity or the true positive rate and measures the model’s ability to properly detect positive events. The recall is determined by the formula (7)${\displaystyle Recall=\frac{TP}{TP+FN}}$

F1 score balances the trade-offs between precision and recalls into a single parameter. The F1 score is determined as follows (8)${\displaystyle F1score=2\times \frac{Precision\times Recall}{Precision+Recall}}$

It is the harmonic mean of precision and recall. These evaluation parameters provide valuable insights into the model’s performance, considering different aspects such as overall accuracy, precision in positive predictions, and the model’s ability to detect positive instances.

Experimental setup

This study conducted multiple experiments to evaluate and compare the performance of the proposed approach with various deep learning and machine learning models. All experiments are executed on a Windows 10 machine equipped with an Intel Core i7 7th generation processor. The proposed technique, as well as the machine learning and deep learning models, are implemented using Python frameworks such as TensorFlow, Keras, and Sci-kit Learn. The maternal health data is divided into 85% for training purposes and 15% for testing purposes. The experiments are conducted separately, using both the original feature set from the maternal health risk dataset and the feature set derived from PCA.

Performance of models using original features

The initial set of experiments utilized the original feature set from the maternal health risk dataset. Table 3 presents the results obtained from various classifiers when applied to the original features. The outcomes indicate that the proposed ensemble model outperformed all individual learning models with an accuracy of 80.03%. The accuracy scores for the ETC and MLP classifiers are 77.08% and 79.45%, respectively. The deep learning model CNN achieved an accuracy score of 72.38%, while the tree-based model RF obtained the lowest accuracy among all models at 70.65%. It is important to note that the ensemble of tree classifiers with linear models (ETC+MLP) exhibited superior performance when applied to the original feature set.

When compared to linear models the tree ensemble model performs noticeably better. The efficacy of the voting model when handling a sizable number of features is the main driver of this development. The individual performances of the ETC and MLP classifiers are both satisfactory, and the combined results are even better. However, despite the commendable performance of the ensemble model the achieved accuracy still falls below the desired level for the accurate prediction of maternal health risks. Consequently, additional experiments are conducted to address this issue by utilizing PCA-extracted features.

Performance of models using PCA features

Table 4 shows the results of the machine learning models developed using the dataset’s PCA features. The outcomes of the subsequent set of experiments conducted using PCA features to evaluate the effectiveness of both machine learning models and the proposed ensemble model indicate better results. The inclusion of PCA features aimed to select the most important features and enhance the accuracy of linear models. These PCA-extracted features are utilized for training and testing the machine learning models.

According to the experimental results, the proposed ensemble model outperforms all other models with a remarkable accuracy of 98.25%. When compared to the original features, this results in a considerable performance improvement of 18.22%. The performance of the individual linear models is also increased when PCA features are used. When compared to the original feature set MLP’s accuracy is 95.43%, a 15.98% improvement while ETC’s accuracy of 93.42% showed a 16.34% improvement. On the other hand, when using the PCA features, LR and the tree-based classifier DT obtained lower accuracy scores of 88.33% and 89.31%, respectively. When PCA is used for feature extraction, the models exhibit significantly better performance. Due to the strong connection between the features produced by PCA and the target class that makes the data linearly separable, linear models are better than other types of models.

Comparison of machine learning models with original and PCA features

We conducted a thorough evaluation by comparing the performance of various machine learning models using both the original feature set and the features extracted through PCA. The objective is to assess the effectiveness of the proposed approach. The outcomes unequivocally demonstrated that incorporating PCA features in the second experiment, as opposed to utilizing the original dataset, led to a significant improvement in the performance of the machine learning models. To provide a comprehensive evaluation of their effectiveness, a comparison of machine learning models in terms of accuracy is presented in Fig. 3, in terms of precision in Fig. 4, in terms of recall in Fig. 5, and in terms of F1 score in Fig. 6.

Results of the K-fold cross validation

We used K-fold cross-validation to further analyze the performance of the proposed approach. The results of 5-fold cross-validation are shown in Table 5, which demonstrates how well the proposed technique performs in terms of accuracy, precision, recall, and F1 score when compared to other models. Furthermore, it shows a low standard deviation, indicating stable performance throughout a variety of folds. These outcomes provide us with more assurance that the proposed technique is trustworthy and reliable.

Discussion

The study conducted a thorough analysis of various machine learning models’ performance using both the original features and features extracted through PCA. Initially, when applied to the original feature set, the proposed ensemble model outperformed individual models, achieving an accuracy of 80.03%. However, this fell short of the desired accuracy level for predicting maternal health risks.

The subsequent experiment utilizing PCA-extracted features showcased substantial improvements across models. The ensemble model’s accuracy significantly increased to 98.25%, marking an 18.22% improvement over the original feature set.

Comparing models using both feature sets, the study highlighted the considerable enhancement in accuracy when employing PCA features. Linear models, especially, demonstrated substantial accuracy improvements, suggesting the features’ ability to make the data more linearly separable.

Overall, the incorporation of PCA-extracted features notably boosted the predictive power of the models, particularly enhancing the ensemble model’s performance, thus signifying the effectiveness of the approach in accurately predicting maternal health risks during pregnancy.

Performance comparison with existing studies

A detailed comparison was made with nine pertinent research works that have produced models with an emphasis on accuracy improvement to assess the performance of the proposed model in comparison to current state-of-the-art models. These chosen works serve as comparisons for determining the efficiency of the proposed model and emphasizing its improvements over present methods. This research work offers insights into the superior performance of the proposed approach in terms of accuracy improvement by comparing the findings of the proposed model with those of the chosen state-of-the-art models. For instance, the SVM was used with a few selected features and attained an accuracy of 94% in Irfan, Basuki & Azhar (2021). In Table 6, the proposed model and the current research on the same dataset are thoroughly compared in terms of performance. In Mutlu et al. (2023) and Umoren, Silas & Ekong (2022), the authors applied DT and achieved 89.16% and 89.2% accuracy respectively. Pawar et al. (2022) have achieved 70.21% of accuracy. It can be observed that individual machine learning models have not shown good results for predicting maternal health because of the diversity of the dataset. In Raza et al. (2022), the SVM model incorporating the DT-BiLTCN features attained a 98% accuracy, outperforming other models presented in Table 6. Their proposed DT-BiLTCN is based on complex learning model layers, especially with multiple layers and parameters, and could be challenging to interpret, making it harder to understand how and why certain predictions are made. However, the proposed model is a simple ensemble model with improved accuracy results. In terms of several performance evaluation parameters, this comparison reveals that the ensemble model using PCA features beats the other approaches.

Shapley additive explanation

Understanding the relationships between inputs and outputs in machine or deep learning models can be challenging, given that these models are often perceived as opaque or black-box algorithms. This lack of transparency, particularly when working with labeled data, hampers a comprehensive comprehension of the importance of features in supervised learning on both a global and local scale. A recent advancement, the SHAP technique, addresses this issue by providing a quantitative approach to assess model interpretability. This breakthrough, initially introduced by Lundberg & Lee (2017) and subsequently expanded upon by Lundberg et al. (2020), allows for a more nuanced understanding of the significance of elements within the model (Ahmad, Eckert & Teredesai, 2018; Lundberg & Lee, 2017).

SHAP employs the linear additive feature attribute method, drawing inspiration from cooperative game theory, to elucidate complex models. This method assigns an importance value to each attribute based on its impact on the model’s predictions, contingent on the presence or absence of specific features during SHAP estimation. By employing this explanatory approach, the intricacies of complex models become more accessible through a simplified model. The application of the linear additive feature attribute technique, grounded in cooperative game theory principles, is extensively detailed in works by Lundberg & Lee (2017) and further expanded upon by Lundberg et al. (2020) (Ahmad, Eckert & Teredesai, 2018; Lundberg & Lee, 2017). (9)${\displaystyle f\left(a\right)=g\left(a^{\prime }\right)={\varphi }_0+\sum _{j=1}^j{\varphi }_j{a}_j^{\prime }}$

The original ensemble learning model under consideration is denoted as (a), while the simplified explanation model is represented as g(a′). Here, $a_j^{\prime }$, where j signifies a simplified input seismic attribute number, refers to these attributes. SHAP values, denoted as ϕ_j, are calculated for all possible input orderings represented by j. The presence or absence of a specific seismic attribute is defined using an input vector, $a_j^{\prime }$, during estimation. Finally, ϕ₀ represents the model prediction when none of the attributes are considered during estimation. The comprehensive feature importance, calculated using SHAPly and arranged in descending order, is presented in Table 7. The analysis with SHAP highlights the significance of features in predicting maternal health. While SHAP feature importance surpasses traditional methods, relying solely on it offers only limited additional insights.