Feature-based enhanced boosting algorithm for depression detection

Motivation and contribution of the study

Recently, detecting and monitoring depression remains a major challenge in healthcare. Government entities, surveillance systems, and mental health professionals are actively seeking effective ways to identify individuals at risk of depression at an early stage, as providing timely intervention and treatment is crucial for improving outcomes.

Social media platforms offer a unique opportunity in this regard, as people often express their thoughts, feelings, and emotions through text, images, and other modalities on these platforms. This has spurred growing research interest in leveraging social media data and developing computational methods to analyse this data for potential markers of mental health conditions like depression. However, recently with the development of machine learning, deep learning and AI have been used to exploit complex patterns and relationships that exist within unstructured social media data. Boosting algorithms, which combine multiple weak prediction models into a powerful ensemble classifier, have emerged as a promising approach for this task.

Various pre-boosting techniques, such as AdaBoost, XGBoost, including custom built boosting methods, have been explored for depression detection on social media. However, these algorithms often struggle with challenges like handling diverse data types, reducing misclassification, and avoiding overfitting in high-dimensional, large-scale datasets. In addition, it remains a key objective to attain clinically acceptable accuracy while managing the higher error rates associated with boosting weak learners. The primary motivation for this study is to develop a novel boosting algorithm framework that addresses these limitations, and aimed at improving computational approaches for social media data analytics and text classification tasks that centres around depression detection as a case study. Thus, the main contributions of this study are:

• (1)

  This research introduces a new feature-enhanced boosting algorithm called F-EBA, which is specifically designed to address the challenging aspects of depression detection on social media platforms

• (2)

  F-EBA employs an adversarial layer to enhance resilience against synonymous and sarcastic social media posts, and uses hyperparameter tuning to refine feature sets for improved accuracy. This approach strengthens mental health analytics by closing the gaps in existing approaches to boosting as well as providing a strong solution for depression detection on social media.

• (3)

  The proposed F-EBA approach contributes a versatile framework that can be applied to other text classification tasks, paving the way for future research in related domains.

Boosting techniques/algorithms

In the context of predictive techniques, machine learning (ML) algorithms stand out as the most widely used predictive techniques for recognizing various mental illnesses, including depression (Abd Rahman et al., 2020; Arya & Mishra, 2021). These techniques are categorized into supervised, unsupervised, and semi-supervised learning methods, each with its own set of algorithms and models. Supervised machine learning techniques such as decision tree (DT), logistic regression (LR) and support vector machine (SVM) are used alongside unsupervised techniques, such as clustering and dimensionality reduction, to develop classification models from labelled data or even unlabelled data. Benton, Mitchell & Hovy (2017) discussed the advanced approaches such as supervised ensemble methods, transfer learning and multi-task learning, which combine several predictive models for improving performances. To this end, boosting algorithms or supervised ensemble methods, currently at the forefront of machine learning, utilized by numerous researchers (Nandanwar & Nallamolu, 2021; Reseena Mol & Veni, 2022; Mohammed et al., 2021; Ding et al., 2022), and have shown outstanding results for depression detection. Boosting (generally means increasing the performance of weak learners) is a popular sequential ensemble learning algorithm used to convert weak learners into strong learners aiming to increase the accuracy of the model (Breiman, 1996). Boosting algorithms are widely used for image recognition, motion tracking, and speech recognition. As previously mentioned, boosting algorithms come in two types, both of which are utilized for various scenarios, including depression, as discussed below:

Dataset

The data utilized in this study were collected from two publicly available datasets: Shen et al. (2017) and Coppersmith et al. (2015). First, TTDD or the so-called Shen et al. (2017) dataset, which has over >46 million records that are already labelled anchor tweets and unlabelled anchor tweets related to “depress” on Twitter from 2009 to 2016 and utilized by various researchers (Tong et al., 2023; Chiong et al., 2021; Shen et al., 2018; Zogan et al., 2021). It includes diverse user profiles, activities, behaviours, and social interactions. It comprises three subsets with labels: depressed (1,402 users and 292,564 tweet records), non-depressed (over 300 million users and 10 billion tweet records), and candidate users (36,993 depression-candidate users and over 35 million tweet records). The second dataset, CLPsych 2015 which was utilized by studies (Ahmad et al., 2020; Tong et al., 2023; de Jesús Titla-Tlatelpa et al., 2021), developed by Johns Hopkins University from 2008 to 2013 through Twitter API, comprises over 12.5 million tweets with labels “depressed” and “control users.” It includes 487 depressed users and 893 control users, each with up to 3,000 tweets (Coppersmith et al., 2015). This dataset was obtained through consent forms from the University of Malaya Research Centre.

As an additional note, both datasets are obtained from Twitter and are labeled based on self-stated labels, which serve as a useful starting point for identifying potentially relevant tweets/posts. Nonetheless, the content underwent additional filtering (further discovered in the data pre-processing section) by removing unrelated posts and employing machine learning techniques to identify broader patterns related to depressive language, facilitating us to recognize potential patterns of depression.

Data pre-processing and conversion

Both datasets utilised in this study exhibited class imbalance, requiring undersampling during training. The data extracted/obtained from both datasets in English, encompassed textual tweets, user profiles, tweet media like images (retrieved using Python’s urllib library), and emojis. It often contains abundant noise and a plethora of irrelevant or ambiguous parts such as scripts, HTML tags, and special characters, rendering much of it negligible for analysis. For this reason, Singh & Kumari (2016) argue that pre-processing and cleaning are vital processes for preparing the extracted data for classification, thereby ensuring its suitability for potential analysis. Due to this, both datasets underwent several preprocessing steps, including tokenization, stop word removal, lemmatization, and stemming to standardise the input. This process begins by eliminating all unnecessary text, punctuation, special characters, images, stop words, hyperlinks, usernames (e.g., “@username”), hashtags (e.g., preferring #sad over #sad_for_you), email addresses, and URLs from tweets or posts (refer to Fig. 2), utilizing Python’s “re” package for regular expressions. In addition, both lemmatization, which converts words to their base form, and stemming, which reduces words to their root form, were applied to the data to facilitate further analysis. Additionally, a list of contractions was generated to transform abbreviated words like “They’ve” into their full forms, such as “They have”. The curse words which convey strong emotions and frustrations, were retained in the text as they play a crucial role in depression detection. Finally, tokenization has been performed for potential classification analysis.

Experimentally, the words cloud, displaying the data cleaning results, can be observed in Fig. 2. Notably, the removal of “HTTP” and the extraneous term “t Co”, as well as the conversion of the word “diagnosed” to its base form “diagnose”, are evident. Additionally, the words “Love” and “Want” have been bolded to highlight non-depressed users. Moreover, the curse words like “shit” or “fuck”, commonly used by those experiencing depression, are deliberately retained in pre-processing as it plays a crucial role in depression detection.

The histogram presented in Fig. 3 showcases the cleaning and pre-processing results, displaying a significant reduction in common words from over 3,000 to under 500, with top frequencies highlighted for words linked to the most depressed users. This streamlines the dataset, improving pattern recognition and relationships between variables for more accurate models and better insights.

To streamline the data onto a single page in a textual data format, visual data such as images and emojis were converted into text. For converting emojis to text, the Python emoji library was utilized, which returns emojis along with their predefined meanings. To convert images (tweet media), a pre-built model a pre-trained model called Generative Image-to-text Transformer (GIT), developed by Wang et al. (2022b) model was chosen, which is more accurate in picking up top captions. Its accuracy rate stands at approximately 84%, compared to other available models. The GIT model, trained on the COCO dataset comprising 118 K images, is capable of generating captions and converting image text into textual data. The result is presented in Fig. 4, showcasing captions for non-depressed (first set of images), and depressed (second set of images), generated by the GIT model.

After converting emojis and images into textual data, it undergoes additional rounds of data cleaning and pre-processing. The resulting cleaned text is then fed into the F-EBA feature-engineering pipeline for further analysis.

Feature engineering pipeline

This approach was designed to generate optimized feature sets. It extracts, assigns weights and sentiment score probabilities, and eliminates low-weight features for optimal model efficacy. It encompasses four key components focused on picking up the most relevant features. The initial phase of this process involves feature extraction, a crucial step in this study’s experiment. It entails extracting various sets of features from the cleaned textual data. Unlike (Almouzini & Alageel, 2019; Safa, Bayat & Moghtader, 2022; Tong et al., 2023) who employed conventional feature extraction methods, this study utilized emerging structures and mechanisms for feature extraction and subsequent analysis. These mechanisms include: First, a pre-trained fourth version of the BERT encoder model, which captures contextual meaning in text, trained on Wikipedia and books corpus, was used with 12 hidden layers of Transformer blocks. The model was set up for embedding using Tensorflow_hub and Tensorflow_text libraries, and embeddings generated for every tweet in both Coppersmith et al. (2015) and Shen et al. (2017) datasets. To validate BERT embeddings, an investigation of similarities in the feature set has been conducted, visually representing the relationships within pairs of sentences for both users with depression and users without depression. The results are shown in Figs. 6, 7, and 8.

The analysis indicates the topmost similarities in feature sets within each group. The similarities in features indicate the semantic meanings and relationships between words, which investigate whether similar groups denote similar relationships or not, and later it aids the attention mechanism in focusing more efficiently on similar features. The closely aligned dots represent the most similar feature sets (labels 1-1 and 0-0). Comparatively, different groups (depressed and non-depressed) display significant variation, with sparse dots highlighting distinct feature sets (labels 0-1).

Second, the Word2Vec model was constructed, with various fined-tuned parameters such as vector size (representing the number of features per sentence or tweet), min_count (specifying the minimum word count necessary for sentence processing and embedding generation), and other variables. A vocabulary was built commencing both the Shen et al., 2017 and Coppersmith et al. (2015) datasets to train the Word2Vec model on tweet texts and image captions, which capture the semantic meaning and relationships among words. Subsequently, embeddings were generated for each tweet using this trained model. This embedding condenses the tweet’s textual content into a vector of 80 numerical features, which captures both the context and semantics meaning of the words present in the tweet. To validate the Word2Vec model, a similar approach was followed as for BERT embeddings, with the results shown in Figs. 9, 10, and 11.

The analysis indicates the topmost similarities in feature sets within each group. The similarities in features indicate the semantic meanings and relationships between words, which investigate whether similar groups denote similar relationships or not, and later it aids the attention mechanism in focusing more efficiently on similar features. The closely aligned dots represent the most similar feature sets (labels 1-1 and 0-0). Comparatively, different groups (depressed and non-depressed) display significant variation, with sparse dots highlighting distinct feature sets (labels 0-1).

Although BERT embedding has its own attention layer, employing feature vectors directly in WordVec embedding may hinder classification and decision tree training. To address this, a self-attention mechanism was applied to Word2Vec embeddings, assigning varying weights to different parts of the embedding. This involved two fine-tuned variables: input_dim (number of features in the tweet) and hidden_dim (dimensions of output vector). Initially, the query weight matrix (Q), key weight matrix (K), and value weight matrix (V) were generated by applying linear transformation to the input vector. Then, attention weights were obtained by performing dot product operations between Q, K, and V. This dot product encoding contextualizes each token, allowing the model to learn long-range dependencies, such as subject-verb agreement and relationships between distant words. Using this learned context, it can predict the next most likely word. Next, the Softmax function was applied for normalizing attention weights. Python NumPy library was utilized for Softmax and dot product computations. The attention mechanism results in Fig. 12 shows each dimension or feature with its weight, reflecting its importance in the sentences. Positive weights highlight significant feature sets, while negative weights indicate the presence of irrelevant features that cause noise or distraction, assigning them negative weights helps to minimize their influence in classification. Additionally, negative weights suggest unimportant feature sets, assigning negative weights to these features helps to diminish their significance (Brauwers & Frasincar, 2021; Junge, 2023).

Additionally, for precise feature selection and interpretation, and offering insights into the most relevant feature sets, the Stanford NLP library (return sentiment score and its associated probabilistic weight) was utilized to generate probabilistic sentiment scores for each word in a tweet and across all tweets in the datasets. The results are presented in a tree structure in Fig. 13, which shows the sentiment scores and probability weights of words in a sentence that are compared and combined with adjacent words. This yields an overall sentiment score and probability weight (reflects the Stanford library’s confidence in assigning sentiment scores, higher weights indicate stronger confidence in the assigned sentiment) for the entire sentence (Manning et al., 2014). The sentiment scores range from 0 to 4: 0 and 1 denote negativity, with 0 for negative and 1 for highly negative, 2 for neutrality, 3 and 4 denote positivity, with 1 for positive and 4 for highly positive sound in the sentences. The final results were then given into a custom decision tree.

The custom decision tree algorithm was developed and adhered to the recursive feature elimination (RFE) layer to select the best feature sets. In the initial phase, the decision tree was set up with fine-tuning variables including min_samples_split, max_depth, and features. Subsequently, the fit method was applied to train the decision tree, and the recursive_feature_elimination method was utilized to execute RFE. The method creates a copy of the input data X to keep track of the selected features. RFE operates within the context of decision trees, assessing feature importance by assigning weights based on accuracy for each iteration. It removes unimportant features (reflects the features that do not play a role/contribution to any iteration of the classification process) and repeats this process until reaching the desired number of features. The RFE result is presented in the tenth iteration in Fig. 14, which assesses the ranking of a feature for a single tweet, and each feature signifies its importance score in the corresponding iteration. Notably, features 30 (scoring 1 in the first iteration and score 3 in the fourth iteration) and features 70 (scoring 1 in the sixth iteration and scoring 3 in the third iteration) highlighting their significance with a score of 3 in two iterations each. Conversely, features 13, 23, and 58 consistently scored 0 across all iterations, indicating no contribution to the classification process and were subsequently removed by RFE. The rest underwent further selection with a decision tree to generate new or an optimized features set. Finally, the newly generated or optimized feature sets are stored in CSV files and then passed for further processing into the next pipeline of the F-EBA model.

The hyperparameters were selected through empirical testing. For BERT, we used a learning rate of 2e−5, batch size of 32, and trained for four epochs. The Word2Vec-based attention mechanism used input_dim = 300 and hidden_dim = 128. For the decision tree with RFE, max_depth = 10 and min_samples_split = 5, and we used 10 RFE iterations to optimize feature ranking and elimination.

F-EBA classification pipeline

This approach was designed in a way that encompasses two innovative concepts aimed at boosting weak learners and strengthening the model.

Addition of adversarial layer into F-EBA model

Modern machine-learning models are vulnerable to adversarial attacks that manipulate input data to mislead the model’s predictions (Apruzzese et al., 2019). Such attacks pose a significant threat to the reliability and integrity of these models. The datasets used in this study encompass a significant amount of synonymous text and sarcastic tweets. Additionally, when deploying the model online, it is crucial to strengthen the model against potential attackers seeking to disrupt its performance with manipulated input. To this end, this study integrated an adversarial layer to perform adversarial attacks on the proposed model, assessing its resilience against manipulated inputs. The adversarial layer employs the FGSM approach to generate adversarial examples, using a standard 0.05 perturbation. These examples are legitimate inputs with imperceptible perturbations that can mislead the model during the testing stage (Wang et al., 2022c). Thus, implementing a defense mechanism during the training of the F-EBA model is crucial to enhance its resilience and robustness (Peng et al., 2019). The F-EBA weak learners were trained by original embedding and the adversarial layer applies perturbations to the original embedding using FGSM-a straightforward yet potent technique, which operates utilizing the gradient of the loss function concerning the input data, then perturbing the input in the direction of the gradient’s sign. This process can be outlined as follows:

• (1)

  Compute the gradient of the loss concerning the word embeddings for each word in the input sentences.

• (2)

  Scale the gradient with a small constant (epsilon), and then add this modified vector to the original word embeddings. This action produces a perturbation that alters each word embedding.

Mathematically, the FGSM’s perturbations of word embeddings for the word “stressed” within an adversarial layer can be outlined as follows:

• (1)

  Magnitude _{stressed_embedding} = Loss (BiLSTM (original_embedding), true_label)

The magnitude shows how loss changes with “stressed” embedding; positive means increasing it raises loss, negative means the opposite, giving insight into the sensitivity of the loss to variations in the embedding.

• (2)

  Sign (Magnitude _{stressed_embedding})

Ascertain the direction in which the loss increases more rapidly by calculating the sign of the gradient.

• (3)

  ∈ × Sign (Magnitude _{stressed_embedding})

Multiplying a small perturbation, symbolized as epsilon, amplifies this sign, indicating both the direction and extent of the modification required to elevate the loss associated with the word “stressed”.

• (4)

  embedding_stressed_adversarial = embedding_stressed + ∈ × Sign (Magnitude _{stressed_embedding})

Merge this outcome with the original embedding to generate the adversarial embedding.

• (5)

  embedding_stressed_combined = embedding_stressed + embedding_stressed_adversarial

The combined embedding was applied to train the BiLSTM or BERT model, following which the boosting procedure resumed.

Below is the general mathematical equation that combines the generation of adversarial examples using the FGSM with the dual-layered training approach: $$L=L_{org}+\lambda .L_{adv}where{L}_{adv}=L\left(\theta ,x+ϵ\cdot sign\left(\mathrm{\nabla }xJ\left(\theta ,x,y\right)\right),y\right))$$

The equation describes:

• (1)

  L represents the total loss during training, combining the losses from both original and adversarial examples.

• (2)

  L(org) is the loss calculated from the original input embeddings

• (3)

  L(adv) represents the loss from the adversarial examples generated using the FGSM.

• (4)

  λ is a hyperparameter that balances the contributions of the original loss and the adversarial loss.

• (5)

  The term $\theta ,x+ϵ\cdot sign\left(\mathrm{\nabla }x|J\left(\theta ,x,y\right)\right),y)$ generates the adversarial example by applying perturbations to the original input $x$, where $ϵ$ controls the intensity of the perturbation and $\mathrm{\nabla }xJ\left(\theta ,x,y\right))$ is the gradient of the loss function with respect to the input, and the last $y$is the true label

This equation generates adversarial examples through the FGSM, which perturbs input data in a direction that maximizes the model’s loss, as explained in the FGSM mathematical representation, elucidating its operation within the algorithms. The defense mechanism involved training the model with both original embeddings and an adversarial layer. Thus, this dual-layered training approach significantly strengthens the model’s ability to recognize synonymous text within datasets and sarcastic tweets.

The F-EBA model outcomes under adversarial attacks with defense and without defense mechanism, employing a standard 0.05 perturbation, are outlined in Tables 3 and 4. By defense, the model was trained using a combination of original embeddings and an adversarial layer; without defense, it relies solely on original embeddings without incorporating an adversarial layer.

Table 3:

F-EBA Model’s performance against adversarial attacks without defense.

Evaluation metric	BiLSTM		BERT
	Shen et al. (2017)	CLPcych (2015)	Shen et al. (2017)	CLPcych (2015)
Accuracy	65.0	72.0	83.9	79.0
Precision	63.2	71.1	82.5	78.1
Recall	66.7	67.8	79.5	79.7
F1-score	63.8	62.5	81.3	73.8

DOI: 10.7717/peerj-cs.2981/table-3

Table 4:

F-EBA Model’s performance against adversarial attacks with defense mechanism.

Evaluation metric	BiLSTM		BERT
	Shen et al. (2017)	CLPcych (2015)	Shen et al. (2017)	CLPcych (2015)
Accuracy	96.9	97.3	97.5	97.3
Precision	96.4	96.2	97.1	96.2
Recall	95.8	95.3	96.9	95.3
F1-score	94.4	96.4	96.8	96.4

DOI: 10.7717/peerj-cs.2981/table-4

The overall performance of the F-EBA model appears to lower/sink when tested against adversarial attacks on both datasets without a defense mechanism. On the contrary, the F-EBA model exhibits superior performance, as demonstrated in Table 4, when tested against adversarial attacks on both datasets with a defense mechanism.

To gain a deeper understanding of the results, the decision boundaries and confidence scores generated by the model’s adversarial layer, employing both BiLSTM and BERT weak learners, were plotted across both datasets. The boundary plots are presented in Fig. 15. By examining Fig. 15, we can explore the model’s decision-making process in the adversarial layer without a defense mechanism, highlighting misclassified data points (depressed and non-depressed) with yellow dots found in purple areas and purple dots in yellow areas, ultimately leading to a decline in model performances.

The performance decline was due to new adversarial examples in the testing set. To address this concern, weak learners were strengthened/boosted by training them with FGSM-generated adversarial samples, using a standard perturbation of 0.5. Despite this, the practice leads to a doubling of the model’s training time, stemming from the large data size caused by the inclusion of adversarial examples in the training set. Upon adding the adversarial defense layer, the model’s decision boundaries shifted as predicted, with yellow dots aligning with yellow areas and purple dots aligning with purple areas. This led to an improved or higher-performing model, as demonstrated in Fig. 16.

Likewise, the confidence scores of the adversarial layer of the F-EBA model are displayed in Figs. 17 and 18, both with and without the defense mechanism, across both datasets for BERT and BiLSTM weak learners. The findings reveal a notable increase/shift in confidence scores from less than 0.6 to over 0.9 with the defense mechanism in Fig. 17, and from 0.7 to over 0.9 in Fig. 18. The uptick in confidence scores, notably surpassing 0.9, implies that the model demonstrates more confidence in its predictions after adding a defense layer. This boost in confidence suggests that the F-EBA model performs more reliably when applied to the specified dataset.

Confusion metrix

The confusion matrix evaluates the accuracy of the model’s predictions by analyzing confusion during the prediction process. The results of the confusion matrix are shown in Fig. 19.

The visualization results show that out of 6,265 data points before the adversarial layer, the model exhibited 310 instances of confusion in its predictions, reflecting a 5% confusion. With the implementation of the adversarial layer and analysis of 43,582 data points, 820 instances of misclassified data points were detected, reflecting a 5% confusion. This observation suggests that the model achieves higher accuracy and diminishes confusion by applying the adversarial layer.

ROC metrix

The ROC curve presents the true positive and false positive rates. As seen in Fig. 20, the ROC curve for the F-EBA model indicates that before the adversarial layer, the model curve approaches 0.95 in the true positive portion, whereas with the addition of the adversarial layer, it reaches 0.99. This highlights how the model’s performance notably improves with the addition of the adversarial layer.

Precision-recall curve

The PRC evaluates a model’s performance in imbalanced datasets by assessing its capability to retrieve positive instances (recall) while minimizing false positives (precision). In Fig. 21, the PRC curve demonstrates a crossing point at 0.96 before the adversarial layer, and by inclusion of the adversarial layer raises it to 0.99, signifying exceptional performances. This shift demonstrates an outstanding balance between capturing positives and limiting false positives, highlighting the significant impact of the adversarial layer on the model’s discriminatory capabilities.

Cross validate the model

Cross-validation evaluates a model’s predictive ability or performance on various portions of the dataset by training it on one portion of the dataset and testing on another, repeating this process for each portion of the entire dataset. The researcher carried out 10-fold cross-validation, testing various numbers of folds across different layers of the model and various portions of the datasets. As illustrated in Fig. 22, on the left side, the results of the cross-validation for the F-EBA model on both datasets exhibit that the blue line represents training and testing the model on the same data portion, displaying similar accuracy. The yellow line shows testing on portions of 20%, 40%, and 80% of the data, resulting in accuracies of 65%, 85%, and 95% respectively. Likewise, the inclusion of an adversarial layer in the F-EBA model yields noteworthy advancements in validation and testing accuracy across various portions of datasets. Notably, at portions of 0%, 20%, 40%, 60%, and 80%, the accuracy consistently rises, reaching 96.8%. This experiment highlights how varying the test portion size impacts model performance and offers valuable insights into the results verification and the model’s generalization capabilities.

Model computational complexity

The computational complexity of F-EBA stems in both pipelines:

• (1)

  Feature-engineering pipeline: BERT embeddings (O(n²) due to the self-attention mechanism), lightweight self-attention over Word2Vec (optimized to approximately O(n)), and recursive feature elimination (O(n × k), where n is the number of features and k is the number of iterations). The final decision tree classifier operates in O(n log n)

• (2)

  Classification pipeline:

The overall time complexity of our F-EBA model can be calculated as as:

$$\otimes (\mathcal{T}.\left(\mathcal{n}.\mathcal{d}+\mathcal{E}+\mathcal{B}\right)$$where:

• $\mathcal{T}$: Number of boosting iterations

• $\mathcal{n}$: Number of training samples

• $\mathcal{d}$: Feature dimensionality (from Word2Vec + BERT embeddings)

• $\mathcal{E}$: Time complexity of embedding generation (BiLSTM: O(n·l·h²), where l = sequence length, h = hidden size)

• $\mathcal{B}$: Time for BERT inference per instance (O(n·l²) due to self-attention)

Since F-EBA combines BiLSTM, and BERT as weak learners in a boosting loop, the model’s complexity is dominated by BERT’s attention mechanism and BiLSTM sequence processing within each boosting iteration.

To mitigate parameter uncertainty, sensitivity analysis was performed on key parameters, and cross-validation along with optimization techniques were applied, ensuring the F-EBA model remains robust and scalable despite parameter variations.

Statistical validations

To statistically validate the findings of the F-EBA model or model’s performance metrics such as confusion matrix, ROC metrics, precision-recall curves, and cross-validation results, the following hypotheses are designed to assess whether the obtained performances are statistically significant or not.

H1: The model’s performance metrics results are not significantly different from the baseline performances.

H2: The model’s performance metrics results are significantly different from and surpass the baseline performances.

To test these hypotheses, the statistically paired t-test has been performed using the following parameters through SPSS.

• (a)

  The model’s performance metrics results as follows

  • (1)

    Confusion metrics: The model achieved 95% accuracy

  • (2)

    ROC metrics: The ROC curves approach an AUC value of 0.99

  • (3)

    Precision-recall curve: The PRC curves approach an AUC value of 0.99

  • (4)

    Cross-validation results: The model was evaluated using different portions of the dataset (20%, 40%, and 80%), yielding accuracies of 65%, 85%, and 95%, respectively.

• (b)

  The baseline performance is derived from the average values of the state-of-the-art approaches from a comparative analysis of studies is presented in Table 7, which is approximately 80%

Using these parameters, paired t-tests were employed to compare the performance metrics against a baseline performance of 80%, which serves as a reasonable benchmark for effective performance that the F-EBA model should defeat. The significance level was set at α = 0.05. It indicates a 5% is the amount of error that the findings can tolerate. This threshold was selected to balance the risk of false positives with the necessity for reliable results, corresponding to a 95% confidence level, and to ensure that any observed differences were statistically significant. All metrics showed significant differences (p < 0.05) between the F-EBA model and the baseline performances, supported by confidence intervals that did not cross zero. Furthermore, effect size (Cohen’s d) was calculated to evaluate the magnitude of improvement. The effect sizes ranged from 0.8 to 1.2, indicating a large and meaningful improvement in the F-EBA model’s performances. These results suggest a clear rejection of the null hypothesis (H1) and strong support for the alternative hypothesis (H2), indicating that the F-EBA model significantly outperforms the baseline performances.

Table 5 represents a summary of the statistical hypothesis testing results.

The results in Table 5, p-value less the 0.05 threshold, indicating that the improvements are statistically significant and unlikely to have occurred by chance, thus reflecting meaningful enhancements in performance. Furthermore, Table 5 demonstrate that the F-EBA model significantly outperforms the baseline performance of 80% based on the findings from the model’s performance metrics such as confusion matrix accuracy, ROC curves, precision-recall curves, and cross-validation with the 80% data portion. However, the results are not statistically significant for the 20% and 40% data splits. This further supports the claim that the F-EBA model performs well with larger datasets. In addition, results with p-values less than 0.05 suggest significant. This confirms that the improvements are not due to random variation but are indeed meaningful.

Explainable AI validations

Normally, modern and state-of-the-art LLMs use different methods such as retrieval-augmented generation (RAG), external fact-checking, post-processing techniques, Explainable AI (XAI) techniques, etc., for validation of the findings. To this end, this study utilises Explainable AI, specifically SHAP values, to assess the model’s interpretability. SHAP values help verify whether the contributions of the processed feature set/words in a sentence align with the model’s predictions.

In this validation process, the researcher selected two feature values, “love” and “excellent,” and plotted their corresponding SHAP values, as shown in Fig. 23. This allows for a clearer comparison of how each feature contributes to the model’s predictions. It further illustrates the relationship between the SHAP value of the feature “love” and its actual value across all samples (each dot representing a sample). The x-axis represents the value of the feature “love,” while the y-axis displays its corresponding SHAP value, providing insight into how variations in the occurrence of “love” influence the model’s predictions. The upward trend indicates that higher occurrences of “love” lead to more positive predictions. In addition, the plot compares the effect of the word “Excellent” on “love,” revealing that “Excellent” has a stronger impact. This suggests that the presence of “Excellent” further amplifies the positive effect of “love” on the predictions. This generally confirms that the results and findings obtained from the F-EBA model are verified and reliable, as well as qualified.

Both comprehensive validation approaches (statistical and XAI validations) serve as statistical controls, tests, and evaluations of the model performances, reinforcing the validity of our results.