Deep learning-based method for sentiment analysis for patients’ drug reviews

Neural networks

Deep learning involves a multilayer approach to the hidden layers of neural networks (Dang, Moreno-García & De la Prieta, 2021). Traditionally, in machine learning models, features are identified and extracted either manually or using feature selection methods. However, in the case of deep learning (Ghulam et al., 2019), features are learned, and extracted automatically resulting in higher accuracy and performance. Figure 1 shows the differences in sentiment classification between the two approaches.

Recurrent neural networks

A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes can create a cycle, allowing output from some nodes to affect subsequent input to the same nodes as shown in Fig. 2 This unique architecture allows RNNs to exhibit temporal dynamic behavior.

Specifically designed for processing sequential data, RNNs play a crucial role in various applications. The fundamental equation governing the recurrent nature of these networks is expressed as follows: RNN are a part of the neural network family used for processing sequential data. In the following equation (1)${\displaystyle h^t=f\left(h^{t-1},x\right),}$ h^t represents the output at time t, h^(t−1) denotes the previous output, and x is the input. The function f captures the recurrent relationship, describing how the network processes and updates information over sequential steps.

Recurrent neural networks (long short-term memory)

The LSTM network is a type of RNN that includes additional memory features to address the challenges of vanishing and exploding gradients (Ghulam et al., 2019). LSTMs consist of recurrently connected blocks, known as memory units, that are designed to mitigate these issues. The memory units have gates, which determine whether to add or remove information from the cell state as shows in Fig. 3. By utilizing these gates, LSTMs can capture long-range dependencies in the input data, which is useful for text classification tasks. Overall, LSTMs offer an advanced approach to RNNs that can better handle complex and sequential data.

The architecture includes several gates and memory cells that enable it to capture and remember information over varying time scales (Tran et al., 2016). The LSTM has an input x_t which can be the output of a CNN or the input sequence directly. h_t−1 and c_t−1 are the inputs from the previous timestep LSTM. o_t is the output of the LSTM for this timestep. The LSTM also generates the c_t and h_t for the consumption of the next time step LSTM

• Forget gate: The forget gate determines what information from the previous cell state should be discarded (Kim et al., 2016). It takes the previous cell state C_t−1 and the current input x_t as inputs and produces a value between 0 and 1 for each element in the cell state (2)${\displaystyle f_t=\sigma \left(W_f\cdot \left[h_{t-1},x_t\right]+b_f\right).}$

• Input gate: The input gate controls how much new information should be added to the cell state. It calculates two values: the candidate values ${\tilde{C}}_t$ (new candidate cell state) and the input gate i_t which determines how much of this candidate value should be added. (3)${\displaystyle i_t=\sigma \left(W_i\cdot \left[h_{t-1},x_t\right]+b_i\right)}$ (4)${\displaystyle {\tilde{C}}_t=tanh\left(W_C\cdot \left[h_{t-1},x_t\right]+b_C\right).}$

• Update cell state: The new cell state C_t is calculated by combining the forget gate, the input gate, and the previous cell state. (5)${\displaystyle C_t=f_t\cdot C_{t-1}+i_t\cdot {\tilde{C}}_t.}$

• Output gate: The output gate determines the next hidden stateh_t and the output of the LSTM cell. It combines the current input x_t and the previous hidden state h_t−1 to calculate the output gate o_t and the output (6)${\displaystyle o_t=\sigma \left(W_o\cdot \left[h_{t-1},x_t\right]+b_o\right)}$ (7)${\displaystyle h_t=o_t\cdot tanh\left(C_t\right).}$

Bidirectional LSTM

A bidirectional LSTM, or Bi-LSTM, is a sequence processing model that consists of two LSTMs as shows in Fig. 4: one taking the input in a forward direction, and the other a backward direction. Bi-LSTMs effectively increase the amount of information available to the network, improving the context available to the algorithm (e.g., knowing what words immediately follow and precede a word in a sentence).

The architecture of bidirectional LSTM comprises of two unidirectional LSTMs which process the sequence in both forward and backward directions (Graves & Schmidhuber, 2005). This architecture can be interpreted as having two separate LSTM networks, one gets the sequence of tokens as it is while the other gets in the reverse order. Both of these LSTM networks return a probability vector as output, and the final output is derived from the combination of these probability vectors. It can be represented as: (8)${\displaystyle p_t-=p_t^f+p_t^b,}$ where

• p_t represents the final probability vector of the network.

• $p_t^f$ represents the probability vector from the forward LSTM network.

• $p_t^b$ represents the probability vector from the backward LSTM network.

Convolution neural networks

CNNs (Moskovitz, Roy & Pillow, 2018) are a type of deep neural network primarily used for analyzing visual imagery. Recently, text classification has been revolutionized by the advancements in deep learning particularly RNN and CNN. In fact, CNN for text classification is the new sexy for NLP practitioners in industry, research, and academia because of the efficient encoding of language semantics and context into mathematical vectors. In our model we proposed how the CNNs offer a powerful approach to analysis of visual data, leveraging their ability to learn and extract features from images. The architecture of a CNN can be deconstructed into two fundamental components as listed below and shows in Fig. 5:

• Convolutional layers: These layers are responsible for extracting features from the input data.

• Fully connected (dense) layers: After the convolutional layers identify relevant features, the fully connected layers utilize this extracted information to generate the final output.

Patients’ drug reviews datasets

Two relevant datasets were planned to be used for this research. Both are publicly available drug reviews.

Data cleaning and prepossessing

These stages were carried out after gaining a thorough understanding of the behavior and features of the selected dataset. It was observed that the user must have to write the drug’s name and intake quantity in the review which was not necessary for this research work. Therefore drugs’ names and quantity measurements were dropped. In addition, the condition and review text emerged in one file. Algorithm 1  explains the next steps which are text cleaning and preparation modules and have to be utilized as listed below:

• Remove any leading or trailing spaces (Sekihara et al., 2016). Removing characters or white spaces from the beginning or end of a string is the basic step to preprocessing the text in NLP which is achieved by using the strip function.

• Convert all the review’s text into lowercase using the lower() function.

• Substitute any occurrence of a new line character. To perform a global search and replace, use a regular expression with a space using the sub() function.

• Remove any punctuation and special characters from each word. If the input string consists of punctuation, then we have to make it punctuation-free. Let us say the input string is $Student@’ then we have to remove $ and @, furthermore, we have to print the plain string ‘Student’ which is free from any punctuation. More details can be found at GeeksforGeeks (2023).

• Remove English stop words (Shilpa et al., 2021), which are words that are very frequent in a language and are thus considered to not contain much information on the analyzed text such as “the”, “which”, “where”, “and”, “is”, “how”, and “who” from the text using the NLK.corpus and remove any word having less than three characters since such words do not provide much information.

• Lemmatize each token (Ingason et al., 2008): which entails reducing a word to its canonical or dictionary form. The root word is called a ‘lemma’.The method entails assembling the inflected parts of a word in a way that can be recognized as a single element. The process is similar to stemming but the root words have meaning.

• Stemming is a rules-based approach that produces variants of a root/base word (Lovins, 1968). In simple words, it reduces a base word to its stem word. This heuristic process is the simpler of the two as the process involves indiscriminate cutting of the ends of the words. Stemming helps to shorten the look-up time and normalize the sentences for a better understanding.

 
 
 
 
Algorithm 1 Clean Text Function 
  1:  function CLEAN_TEXT(text) 
  2:                                                                                                    ⊳ Remove Pre and Post Spaces 
  3:       text ← strip(text) 
  4:                                                                                                         ⊳ Lower case the entire text 
  5:       text ← to_lower(text) 
  6:                                                                            ⊳ Substitute New Line Characters with spaces 
  7:       text ← substitute(text) 
  8:                                                                                                               ⊳ Tokenize the sentence 
  9:       word_tokens ← tokenize(text) 
10:                                                                                                         ⊳ Initialize cleaned text list 
11:       cleaned_text ← [] 
12:                                                                              ⊳ Remove punctuation and special characters 
13:       for word in word_tokens do 
14:            cleaned_word ← ”” 
15:            for char in word do 
16:                  if IS_ALPHANUMERIC(char) then 
17:                       cleaned_word ← cleaned_word + char 
18:                  end if 
19:            end for 
20:            cleaned_text.append(cleaned_word) 
21:       end for 
22:                                                                                                        ⊳ Specify the stop words list 
23:       stop_words ← get_stopwords(′english′) 
24:                                                                                                           ⊳ Initialize text_tokens list 
25:       text_tokens ← [] 
26:                                               ⊳ Remove stopwords and words containing less than 2 characters 
27:       for word in cleaned_text do 
28:            if len(word) > 2 and word / ∈ stop_words then 
29:                  text_tokens.append(word) 
30:            end if 
31:       end for 
32:                                                                                      ⊳ Lemmatize each word in the word list 
33:       text ← lemmatize(text_tokens) 
34:       return text 
35:  end function 
_________________________________________________________________________________________________________________________    

Word embedding by using GloVe encoding

In this work, contextual encoding which is already trained on large text corpus using semantic encoding is utilized. The encoding GloVe will be used in the LSTM bidirectional model and in the hybrid model LSTM-CNN shows in Algorithm 2  and Fig. 7. GloVe stands for global vectors for word representation. It is an unsupervised learning algorithm developed by Stanford University (Pennington, Socher & Manning, 2014) for generating word embedding by aggregating a global word-word co-occurrence matrix from a corpus.

GloVe (Naili, Chaibi & Ghezala, 2017) is a widely used pre-training method in natural language processing (NLP) for generating word embedding, which are dense vectors that represent the semantic meaning of words. These embeddings are useful in various NLP tasks and can significantly improve the performance of deep learning models. By incorporating GloVe embedding as input in CNN, RNN, and LSTM models, the semantic meaning of words can be captured, leading to accurate predictions (Mitchell et al., 2008). Therefore, using GloVe embedding in pertaining deep learning models can be a powerful technique to improve the performance of NLP tasks. The GloVe model is trained with a massive corpus and uses it for the mathematical conversion of the input text.

 
________________________________________________________________________________________________________________________ 
 
Algorithm 2 Create Embedding Matrix Function 
  1:  function CREATE_EMBEDDING_MATRIX(FP,WI,ED) 
  2:                                                                            ⊳ File path, word index, embedding dimension 
  3:       FP ← ”GloVe.6B.300d”                                                                         ⊳ File path 
  4:       WI ← word_index                                                                         ⊳ Word index 
  5:       ED ← embedding_dim                                                       ⊳ Embedding dimension 
  6:                                                                                                                        ⊳ Vocabulary size 
  7:       vocab_size ← length(word_index) + 1 
  8:                                                                               ⊳ Initialize the embedding matrix with zeros 
  9:       embedding_matrix ← zeros((vocab_size,embedding_dim)) 
10:                                                                                                           ⊳ Open the embedding file 
11:       with open(file_path,encoding=’utf-8’) as file: 
12:       for every line in file: do 
13:                                                                         ⊳ Extract the word and its corresponding vector 
14:            word,∗vector ← line.split() 
15:            if the word is in word_index: then 
16:                                                                                                                   ⊳ Get the word index 
17:                  IDX ← word_index[word] 
18:                                                                                                  ⊳ Extract the embedding vector 
19:                  embedding_matrix[IDX] ← array(vector)[:embedding_dim] 
20:            end if 
21:       end for 
22:       return embedding_matrix 
23:  end function 
________________________________________________________________________________________________________________________    

BiLSTM-CNN

Figure 8 shows the detailed flow diagram of all the steps within the modeling phase.

• The review underwent pre-training and embedding using GloVe vectorization to generate the embedding matrix, which was then fed into a dedicated convolution layer.

• Bi-LSTM: The output of the max pooling layer is then passed through a bidirectional LSTM layer. Hence, can be used to understand the long-distance relationship between the regions and predict the ratings effectively.

• A convolutional layer is added to extract features from the embedded sequences.

• Max pooling layer is added to reduce dimensionality.

• Finally, a dense layer with softmax activation is added to predict the sentiment class of the input sequence.

Bidirectional LSTM model

As shown in Fig. 9 a bidirectional LSTM layer is added to capture the sequential dependencies in the data. Finally, the output layer is added with three units, one for each class in the classification task. The softmax activation function is used to output class probabilities.

The architecture of the model

For both Model A and Model B, the Bi-LSTM and Bi-LSTM-CNN architectures are applied as explained below:

• The model starts with the embedding layer which creates a dense vector representation of each word in the input sequence. It has three arguments the size of the vocabulary, output dimensions, and the length of the input sequence.

• The second layer is the SpatialDropout1D layer which helps prevent over-fitting.

• A bidirectional LSTM layer was applied enabling the input sequence to be applied in both forward and backward directions. This bidirectional processing allows the model to glean insights from the entire context, capturing dependencies not only from preceding words but also from subsequent ones. Consequently, the bidirectional LSTM layer enables the model to discern complex patterns and nuances in drug reviews, facilitating a more nuanced sentiment analysis. The inclusion of a SpatialDropout1D layer is crucial for preventing overfitting, enhancing the model’s generalization capabilities by selectively dropping entire 1D feature maps during training

• The dense layer applies a linear transformation to the output of the previous layer, followed by a softmax activation function. The softmax function normalizes the output to a probability distribution over the three classes. The concluding Dense layer, coupled with a softmax activation function, serves as the ultimate decision-making layer. It transforms the output from preceding layers into a probability distribution across the three sentiment classes

• The method configures the learning process for the model by compile processing which was used in programming contexts. The loss argument specifies the loss function to optimize, the optimizer argument specifies the optimization algorithm and the metrics argument specifies the evaluation metric(s) to use during training and testing.

The bidirectional LSTM layer in the Bi-LSTM-CNN architecture maintains its role in considering both past and future context, synergizing with the convolutional layer’s local feature extraction. This combination proves particularly effective in capturing intricate nuances in the sentiment expressed within drug reviews. Similar to the Bi-LSTM architecture, the Dense layer, and compile processing are integral components for transforming learned features into actionable predictions, ensuring optimal learning and evaluation processes. In summary, the Bi-LSTM-CNN architecture amalgamates the strengths of both LSTM and CNN components, providing a holistic approach to sentiment analysis in drug reviews.

Architecture A and B Bi-LSTM-CNN

The architecture of Models A and B Bi-LSTM-CNN are the followings:

• The model starts with the embedding layer which creates a dense vector representation of each word in the input sequence. it has three arguments the size of the vocabulary, output dimensions, and the length of the input sequence).

• In the second reiteration, the arguments for the Embedding layer will be four because adding the weights parameter specifies the pre-trained word embedding by using GloVe encoding GloVe.6B.300d.

• The second layer is added as a 1D convolutional layer with 64 filters and a filter size of 5. The activation function used is ReLU (rectified linear unit).

• Max pooling layer with a pool size of 4 was applied to reduce the dimensionality of the data.

• A bidirectional LSTM layer was applied with 64 units, a dropout rate of 0.1, and a recurrent dropout rate of 0.3. The bidirectional LSTM allows the model to consider both past and future context in the input data.

• The dense layer applies a linear transformation to the output of the previous layer, followed by a softmax activation function. The softmax function normalizes the output to a probability distribution over the three classes

• The method configures the learning process for the model by compile processing. The loss argument specifies the loss function to optimize, the optimizer argument specifies the optimization algorithm and the metrics argument specifies the evaluation metric(s) to use during training and testing.

Bi-LSTM without GloVe embedding

Based on the evaluation metrics provided in Table 5 above, and Figs. 10 and 11 parts a and b, respectively, it seems that Model A without GloVe outperforms Model B without GloVe in terms of precision, recall, F1-score, and accuracy for all three classes. The precision, recall, and F1-score for the neutral class are 0 for Model A without GloVe, indicating that the model does not predict this class well. However, Model B without GloVe has a higher precision, recall, and F1-score for the Neutral class, which means it can predict this class better than Model A without GloVe. The overall accuracy for Model A without GloVe is 0.847564, which is higher than the accuracy of Model B without GloVe at 0.744680. Additionally, the Cohen-score for Model A without GloVe is 0.660347, which is higher than the Cohen-score for Model B without GloVe at 0.546443. Therefore, we can conclude that Model A without GloVe performs better than Model B without GloVe in terms of overall sentiment classification for drug reviews.

Bi-LSTM-CNN without GloVe embedding

Overall, Model A outperformed Model B on all evaluation metrics, indicating that Model A is a better-performing model. As show in Table 6 and Figs. 12, 13 parts a and b, respectively. Model A performed well in all classes, achieving high precision, recall, and F1-scores on all sentiment classes, indicating that the model is performing well. The accuracy score for Model A is also higher, indicating that the model is performing better overall. Cohen’s kappa scores for both models are relatively high, indicating that the models have a high level of agreement with the actual sentiment labels. Overall, depending on the result, the model Bi-LSTM-CNN achieved height accuracy on both models with GloVe embedding and without embedding than Bi-LSTM, and the Cohen score is perfect.

Bi-LSTM with GloVe embedding

From the evaluation results in Table 7 above and Figs. 14, 15 parts a and b, respectively, it can be seen that the models with GloVe embeddings (Model B) outperformed the models without GloVe embeddings (Model A) in terms of precision, recall, and F1-score. However, there is a trade-off between precision and recall for the negative class, as the models with GloVe embeddings had higher precision but lower recall compared to the models without GloVe embeddings. The neutral class had poor performance overall, with F1-scores close to zero, while the positive class had the best performance. The accuracy of the models was similar both with and without GloVe embeddings. The Cohen score was higher for Model B, which indicates better inter-rater agreement. Overall, the results suggest that using GloVe embeddings can improve the performance of sentiment classification models on drug reviews from patients.

Bi-LSTM-CNN with GloVe embedding

Based on the evaluation metrics in Table 8 and Figs. 16 and 17 parts a and b respectively, it can see the LSTM-CNN algorithm with GloVe embeddings performed better on Model A than on Model B. The LSTM-CNN algorithm with GloVe embeddings performed well on both models, with higher scores on Model A compared to Model B. The model achieved high precision, recall, and F1 scores on all sentiment classes in Model A, indicating that the model is performing well in all classes. However, the performance Model B was lower for the neutral sentiment class, indicating that the model is having some difficulty correctly classifying neutral sentiment. The accuracy scores for both models are relatively high, indicating that the model is performing well overall. The Cohen score was higher for Model B, which indicates better inter-rater agreement.