An Al-BERT-Bi-GRU-LDA algorithm for negative sentiment analysis on Bilibili comments

Pre-trained model

Most researchers (Szegedy et al., 2017; Wu, Shen & Van DenHengel, 2019; Singh, Hoiem & Forsyth, 2016) have confirmed through experimental studies that using pretraining technology can significantly improve the effect of downstream tasks. The early pretraining technique was a static encoding technique. In 2003, Bengio incorporated the idea of deep learning into the language model and proposed the NNLM (Bengio, Ducharme & Vincent, 2000) for a pretraining model.

In 2013, Word2Vec borrowed the idea from NNLM and employed a language model to construct word vectors. This was followed by Global Vectors (GloVe) (Pennington, Socher & Manning, 2014) and FastText (Joulin et al., 2016). Word2Vec and GloVe are the most widely used language models in natural language processing (NLP). Many studies have shown (Al-Amin, Islam & Uzzal, 2017; Tang et al., 2015; Grover & Leskovec, 2016) that the Word2Vec and GloVe models can significantly improve text vectorization’s representation effect in NLP tasks. Therefore, the vectorized representation of a text is fixed, and the model cannot effectively combine the text context information of the context globally. To resolve the problem of polysemy and word ambiguity and capture more useful deep semantic information, In 2018, ELMo (Sarzynska-Wawer et al., 2021) was proposed as a text representation method that could combine context. Since then, the GPT model (Radford et al., 2018), BERT algorithm (Devlin et al., 2018), etc., have been suggested.

Moreover, it outperforms other pre-trained language algorithms in many typical tasks in NLP, which becomes an essential milestone in NLP. Our work uses the Al-BERT algorithm as the basic pretraining model. It obtains rich dynamic encoded semantic information through the Al-BERT algorithm to pave the way for deep text sentiment classification to extract text features better.

Sentiment classification approaches

Al-BERT layer

The BERT model was suggested by Google in 2018 and achieved good results by stacking the Transformer substructure as a feature extractor and implementing the self-attention mechanism to encode words directly. However, when the model is expanded to a particular scale, the model training will take up excellent memory usage, the training time will be so long, and the model will degenerate and cause overfitting problems. To resolve these issues, scholars such as Lan et al. (2019) proposed a lightweight version of the BERT model, namely, Al-BERT (A Lite BERT), by employing the factorized embedding parameterization method and the cross-layer parameter sharing approach to cause the amount of the model’s parameter significantly reduced. It performs better on NLP tasks while reducing the number of parameters. Therefore, this study uses Al-BERT as a pretraining model. In the vectorized representation of text generated by Al-BERT, the vector corresponding to input in the algorithm has three superimposed vectors. It is called the Token Embedding Segment.

Embedding and Position Embedding. By converting each word into matrix–vector forms of Q, K, and V through different linear transformations and calculating independently of each other, a unique vector encoding for each term is obtained so that it can have a deeper perception of the context (Liu et al., 2020). Equation (1) presents the self-attention calculation. (1)${\displaystyle \text{Attention}\left(Q,K,V\right)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)\cdot V}$ Q, K, and V denote the query, key, and value matrix, respectively. The research implements the trained Al-BERT algorithm, which consists of 12 layers of Transformer. The embedding dimension of the word vector is assigned to 768. To ensure that the vector input to the Al-BERT algorithm is easy to operate, this research sets the length of the input sequence to 512. Only the part within the maximum range is reserved for sentences exceeding the maximum length, and < padding > is used to fill in the insufficient length sequence.

Bi-GRU layer

The GRU maintains the same effect as the LSTM and combines the input gate and the forget gate in LSTM into one gate, called the update gate, so the algorithm only consists of the reset gate and the update gate. Figure 2 shows the GRU structure. The parameters are more simplified and have better convergence. Equation (2) presents the update gate. (2)${\displaystyle Z_t=\sigma \left(W_z{S}_t+U_z{H}_{t-1}+B_z\right)}$ where W_z and U_z denote weights, and H_t−1 represents the input and B_z denotes the bias.

Then, GRU calculates the candidate content that needs to be retained. W and U denote weighted, and B represents the bias: (3)${\displaystyle {\hat{H}}_t=tanh\left(W{S}_t+U{R}_t{H}_{t-1}+B\right).}$

Finally, the GRU calculates the final output based on the above-obtained results: (4)${\displaystyle H_t=\left(1-Z_t\right){\hat{H}}_t+Z_t{H}_{t-1}.}$

In the Bi-GRU layer, the emotional features are extracted in the forward and backward GRU networks, which can capture the global features of the context. The two reverse GRUs jointly determine the output of the Bi-GRU network.

Attention layer

As one of the most substantial concepts in deep learning, the attention mechanism is a weight allocation method that simulates the allocation mechanism of human brain attention. At a particular moment, the human brain will focus on a specific area that needs attention, reducing or ignoring other regions to grasp more valuable information (Niu, Zhong & Yu, 2021). The Attention mechanism assigns more significant weight to crucial details based on probability distribution so that the algorithm can pay more attention to this type of information. In short, an attention mechanism-based text classification can amplify core keywords’ impact on text features. As shown in Fig. 1, The result of the weight distribution will affect the final classification outcome (Liu et al., 2019). Equations (5) and (6) present attention allocation. (5)${\displaystyle C_t=\sum _{t=1}^{T_h}{a}_{ij}{H}_t}$ (6)${\displaystyle a_{ij}=\text{softmax}\left(W_{a2}tanh\left(W_{a1}{H}_t\right)\right)}$ where a_ij denotes attention weight, H_t represents the length of the data, W_a1 and W_a2 designate weights.

Experimental procedure

• Data collection and pre-processing: First, we utilized crawler technology to collect the comments written by ten high school mathematics courses that high school students participated in on the Bilibili platform. The total number of texts is 20,000 in Chinese. Furthermore, the collected corpus is pre-processed to remove irrelevant comments, meaningless URLs, and non-Chinese comments (including comments with only symbols and English characters). A total of 12,655 usable comments are obtained.

• Labeling: The comments with obvious positive adjectives and adverbs are marked as positive, sentences without adjectives are marked as unfavorable, and adjectives or adverbs with negative emotional color and interrogative sentences are marked as negative. Through iterative processing, 6,929 positive text comments, 3,686 neutral comments, and 2,042 negative comments were obtained. Since the number of features has a particular impact on the precision analysis, “more is better” does not guarantee better outcomes. Therefore, the feature selection process must consider both efficiency and precision (Zheng, Wang & Gao, 2018). Therefore, 2,500 positive reviews were selected randomly. The distribution of the data set is shown in Table 1.

• Word representation: We use the trained Chinese model to vectorize Chinese text comments, and the processed text vector has 768 dimensions.

• Building deep sentiment classification model: Al-BERT-Bi-GRU-Att+LDA: The bi-GRU model with attention mechanism based on the proposed Al-BERT pretraining algorithm. In the Bi-GRU layer, the input dimension is consistent with the dimensions of the Al-BERT layer, and the hidden layer dimension is assigned to 256. The input sentence length is assigned to 100. In the attention layer, different weights are assigned to each word through the attention mechanism, and topic clustering is performed on the comments classified as negative.

• Optimizing model: During the training stage, the hyperparameters are modified to optimize the final model’s performance.

• Model evaluation: We evaluate the prediction results with precision, recall, and F1 score.

• Topic clustering After the model obtains the final classification result; the LDA topic model is used for topic clustering for the comments whose classification results are negative. Thus, the problems that learners are concerned about can be detected and the course developers can present solutions.

Evaluation criteria

Evaluating the model performance is an essential link, and different evaluation indicators comprehensively reflect the quality of the model’s performance. The model evaluation indicators used in the article are precision, precision, recall, and the F1-scores, consistent with most studies’ indicators. They are presented in Table 2. (7)${\displaystyle precision=\frac{TP+TN}{TP+TN+FP+FN}}$ (8)${\displaystyle \text{Precision}=\frac{TP}{TP+FP}}$ (9)${\displaystyle \text{Recall}=\frac{TP}{TP+FN}}$ (10)${\displaystyle F1=\frac{2\times Precision\times Recall}{\text{Precision}+\text{Recall}}.}$

Parameter setting

The experiment’s data set was 5-fold cross-validated to obtain accurate experimental results, and the samples were randomly divided into five groups. One group was used for training, and the remaining four were used for the validation and test sets. The research compares the performance of LSTM, Bi-LSTM, Bi-LSTM-Att, Text-CNN, and Al-BERT-Bi-GRU-Att algorithms regarding precision and loss value, respectively. The hyperparameters of the proposed algorithm are shown in Table 3:

Baseline methods

To verify the actual performance of the proposed algorithm, baseline text classification models are implemented as controls.

LSTM: The input dimension is assigned to 300, and the number of hidden layer dimensions is 300.

Bi-LSTM: the input dimension is word vector size, and the number of hidden layer dimensions is set to 300

Bi-LSTM-Att: The hidden layer dimension is assigned to 300 and combined with an attention mechanism to achieve accurate weight allocation.

Text-CNN: The filter length is 300, and the widths are set to 3,4 and 5, respectively.

Findings

Model comparison

The precision and loss values of the algorithm on the dataset are shown in Fig. 3. According to the precision rate curve, the precision rate of LSTM is lower than that of other classification algorithms. By the 10th training cycle, the precision rate has just reached 0.62 in the first training cycle of other algorithms. Text-CNN reached the highest precision rate of 0.86 in the 10th training cycle, and then the precision rate stabilized at 0.86 and fluctuated back and forth. The overall change in the precision of Bi-LSTM and Bi-LSTM-Att is relatively small, reaching the same level in the tenth training cycle. Both are 0.85, which is slightly lower than the Text-CNN model. Compared with the baseline algorithm, the proposed algorithm reaches the highest level of 0.93 in the 10th training cycle. It employs Al-BERT as the pretraining model, which reduces the risk of overfitting and the computational overhead, which is better than the baseline algorithm.

According to the loss rate curve, the loss value of LSTM is the largest among all deep learning algorithms. Compared with the LSTM algorithms, the loss rate of Bi-LSTM decreases significantly, and it has better convergence. When compared with the Bi-LSTM algorithm, the precision and loss rates of Bi-LSTM-Att do not considerably change the precision and loss rate scores. Compared with other types of neural networks, Text-CNN performs well overall because the convolutional layer is good at extracting local features to handle classification tasks better.

The proposed model has undergone 5-fold cross-validation, and the loss value is reduced to about 6% in the final stage. Compared with other experimental algorithms, the precision of the proposed algorithm has increased by at least 5%, and the loss rate dropped by at least 3%. Experiments show that Al-BERT-Bi-GRU-Att-Softmax can enhance feature extraction capability and improve classification precision.

Discussion

According to the statistical results of negative comments under each topic, the following conclusions are attained: First, most online learners generally reflect that the course content is challenging to understand, and the verbal expressions such as “difficult,” “complex”, and “do not understand” appear the most frequently, generally reflecting that the course content is hard to understand. That is mainly related to the nature of the mathematics itself. Some students also discuss questions about mathematics knowledge in the comment area. Students generally feel anxious about learning mathematical subjects and the exams they take. Secondly, there is a lot of feedback on bullet chatting. Many students complain that the content of bullet chatting is very messy and chaotic, and most of them have nothing to do with the course content, which affects their learning success. Some students say that the video screen suddenly turned black, they could not watch it, and the video sound was low.

Moreover, some students reported that more than the video content was needed and hoped to increase the range. Students have less negative feedback about teachers regarding teacher attributes since high teaching quality is available in online courses. Participants also reported that the order of the courses needed to be corrected. Regarding resource acquisition, a few students noted that courseware and handouts could not be downloaded because links expired.