Learning label smoothing for text classification

Label smoothing

As previously discussed, LS has boosted the performance of computer vision (Xu et al., 2020) and natural language processing tasks (Lukasik et al., 2020). Unlike LS, the unigram label smoothing developed by Pereyra et al. (2017) assigns the frequency of each label as the prior distribution rather than the uniform distribution. Both share a fixed prior-label distribution, which may not be satisfied by numerous complex tasks. To fill this gap, substantial advancements have been made in adaptive LS, which can be divided into two categories.

a) Revising the uniform distribution of LS. Incorporating the idea of the k-nearest neighbor algorithm, Bahri & Jiang (2021) assigned weights between the uniform distribution and the number of correct samples within radius $K$ divided by the total amount of samples. Penha & Hauff (2021) replaced non-target labels with negative sampler scores. Song et al. (2020) selected candidate words that shared the history of the previous step, thereby redefining and calculating the probability distribution of candidate words as a smoothing distribution based on context. However, these methods either require a specific model structure or are only performed for specific tasks, posing challenges for text classification tasks. In contrast, our method can be applied to any model. For image processing, Zhang et al. (2021) improved the loss function by accumulating the distributions of correctly classified labels to enhance image recognition; however, this method ignores the adjusted effect of negative samples on the models.

b) Changing the smoothing factor. Krothapalli & Abbott (2020) chopped images by considering the relative sizes of the objects in the training set. Li, Dasarathy & Berisha (2020) performed clustering on the training data and learned the smoothing intensity of each cluster. Wei et al. (2022a) proposed the use of a negative smoothing factor in high-noise regimes.

Our method falls into the first category, as we determine a more natural label distribution in the training process.

Calibration

Calibration predicts the probability or confidence in the model to approximate its true accuracy. The calibrated probability is important for interpreting the model (Guo et al., 2017) because it reflects the confidence level in an actual scenario. Efforts aimed at estimating calibration in well-trained models are mainly divided into two classes: post-processing and model calibration. Some classic binary models that use post-processing steps include Platt scaling (Platt, 1999), histogram binning (Zadrozny & Elkan, 2001), and isotonic regression (Zadrozny & Elkan, 2002). For multiclass settings, temperature scaling is a competitive calibration method (Guo et al., 2017; Balanya, Maroñas & Ramos, 2022; Khan, Wang & Liu, 2023) prevalent in knowledge distillation (Hinton, Vinyals & Dean, 2015). LogitNorm (Wei et al., 2022b) optimizes the logit vector as a unit vector with a constant magnitude. Model calibration introduces calibration terms for loss (Kumar, Sarawagi & Jain, 2018; Mukhoti et al., 2020), LS (Szegedy et al., 2016; Wang et al., 2021b), and data augmentation (Thulasidasan et al., 2019; Yun et al., 2019). Pereyra et al. (2017) were the first to propose LS for model calibration. Müller, Kornblith & Hinton (2019) conducted an in-depth study on LS calibration. The principle of LS is to increase the entropy of the output probability distribution to alleviate the problem of overconfidence. We also investigated the calibration effects of the proposed method.

Label regularization

Label-correction techniques that consider label quality have been developed to prevent mistakes in handcrafted labeling. Bootstrapping loss was proposed by Reed et al. (2014), which involves weighting the real labels with a predicted probability to reduce the influence of noise on parameter updating. Another approach described by Ma et al. (2018) decreases the weight of the hard labels over time. Arazo et al. (2019) integrated the concept of bootstrapping loss with dynamic weight adjustment, updating the loss of normal and noisy samples in opposite directions. Other regularization methods are employed at the loss level. For instance, Patrini et al. (2017) introduced a matrix $T$ to estimate the transition probability from real to noisy labels, proposing forward and backward losses based on $T$ to optimize real labels. In DisturbLabel (Xie et al., 2016), a few samples are randomly selected and trained using incorrect labels during each iteration. Similarly, the proposed method utilizes loss-function augmentation, which enables more flexible operations to adaptively adjust to the target distribution.

Label smoothing

Let $D={\left\{\left(x_i,y_i\right)\right\}}_{i=1}^N$, where $x_i$ denotes the $i$-th document; $Y=\left\{y_i\in {\left\{0,1\right\}}^K\right\}$, where $K$ is the number of document category. When $x_i$ is fed into the deep neural network, the model outputs a K-dimensional representation. The softmax function is used in the output layer of the neural network models to predict probability $p\left(k|x_i\right)$ for class $k$. The output distribution of the model is denoted by $p$. The standard cross-entropy (CE) loss function can then be written as

(1) $${\mathcal{L}}_{hard}=H\left(q,p\right)=-\underset{k=1}{\sum ^K}q\left(k|x_i\right)\log \left(p\left(k|x_i\right)\right)$$where $q$ is the ground-truth label, which is typically a one-hot distribution; $q\left(k|x_i\right)$ is marked as 1 if and only if category $k$ is the target class and 0 otherwise. Following this, we use the backbone to denote the model with CE. However, LS does not use a one-hot distribution to calculate the loss, introducing the noise distribution $u\left(k|x_i\right)instead$ instead. Thus, the ground-truth label becomes

(2) $$q^{\mathrm{\prime }}\left(k|x_i\right)=\left(1-\epsilon \right)q\left(k|x_i\right)+\epsilon u\left(k|x_i\right)$$and the loss is changed to

(3) $${\mathcal{L}}^{\mathrm{\prime }}=\left(1-\epsilon \right)H\left(q,p\right)+\epsilon H\left(u,p\right)$$where $\epsilon$ is the smoothing factor. The loss function comprises two parts: 1) CE between the one-hot distribution and the predicted distribution $H\left(q,p\right)$ and 2) CE between the noise distribution and the predicted distribution $H\left(u,p\right)$.

During the training process, if a machine learning model becomes overconfident in its predictions, $H\left(q,p\right)$ approaches 0, whereas $H\left(u,p\right)$ increases significantly. This implies that LS introduces a regularizing effect, $H\left(u,p\right)$, to prevent overconfident model predictions.

In traditional LS, $u\left(k|x\right)$ follows a uniform distribution, that is, $u={\displaystyle \frac{1}{K}}$. The loss function is expressed as follows:

(4) $${\mathcal{L}}_{ls}=-\underset{k=1}{\sum ^K}\left[\left(1-\epsilon \right)q\left(k|x_i\right)+{\displaystyle \frac{\epsilon }{K}}\right]\cdot \log \left(p\left(k|x_i\right)\right)$$where $\epsilon$ is usually set to 0.1 in LS. When $\epsilon =0$, this is equivalent to calculating CE using hard labels.

However, $u$ is independent of the data: $u\left(k|x\right)=u\left(k\right)$. Hence, the uniform distribution is questioned when applying the same probability distribution to incorrect labels. We assume that the label distribution correlates with the similarity between categories. One way to reduce the loss and optimize model performance is to reduce $H\left(u,p\right)$, specifically by making the $u$ distribution as close as possible to the predicted distribution. We posit that an iterative approach for updating soft labels is more reasonable than using fixed values, as inspired by Zhang et al. (2021) and Zhou et al. (2023). Therefore, we designed a DALS method based on this strategy. DALS considers the real relationships between different categories and uses predictions to extract inter-class relationships that are more discriminative for the model.

Discrimination-aware label smoothing

DALS uses category correlation in model prediction and dynamically updates soft labels during the training stage. The soft-label distribution is $u\left(k|x\right)$, which differs for each epoch. The model is supervised by the soft label calculated in the previous epoch, and the soft label is updated at the end of the current epoch. For a one-hot distribution, the probabilities for all classes are 0 except for the target class, which is marked with a probability of 1. Traditional LS employs a uniform distribution for non-target classes and reduces the probability of the target class to slightly below 1. In contrast, DALS discards the uniform distribution for the assignment of non-target classes and adaptively adjusts the label distribution.

We define $q_{x_i,k}^t$ as the soft distribution of class $k$ in the $t$-th epoch, where these distributions are specified for $x_i$. The soft label $q_{x_i,k}^{t-1}$ calculated in the ( $t$−1)-th epoch will be used to guide the training process of the $t$-th epoch. The training loss at this time can be represented as

(5) $${\mathcal{L}}_{soft}^t=-\underset{k=1}{\sum ^K}{q}_{x_i,k}^{t-1}\cdot \log \left(p\left(k|x_i\right)\right)$$

Figure 1 illustrates the overall framework. Text $x_i$ can be classified using any classification backbone, such as TextCNN (Kim, 2014) or BertGCN (Lin et al., 2021). The logits from the last layer are denoted as $Z$. The predicted score $p\left(x_i\right)$ is then obtained using a softmax layer. The $p\left(x_i\right)$ scores of the true positive and false positive samples are accumulated separately.

Specifically, we denote the set of samples with the prediction class $c=\underset{k}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}p\left(k|x_i\right)$ as $D$. We also define a true positive set called $D_{+}$ and a false positive set called $D_{-}$. At the end of each epoch, the accumulated class distribution is processed to balance the score contributions of each sample. The following equations are used for the calculation:

(6) $$\{\begin{array}{c}{\overline{q_{+}}}_{x_i,k}^t={\displaystyle \frac{1}{\left|D\right|}\sum _{x_iϵD_{+}}p\left(k|x_i\right)}\\ {\overline{q_{-}}}_{x_i,k}^t={\displaystyle \frac{1}{\left|D\right|}\sum _{x_iϵD_{-}}p\left(k|x_i\right)}\end{array}$$where ${\overline{q_{+}}}_{x_i,k}^t$ is the soft label of class $k$ calculated using true positive samples corresponding to sample $x_i$. Similarly, ${\overline{q_{-}}}_{x_i,k}^t$ is computed using false positive samples. Our soft label $q$ is calculated in two parts: $\overline{q_{+}}$ and $\overline{q_{-}}$. A truncation value of 0 is set to limit the impact of false positive distributions on the overall values. We define

(7) $$q_{x_i,k}^t=max\left(0,\lambda {\overline{q_{+}}}_{x_i,k}^t-{\overline{q_{-}}}_{x_i,k}^t\right)$$where $\lambda$ is a hyperparameter to balance the impact of true positive and false positive cases.

By adding hard label supervision, the updated training loss is changed to

(8) $${\mathcal{L}}_{all}=\left(1-\alpha \right)\cdot {\mathcal{L}}_{soft}^t+\alpha \cdot {\mathcal{L}}_{hard}$$where $\alpha$ determines the trade-off between soft and hard losses, and the value of $\alpha$ represents the confidence in the hard label. When $\alpha$ equals 1, it is equivalent to calculating with the hard label. We define ${\overline{q_{+}}}^0$ as a uniform distribution, and ${\overline{q_{-}}}^0=0$ because the soft label in the 0-th epoch is unavailable, according to Eq. (5). Thus, ${\hat{q}}^0\triangleq {\displaystyle \frac{1}{K}I}$, where $I$ denotes the identity matrix. In the early stage; this model is equivalent to using traditional LS.

True positive samples enable the model to generalize, allowing it to identify documents that were previously misclassified because the correct class was similar to the other classes. However, for categories with low precision, the model carries the risk of underfitting and an increased error rate. Thus, more generalization capabilities are not urgently needed. In particular, for these classes, we need to reduce the impact of true positive sample-based generalization on the model and implement a more rigorous loss assessment. Subtracting the value of ${\overline{q_{-}}}_{x_i,k}^t$ weakens the regularization degree and reduces the blurring degree of the boundary between classes, making the class boundary clearer.

Thus, we propose DALS to reduce the underfitting caused by excessive regularization.

Datasets

The datasets included 20-Newsgroups (20NG), R8 and R52 in Reuters 21,578, Ohsumed, and a movie review (MR) (Table 1).

20NG has 18,846 news documents, of which 113,134 and 7,532 were used for the training and test sets, respectively, which were classified with 20 labels.

R8 and R52, extracted from Reuters 21,578, have eight and 52 categories, respectively. R8 was divided into 5,485 documents for training and 2,189 documents for testing, whereas R52 was split into 6,532 training documents and 2,568 testing documents.

The Ohsumed Corpus comes from the MEDLINE database, which contains bibliographies of medical literature and has been processed to retain only 7,400 documents belonging to a single category. There were 3,357 documents in the training set and 4,043 documents in the test set, which were divided into 23 classes.

MR (Pang & Lee, 2005) is a short-text dataset of film reviews containing one sentence for each document and is mainly used for dichotomous emotional classification. There were 5,331 positive and 5,331 negative comments.

Baselines

The various models chosen for the baselines are listed as follows:

TextCNN (Kim, 2014) automatically combines and filters n-gram features to obtain high-level semantic information.

LSTM (Hochreiter & Schmidhuber, 1997) is a special form of recurrent neural network. The hidden state in the final step is used to represent the entire text.

FastText (Joulin et al., 2017), wherein the word vector and average n-gram vector are regarded as the document embedding.

TextGCN (Yao, Mao & Luo, 2019) constructs the entire corpus as a heterogeneous word-document graph, whereby the document classification problem is transformed into node classification.

SGC (Wu et al., 2019) reduces complexity by removing nonlinearities between the GCN layers, thereby collapsing the function into a linear transformation.

TensorGCN (Liu et al., 2020) constructs a text-graph tensor to describe semantic, syntactic, and sequential contextual information. Intragraph and intergraph propagations were conducted.

BERT (Kenton & Toutanova, 2019) and its variant RoBERTa (Liu et al., 2019): BERT refers to the bidirectional encoder representations from transformers that create numerous state-of-the-art models. RoBERTa is a robust, optimized BERT pre-training method.

BertGCN also builds a heterogeneous graph in which the document nodes are initialized with a pre-trained Bert. Subsequently, they are jointly trained with Bert and GCN for text classification. RoBERTaGCN, BertGAT, and RoBERTaGAT share this concept.

Experimental setup

Five models were selected for topic classification and sentiment analysis: TextCNN, LSTM, FastText, TextGCN, and BertGCN. For TextCNN, three types of kernels with sizes of two, three, and four were set, and the number of kernels for each type was 100. For LSTM, we chose a hidden layer size of 64. In BertGCN, the [CLS] token of the output feature was treated as the document embedding. The Bert-base-uncased model from HuggingFace (https://huggingface.co/bert-base-uncased) was used following (Lin et al., 2021), randomly dividing 10% of the training data for validation. All models used the Adam (Kingma & Ba, 2015) optimizer and adopted 300-dimensional GloVe word embeddings (Pennington, Socher & Manning, 2014). The main parameters included the number of epochs, batch size, learning rate, early stopping, $\alpha$, and $\lambda$. Early stopping indicates that the training process is terminated in advance if the performance of the validation set does not improve within a certain number of steps. Table 2 lists the parameter configurations of different models selected for comparison. We retained the default parameters in the original methods, setting $\lambda =1.4$ and $\alpha =0.96$ as moderate choices for the experiments. The performance was enhanced through further tuning. The models were trained using an NVIDIA A100 Tensor Core GPU.

Performance of text classification

Experiments were conducted on five benchmark datasets, and the results are listed in Table 3. The experimental results on the original benchmark models were obtained from TextGCN (Yao, Mao & Luo, 2019) and other original studies (Liu et al., 2020; Lin et al., 2021). The results in Table 3 show that DALS performed well on several datasets when using the BertGCN backbone, achieving higher classification accuracy than some traditional and outstanding methods.

Table 4 lists the accuracy and Macro-F1 results of five models with DALS. It shows that systems with DALS outperform those without DALS in all datasets. Compared with the baselines, the accuracy of models using DALS on Ohsumed improved by 5% on average and by 2% on the 20NG and R52 datasets. The existence of several categories in these datasets, where some categories are difficult to distinguish, may be a possible explanation for this phenomenon. The enhancements are not evident in MR because it only has two opposing labels without any explicit label correlation. However, our method takes advantage of label correlations, which provide limited help on the MR dataset. Although some results cannot meet the desired performance on the MR dataset, the overall results remain competitive, which proves the effectiveness and flexibility of DALS.

Accuracy and calibration performance with different losses

In this study, we also explored the test performance and calibration ability of DALS compared with other methods. The expected calibration error (ECE) (Naeini, Cooper & Hauskrecht, 2015; Guo et al., 2017) is a commonly used method for measuring calibration. The samples were evenly distributed in $M$ bins. $B_m$ represents the set of predicted samples belonging to the $m$-th bin. The average accuracy of the samples in $B_m$ is denoted as $Ac{c}_m$, and the average confidence within $B_m$ is denoted as $Con{f}_m$. Here, we set $M=10$.

(9) $$\mathrm{E}\mathrm{C}\mathrm{E}=\underset{m}{\sum ^M}{\displaystyle \frac{\left|B_m\right|}{N}\left|Ac{c}_m-Con{f}_m\right|}$$

As shown in Table 5, the accuracy and ECE of the different methods are reported on the four datasets, and only the state-of-the-art BertGCN model is chosen for comparison. Our method achieves a higher accuracy than other existing methods. The performance gains suggest that DALS helps improve text classification models, such as BertGCN. The ECE results also show that our method achieves a lower ECE than most other methods, including CE, and enables the calibration of neural models. Although our ECE result on Ohsumed is less satisfactory than that on LS, a balanced performance is achieved with higher accuracy.

Effect of corrupted labels

To explore the ability of the model to deal with mislabeling, 5%, 10%, 15%, 20%, 25%, and 30% of the training data were randomly selected, and the labels were randomly replaced from among the remaining labels with the same transition probability. The test set remained unchanged. Figures 2 and 3 both show the effects of different proportions of corrupted labels on the test results. Figure 2 presents the results of the experiments conducted on TextCNN, whereas Fig. 3 uses TextGCN. In general, as the percentage of fake labels increases, the accuracy decreases. After using DALS, the performance of the backbone improved in most cases. These experiments prove that the proposed method maintains its robustness and reduces the negative impact of labeling errors on the model.

Confusion matrices were generated on the Ohsumed dataset (Fig. 4), where each case tends to be classified into the C23 category of the backbone; therefore, the color of this column is darker. After applying DALS, the model reduces the predicted probability of C23, and the color of column C23 is lighter. This suggests that if the sample is often misclassified into certain categories that are confusing, more supervision is required for that class instead of excessive regularization.

Effect of hyperparameters

Among the hyperparameters, $\alpha$ measures the contribution of soft and hard targets, which represents the degree to which model prediction deviates from the hard labels. As shown in Fig. 5A, when $\alpha =0.96$, TextGCN with DALS achieves the best result on the Ohsumed dataset. When the value exceeds 0.96, the model performance degrades, caused by the small proportion of soft labels. The increase in the non-target distribution is too small to show a difference between labels. When $\alpha$ is lower than 0.96, the contribution of the soft label is higher, increasing error tolerance. This can easily cause underfitting, reducing the learning ability of the model. Figure 5A also shows that the highest accuracy of BertGCN with DALS is obtained when $\alpha$ = 0.95. We also explored the settings of $\lambda$ to balance the impact of true positive and false positive cases, as shown in Fig. 5B. A $\lambda$ value of approximately 1.4 emerges as the optimal balance point, yielding the highest accuracy on the test set with the TextGCN method. Deviations towards smaller or larger values of $\lambda$ result in a decline in model efficacy. Tuning $\lambda$ to an appropriate value can effectively control the smoothness of the model, thereby enhancing the overall model by managing true positive and false positive instances.

Connection with model complexity

We also investigated whether our regularization method affects the complexity of the model. The TextCNN model requires several kernels to capture different text features, with a default kernel size of 100. Our experiments reduced the number of neurons by reducing the number of kernels to 2, 4, 6, 8, and 10. The results shown in Fig. 6 indicate that the model captures more features with an increasing number of kernels, and its accuracy remains higher than that of the backbone.