Self-aware cycle curriculum learning for multiple-choice reading comprehension

MRC datasets

Each type of MRC task has a more typical dataset. CNN & Daily Mail, proposed by Hermann et al. (2015), is a cloze-style reading comprehension dataset created from news articles using heuristics and is a classic dataset in the field of MRC. RACE is a multiple-choice dataset, and it covers a wide range of fields. It contains more than 100,000 questions posed by experts, and it focuses more on reasoning skills (Lai et al., 2017). SQuAD is a span extraction dataset proposed by Rajpurkar et al. (2016), which limits the answer to continuous fragments in the original text. It promoted the development of machine reading comprehension. MS MARCO is a free answering dataset (Nguyen et al., 2016). It does not limit the answer to a fragment in the document. It requires the machine to have the ability to comprehensively understand multi-document information and aggregate it to generate the answer to the question, which is closer to the real world.

Multiple-choice datasets provide a more accurate assessment of machine understanding of language, because questions and answers may come from human generalizations or summaries and may not appear directly in the document. Methods that rely only on information retrieval or word frequency cannot achieve good results. Sun et al. (2020) presents the first free-form multiple-choice Chinese machine reading comprehension dataset (C³). Various question types exist such as linguistic, domain-specific, arithmetic, connotation, implication, scenario, cause-effect, part-whole, and precondition in this dataset. Therefore it requires more advanced reading skills for the machine to perform well on this task. This article conducts related experiments on the C³ dataset.

Multiple-choice methods

Previous research on multiple-choice is diverse. Dai, Fu & Yang (2021) propose an MRC model incorporating multi-granularity semantic reasoning. The model fuses the global information with the local multi-granularity information and uses it to make an answer selection. Sun et al. (2022) extract contextualized knowledge to improve machine reading comprehension. However, these approaches add extra features to the model, without the ease of in-depth analysis of the existing data. Zhang et al. (2020b) introduce the context vector of the syntax-guided to parse the passage and question separately, and obtain finer vector representations of passage and question, so as to give more accurate attention signals and reduce the influence of noise brought by long sentences. However, it is required to build a specific parsing tree for passages and questions. Zhang et al. (2020a) consider the interaction between documents, questions, and options, and introduce a gated fusion mechanism to filter out useless information, but the study does not consider the difficulty of the samples themselves. In this article, we propose the SACCL to mine the own characteristics of existing samples, and rationally arrange and use them to achieve better results.

Curriculum learning

Extensive researches have been carried out, with the curriculum learning proposed in Bengio et al. (2009). It aims to facilitate the model training in a specific order, which leads to improved model performance (Hacohen & Weinshall, 2019; Xu et al., 2020). It has been applied to many fields, such as machine translation (Liu et al., 2020; Zhang et al., 2018; Platanios et al., 2019,), image recognition (Büyüktas, Erdem & Erdem, 2020; Huang et al., 2020), data-to-text generation (Chang, Yeh & Demberg, 2021), reinforcement learning (Narvekar et al., 2020), information retrieval (Penha & Hauff, 2020), speech emotion recognition (Lotfian & Busso, 2019), emotion recognition (Yang et al., 2022), spelling error correction (Gan, Xu & Zan, 2021). Curriculum learning has brought different degrees of improvement to these fields. Kumar et al. (2019) adopt CL to reinforcement learning to optimize the model parameters. CL also has shown to be useful for data processing to improve the quality of the training data (Huang & Du, 2019). Liu et al. (2018a) proposed a CL-NAG framework that utilizes curriculum learning to improve data utilization. CL-NAG makes full use of both noisy and low-quality corpora. In addition to the fact that the data is arranged in order from easiest to hardest, there are also some cases (Zhang et al., 2018, 2019; Kocmi & Bojar, 2017) where the data is arranged from difficult to easy with good results. With the wide application of deep learning in various fields, the use of CL to control the order of training data has received more and more attention.

Task description

For multiple-choice reading comprehension, a document (denoted as D) and a question (denoted as Q), and a set of options (denoted as O) are given. Our task is to learn the predictive function F, which generates the answer (denoted as A) by receiving the document D and the related question Q and the options O, we define the task as follows:

(1) $$F(D,Q,O)=A\in (o^1,o^2,o^3,o^4)$$

The document $D=d_1,d_2,\dots ,d_n$ (n indicates the total number of words in the document), the question $Q=q_1,q_2,\dots ,q_m$ (m indicates the total number of words in the question), and the options $O=o^1,o^2,o^3,o^4,o_k^u=o_1^u,o_2^u,o_3^u,o_4^u,\dots ,o_k^u$, where, $u=(1,2,3,4)$, each option consists of K words. A belongs to one of the four options, which is shown in Eq. (1). This setup challenges us to understand and reason about both the question and document in order to make an inference about the answer.

Model

Following the implementation of BERT (Devlin et al., 2019), we concatenate the document, the question, and the option together as shown in Eq. (2), which is the input sequence. The input sequence is concatenated using two special tokens, < CLS > for the beginning, < SEP > for the end and the middle separator. The length of the input sequence is the sum of the document, the question, the option, and special tokens. We pass them to BERT and use a linear classifier on top of it to get the probability distribution. These are shown in Eqs. (3) and (4).

(2) $$Input=[<CLS>D<SEP>Q<SEP>o^i<SEP>]$$

(3) $$H=BERT(Input)$$

(4) $$logits=Linear(H)$$where the logits are the probability distribution of the option.

Our SACCL approach

Our SACCL (Self-Aware Cycle Curriculum Learning) approach consists of two important parts. Firstly, Difficulty Assessment judges the difficulty score of each sample. Secondly, Training Strategy arranges the order in which the samples are trained.

Difficulty assessment

As illustrated in Section Introduction, in order to avoid the limitations of rule-based design, and increase the scalability of the model, we design a self-aware approach, which independently judges the difficulty score of the samples by the model itself, and then sorts the samples according to the difficulty scores. The whole process consists of two parts. Firstly, the training model divides the original training data into six blocks, each block is trained with a model. Secondly, the trained models are used to score and sort the other samples, as shown in Fig. 1.

Let R be the training examples set, $r_k$ is the k-th example in R, difficulty assessment is to calculate the difficulty score $s_k$ of $r_k$ according to a certain evaluation standard, let S be the whole difficulty score set corresponding to R.

In order to evaluate the difficulty, we first randomly divide the training set R into six blocks according to the size of the dataset, denoted as $\left\{R_j^{\mathrm{\prime }}\text{:}j=1,2,\dots ,6\right\}$. Then we train six corresponding models $\left\{M_i^{\mathrm{\prime }}\text{:}i=1,2,\dots ,6\right\}$ on them, each model $M_i^{\mathrm{\prime }}$ uses $R_j^{\mathrm{\prime }}$ as the training set, and the remaining five pieces of data are used as the validation set to obtain the difficulty score. We also try to split the data into seven or five blocks, but the experiment works best when the data is divided into six blocks, so we choose to split the data into six blocks.

Specifically after training, $M_i^{\mathrm{\prime }}$ predicts examples $R_j^{\mathrm{\prime }}$, where $j\ne i$. F is the metric calculation formula, correct prediction is 1, error prediction is 0, as:

(5) $$s_{ki}=F(M_i^{\mathrm{\prime }},r_k^{\mathrm{\prime }})(r_k^{\mathrm{\prime }}\in R_j^{\mathrm{\prime }},j\ne i,i,j=1,2,\dots ,6)$$

The scores of each example are counted after all predictions are over, as:

(6) $$s_k=\sum _{i=1}^5{s}_{ki}$$

The examples are sorted according to $s_k$. If $s_k=5$ indicates that all five models predict correctly. This type of example is the simplest and it is ranked first, and so on. If $s_k=0$ indicates that all five models predict incorrectly. This type of example is the hardest and it is ranked last.

Datasets

To verify the effectiveness of SACCL on multiple-choice tasks, the multiple-choice Chinese machine reading comprehension dataset (C³) is considered. This dataset contains 13,369 documents collected from questions in the general domain of the Chinese Proficiency Test and 19,577 multiple-choice free-form questions associated with these documents. In this dataset, there are 11,869 questions in the training set, 3,816 questions in the validation set, and 3,892 questions in the test set. The documents contain conversational form documents and non-dialogical documents with mixed topics (e.g., stories, news reports, monologues, or advertisements), which requires models to have stronger reasoning capabilities. C³ tasks can be classified into C³-Dialogue $(C_D^3)$ and C³-Mixed $(C_M^3)$ based on the two types of documents. Also, 86.8% of the questions in this dataset require a combination of internal and external knowledge of the document (general world knowledge) to better understand the given text.

Evaluation metric

Exact Match (EM), Precision, Recall, and macro-F1 (Rajpurkar et al., 2016) are used to evaluate the model performance. Exact Match measures the proportion of the correct results (including positive and negative cases) predicted by the model. A higher value of EM means that the model answers more questions correctly. Precision is the precision rate, indicating the proportion of the number of correctly predicted samples in all the samples whose prediction results are positive examples, and Recall is the recall rate, indicating the percentage of all the samples with positive results that are correctly predicted. F1 is used to measure the repetition rate of the prediction compared to the ground truth answer.

(7) $$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(8) $$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(9) $$macro-F1={\displaystyle \frac{2\cdot Precision\cdot Recall}{Precision+Recall}}$$

Experimental settings

Our model is built using the PyTorch deep learning framework. We further pre-train with this model on one 3,090 GPU. We adjust the parameters in our model to what is shown in Table 2. In order to prevent model overfitting and excessive training time, the validation set is tested every round during the training phase of the model.

Experimental results

To evaluate the SACCL approach, the BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), and ALBERT (Lan et al., 2020) are used as the baseline. BERT is a bidirectional transformer (Vaswani et al., 2017) network. It uses the Encoder module in the transformer architecture and abandons the Decoder module, so that it automatically has bidirectional encoding capabilities and powerful feature extraction capabilities. BERT introduces masked language modeling (MLM) and next sentence prediction (NSP) tasks to train on plain text. RoBERTa is a finer-tuned version of the BERT model. The RoBERTa model removes the NSP task, uses a dynamic mask strategy, and trains with larger batch size and learning rate. ALBERT is a Lite BERT. It introduces parameter reduction technology, which significantly reduces the number of parameters of BERT without significantly compromising its performance, thereby improving parameter efficiency. The base pre-trained language models are used in the main experiments. We use BERT-base to re-implementation the article (Sun et al., 2020) and get better results on the test set.

Main comparison

In order to illustrate the effectiveness of our method, we first arrange the training data in the way of self-aware curriculum learning (SCL); that is, arranging the data from easy B₁ to hard B₆, and add a bucket each time, until all the data are added to the training set, and then train for several epochs until convergence. From Table 3, we can see that the SCL approach achieves 0.99% gain on EM over the results in the article (Sun et al., 2020) on C³-test. It can achieve improvements of 0.28% gain on EM over the baseline RoBERTa model on the C³-test, and achieve improvements of 1.92% gain on EM over the baseline ALBERT model on the C³-test, which proves that giving the model the easiest data at the beginning of training and then gradually increasing the difficulty of the data will get better results than randomly arranging the data.

Table 3:

Comparison table of main experimental results.

	Dev				Test
Method	EM	Prec	Rec	F1	EM	Prec	Rec	F1
BERT Base	65.70	–	–	–	64.70	–	–	–
BERT Base*	65.67	65.64	65.62	65.58	65.95	66.09	66.05	65.97
RoBERTa Base	67.76	67.67	67.77	67.70	66.98	66.94	66.97	66.91
ALBERT Base	53.87	53.68	53.86	53.68	53.62	53.48	53.66	53.51
BERT Base+SCL	65.85	65.80	65.85	65.78	65.69	65.74	65.78	65.66
RoBERTa Base+SCL	67.85	67.74	67.92	67.81	67.26	67.19	67.26	67.20
ALBERT Base+SCL	55.79	55.68	55.86	55.73	55.54	55.47	55.58	55.48
BERT Base+CTS	65.95	65.86	65.90	65.84	65.80	65.80	65.88	65.76
RoBERTa Base+CTS	68.18	68.07	68.23	68.13	67.54	67.50	67.57	67.51
ALBERT Base+CTS	55.79	55.68	55.86	55.73	55.54	55.47	55.58	55.48

DOI: 10.7717/peerj-cs.1179/table-3

Note:

* Our re-implementation.

In order to verify the effect of the CTS (Cycle Training Strategy) approach, we arrange the data in an iterative manner in Fig. 2. As shown in Table 3, the CTS approach achieves 1.1% gain on EM over the results in the article (Sun et al., 2020) on the C³-test. It can achieve improvements of 0.56% gain on EM over the baseline RoBERTa model on the C³-test, and achieve improvements of 1.92% gain on EM over the baseline ALBERT model on the C³-test, which proves that from easy to difficult, and then from difficult to easy, this repeated arrangement of data can get better results than SCL approach. In summary, the decent performance on the benchmark dataset can validate the effectiveness of the proposed CTS.

Document length analysis

The maximum text length of the BERT model is 512. According to the analysis in Table 7, it is known that the ratio of C³ training dataset document length exceeding 512 is 14.2%. When we are dealing with multiple-choice tasks, the sum of the document length, question length, and option length exceeds 512, we select the document for additional processing. Because, generally speaking, the document length is the longest, or when the question length exceeds the document length, the total length will not exceed 512. Following (Sun et al., 2019), three strategies are chosen to truncate the document length. The first is deleted from the tail. The second is deleted from the head. And the third is deleted from the middle (we select the first 128 and the last 380 tokens). Through experimental analysis, it is found that most of the key information of long text comes from the head of the document in the C³ dataset, as shown in Table 8. Therefore the experiments are based on this setting in this article.