What does Chinese BERT learn about syntactic knowledge?

Setup

In this subsection, we probed which individual heads could best learn dependency relations. When we input a sentence into Chinese BERT, we obtained the attention matrix about characters in this sentence for each head. Considering that no explicit word boundary exists in Chinese sentences, we used the word segment of datasets in ‘Datasets’ as the standard. Then, we summed the columns and averaged the rows corresponding to the constituent characters of the standard words: ${\displaystyle {\alpha }_{w_p\to w_q}=\frac{1}{\left|w_p\right|}\sum _{{\mathrm{c}}_{\mathrm{i}}\in {\mathrm{w}}_{\mathrm{p}}}\sum _{{\mathrm{c}}_{\mathrm{j}}\in {\mathrm{w}}_{\mathrm{q}}}{\alpha }_{c_i\to c_j}}$ where, w_p and w_q are the words in the input sentence. c_i and c_j are the constituent characters in words w_p and w_q, respectively. α ∈(0, 1) ^n×n is the attention weight of a certain head regarding the input sentence. — w_p— is the number of characters in w_p.

Figure 2 shows that an example sentence parsed by dependency relations and expressed by attention weights from head 6-6. If an attention head learned a certain dependency relation well, this head had a higher probability of allocating the maximum weight to the head word in each row of the attention matrix. During the evaluation, we ignored the direction between the dependent word and head word, and tested the performance of each attention head on each dependency relation and overall relations. We used the undirected unlabeled attachment score (UUAS) as our evaluation: ${\displaystyle UUAS=\frac{\mathrm{c}orrec{t}_{\mathrm{i}}^{\mathrm{k}}}{\left|{\mathrm{rel}}_i\right|}}$ where, —rel _i— is the number of dependency relation i in the datasets, and ${\text{correct}}_i^{\mathrm{k}}$ is the number of correct predictions of relation i for a given head k.

Baselines

We adopted positional offset and Random BERT as baselines. For the positional offset baseline, we determined the most common position where the head word could occur for each attention word. For the Random BERT baseline, we used a BERT-base model with randomly initialized weights.

Results

Tables 1 and 2 show the results of our probing method and baselines on UD2.11 and CDT1.0, respectively. The number in the parentheses in the line “positional offset” is the offset location with the best performance (e.g., (-1) means the head word was located to the left of the dependent word). The number in the parentheses in the line “Chinese BERT” denotes the best performance head, i-j denotes the j-th head in the i-th layer. The 17 most common relations are shown in Table 1 and all relations are shown in Table 2.

From the two tables, we found that Chinese BERT >Positional Offset >Random BERT in terms of performance. This indicated that the attention heads in Chinese BERT learned some dependency relations implicitly, while Random BERT captured very little syntactic knowledge (<10%). Meanwhile, positional offset performed similarly on some dependency relations, such as “compound” in Table 1 and “RAD” in Table 2. This could be because the head word appeared fixed in the distance of the dependent word. The attention head could only learn positional or distance information between the two words to achieve general performance.

In addition, we also found that some dependency heads did significantly learn some specific syntactic relations, sometimes achieving high accuracy, such as “obj” and “aux” in Table 1, and “VOB” and “POB” in Table 2. However, no single heads performed well on the total relations. The best single heads only obtained 35.1 UUAS on the two datasets. This finding is similar to the work of Clark et al. (2019) on English treebanks. We also found that most of the heads with the best performance in the specific dependency relations were located in the middle layers (layers 5–9). This was due to the fact that Chinese BERT encodes how to organize words into a sentence mostly in the middle layer, similar to English BERT (Jawahar, Sagot & Seddah, 2019).

Setup

According to the previous subsection, we found that positional offset could also achieve good performance on some dependency relations. Therefore, we investigated whether the distance between dependent words and head words could affect the performance of Chinese BERT in capturing syntactic knowledge. UUAS was still used as our evaluation metric for this experiment.

Baselines

We adopted Random BERT as a baseline. For the full details, please refer to ‘Probing individual attention heads’.

Results

Figure 3 shows the accuracy of relative positions on UD2.11 and CDT1.0, respectively. We also found that Chinese BERT apparently exceeds Random BERT in different positional distributions. In addition, the performance of Chinese BERT decreased as the distance increased. This indicates that positional information between words is important for Chinese BERT. The closer the distance between the head word and dependent word, the better Chinese BERT can capture the dependency relation between the two. Among all relative positions, Chinese BERT achieves very high performance (>99%) when the head word and dependent word are next to each other (±1).

Furthermore, in order to analyze the influence of positional distribution on different dependency relations, we calculated the accuracy of relative positions on the common relations of the two datasets, shown in Fig. 4. From this figure, we can easily see that the relation between model performance and dependency distance still exists in most dependency relations.

Setup

Based on the findings from the previous subsections, we were very interested in exploring particular linguistic phenomena existing in Chinese, as well as determining whether Chinese BERT had captured them. Hence, we designed a test suite for evaluation.

Our test suite covered two sentence constructions unique in Chinese and three auxiliary words for expressing aspects in Chinese sentences. For sentence construction, we chose baˇ (把) construction and bèi (被) construction. For auxiliary words, -zhe (着), -le (了), and -guò (过) were adopted.

The particle baˇ (把) is commonly used in Chinese. It can change the word order from “subject - verb - object” to “subject - baˇ - object - verb” (Ye, Zhan & Zhou, 2007). The construction is always used to express the result of the action on the object. Different from English, the bèi (被) construction is used to express passive voice in Chinese. Due to lack of morphological inflection, the particle bèi as a fixed word is used before an agent to express passive voice (Wang & Xu, 2015). The basic structure consists of “object - bèi - subject - verb - other components”. The particles -zhe (着), -le (了), and -guò (过) in Chinese can come after a verb to express aspects in Chinese sentences. The durative aspect can be reflected by the marker -zhe, which describes an enduring or continuing situation. The perfective aspect can be expressed by the markers -le and - guò. The perfective particle -le expresses a situation in its entirety, an event bounded at the beginning and the end, while the other perfective particle -guò presents an event that has been experienced at some indefinite time (Chen & Shirai, 2010). We give some example sentences for illustration:

(1) 他把杯子打破了

(He ba cup break le.)

He broke the cup.

(2) 杯子被他打破了。

(Cup bei he break le.)

The cup was broken by him.

(3) 他去北京了。

(He go Beijing le.)

He went to Beijing.

(4) 他去过北京。

(He go guo Beijing.)

He has been to Beijing.

(5)门开着。

(Door open zhe.)

The door is open.

In our experiment, we probed the model’s competence at capturing these phenomena through an attention matrix. Specifically, we measured whether those particles allocated the maximum attention weight to their dependency heads.

Baselines

We adopted positional offset and Random BERT as baselines. For the full details, please refer to ‘Probing individual attention heads’.

Results

Our experimental results are displayed in Table 3. The number in the parentheses is the specific head with the best performance. The performance still illustrates: Chinese BERT >Positional Offset >Random BERT. Meanwhile, we also saw that the results of Chinese BERT and Positional Offset were the same on the particles -zhe (着) and -guò (过). By analyzing the corpus, we found that -zhe and -guò followed the main verb most of time, indicating that Chinese BERT could only learn some positional information used in predicting the dependency relations of the two particles. In particular, we discovered that the two constructions (bèi and baˇ) were learned very well by the heads in the middle layers (layers 4–6), while the three particles (-zhe, -le, and - guò) were captured best by the heads in the lower layers (layers 2–4). This indicates that the structure information about sentences exists in the middle layers. Some lexical or morphological knowledge is embedded in the lower layers (Jawahar, Sagot & Seddah, 2019).

Setup

In ‘Probing individual attention heads’ we found that some single attention heads were good at learning the corresponding dependency relations, but no heads could capture the whole dependency structures of sentences. Hence, we considered to combine all heads to perform sentence parsing. We followed the setting from Clark et al. (2019) by training a classifier combing with all attention heads linearly: ${\displaystyle UUAS=softmax\left(\sum _{k=1}^{144}{w}_k{\alpha }_{ij}^k\right)}$ where softmax is a function for classification, 144 is the number of heads in Chinese BERT, w_k are weights for training, and $a_{ij}^k$ is the attention weight of word i on word j produced by head k. We refer to this method as “Attn”.

Additionally, we also considered the impact of words in carrying out parsing tasks. We incorporated word embeddings from Song et al. (2018) into the classifier. This method is called “Attn + embeddings”.

Baselines

Similar to Clark et al. (2019), “Random Initial Attention + embeddings”, “Right Branching”, and “Distances + Embeddings” were adopted as baselines in this experiment. “Random Initial Attention + embeddings” used a randomized network and incorporated the pre-trained word embeddings for head and dependent words. Meanwhile, “Right Branching” predicts that the head word was always on the right of the dependent. “Distances + Embeddings” is used to replace the attention matrix of Chinese BERT with pre-trained word and positional embeddings, and randomly initialized other weights.

Results

Results are exhibited in Table 4. We can see that both “Attn + embeddings” and “Attn” achieved better performances than the baselines on the two datasets. The accuracy of “Attn” was higher than 50%, and “Attn + embeddings” obtained nearly 70% accuracy. These results are similar to the findings in English (Clark et al., 2019; Hewitt & Manning, 2019). This indicates that the attention heads of Chinese BERT did acquire many organizational structures in language. “Attn + embeddings” outperformed “Attn”(∼15%), which proves that specific vocabulary contributes to Chinese BERT capturing dependency relations. Together with the findings from individual attention heads, we believe that Chinese BERT encodes abundant information in syntax by a way of indirect supervision, even though the word boundaries do not exist in the Chinese language.

Setup

Besides probing attention heads, we explored the ability of hidden representation in capturing syntactic knowledge. Because no suitable datasets for testing Chinese syntax were available, we designed three Chinese syntactic tasks by imitating the work in English (Conneau et al., 2018): Bigram Shift (BShift), Tree Depth (TreeDepth), and Dependency Relation (DepRel).

In the BShift task, we inverted two random adjacent characters and let the model predict whether the sentence was inverted. In the TreeDepth task, the depth of the dependency tree of a sentence was predicted. The task DepRel refers to the prediction of the dependency relation of a phrase consisting of two words.

As shown in Fig. 5, we overlaid a one-layer MLP on the hidden state of each layer to construct a classifier. After trying different parameter combinations, an optimal set of parameters were finally determined. We then only trained each classifier one epoch, so that the classifier was forced to pay attention to information encoded in the hidden state representation as much as possible (Choenni & Shutova, 2020).

Baselines

We adopted Random BERT as a baseline. For the complete details, please refer to ‘Probing individual attention heads’.

Results

Table 5 displays the prediction result of each layer. The number in parentheses denotes the baseline from Random BERT. The bold numbers are the maximum values among all 12 layers in each task. According to the results, the best performances of Chinese BERT were all achieved in the final layer (layer 12). Chinese BERT still outperformed Random BERT in most of the layers across the three tasks. Compared to probing in the attention head, the performance of the hidden state from Random BERT was not very poor. This indicates that these hidden states contain some information that will contribute to predicting syntactic knowledge. Interestingly, we found the prediction results from Random BERT were better than Chinese BERT’s corresponding to lower layers (layers 1–3) in the TreeDepth task. This could be because a relatively complete structure is needed to be captured in the TreeDepth task. However, Chinese BERT may not encode structural information well in the lower layers. Therefore, its performance was outperformed by Random BERT.