A lightweight speech recognition method with target-swap knowledge distillation for Mandarin air traffic control communications

Automatic speech recognition for air traffic control communications

When it comes to automatic speech recognition for air traffic control communications, numerous works have been conducted by academia and industry from around the world. The majority of them concentrate on recognition performance, such as the recognition error rate of words, characters, and sentences, which can reduce the workload of ATCOs and potential flight safety risks associated with miscommunications (Helmke et al., 2016).

Helmke and Ohneiser et al. adopted ontology-enhanced approaches for speech recognition applications in a variety of real-world ATC scenarios, such as robust command recognition and extraction (Ohneiser et al., 2021a; Ohneiser et al., 2021b) and callsign recognition for automatic readback error detection (Helmke et al., 2021). Zuluaga-Gomez et al. proposed a contextual semi-supervised learning approach combining untranscribed ATC speech and air surveillance data (Zuluaga-Gomez et al., 2021) and a two-stage method incorporating contextual radar data (Nigmatulina et al., 2022), which enhanced the callsign recognition. Lin et al. (2021c) proposed several end-to-end multilingual speech recognition frameworks for ATC communications (Lin et al., 2021c; Yang et al., 2020; Lin et al., 2021a; Lin et al., 2021b), which focus on the multilingual ASR on parallel Chinese and English ATC recordings. Shi et al. (2022) proposed an end-to-end Conformer-based multi-task learning speech recognition model for Mandarin radiotelephony communications in civil aviation, which can effectively extract global and local acoustic features, especially the contextual long-distance dependent local similarity features.

The above ASR works for air traffic control (ATC) is mainly concerned with recognition performance while paying little attention to recognition speed. To address this issue, this article introduces the lightweight idea of knowledge distillation into the ASR for Mandarin ATC communications to enhance the generalization performance of the light model. In this way, the generated lightweight ASR model achieves a balance between recognition performance and transcription latency, allowing ATCOs and pilots sufficient time to respond immediately and effectively.

Knowledge distillation

Over the past decade, advancements in model compression (Cheng et al., 2018; Choudhary et al., 2020), such as knowledge distillation (Li et al., 2021; Huang et al., 2018), low-rank matrix factorization (Povey et al., 2018), network pruning (Gao et al., 2020), parameter quantization (He et al., 2019; Sainath et al., 2020), and lottery ticket hypothesis (Ding, Chen & Wang, 2022), have attracted much attention. Knowledge distillation (KD) aims to transfer knowledge from the heavy pre-trained teacher model into the relatively compact student model, regardless of their structural distinctions (Wang & Yoon, 2021). It can enhance the lightweight student model’s generalization without introducing extra costs, since soft targets convey more generalization information than hard ones to prevent overfitting of the student model. Regarding knowledge source types, distilled knowledge consists of response-based knowledge from the final output layer, feature-based knowledge from intermediate layers, and relation-based knowledge within feature maps or data samples (Gou et al., 2021).

After the idea of KD was first proposed in Hinton, Vinyals & Dean (2015), the logits-based KD and the feature-based KD, like FitNets (Romero et al., 2015), take turns playing the central role throughout the history of KD research. Compared with logits-based KD, feature-based KD achieves superior performance at the expense of extra computation and storage requirements during training. It is worth noting that decoupled knowledge distillation (DKD) (Zhao et al., 2022) has recently brought the logits-based KD approach back into the state-of-the-art, which can efficiently extract the rich “dark knowledge” (Hinton, Vinyals & Dean, 2015) from the teacher network. However, DKD still suffers from two issues: indirect strategies to mitigate overconfidence regarding the target class and insufficient investigation into hyperparameters for ASR tasks. As for the former, with the help of an efficient decoupling mechanism, the conventional logits-based method’s restrictions on the flexibility of knowledge transfer have been somewhat alleviated; therefore, NCKD can play a more prominent role, particularly when the target class encounters overconfidence. Nevertheless, it requires more direct strategies to prevent overconfidence in the target class (Ren, Guo & Sutherland, 2022). As for the latter, the scale factor hyperparameters for the two decoupled components differ considerably for various tasks. While DKD explored hyperparameters for a number of computer vision tasks, further investigation is required for assignments like ASR.

To address this issue, this article proposes Target-Swap Knowledge Distillation (TSKD), a simple yet effective lightweight strategy that swaps the logit output of the teacher and student models for the target class. On the one hand, it mitigates the potential overconfidence of the teacher model regarding the target class; on the other hand, it enables the student model to concentrate on the distillation of knowledge from non-target classes. In addition, hyperparameters between TKD and SKD are elaborately investigated through parameter sensitivity experiments for ASR tasks.

The overall architecture of target-swap knowledge distillation (TSKD)

In logits-based offline knowledge distillation for speech recognition, the teacher model is a sophisticated pre-trained model with good performance. Their output logits are valuable prior knowledge, which can guide the student model to learn and represent the mapping relationship between acoustic features and predicted text through knowledge transfer.

In this article, we propose a simple yet effective lightweight strategy named Target-Swap Knowledge Distillation (TSKD), which swaps the logit output of the teacher and student models for the target class. Figure 1 depicts the overall architecture of knowledge distillation from the pre-trained teacher model to the student model with the TSKD strategy. The overall loss ${\mathcal{L}}_{overall}$ comprises two components, ${\mathcal{L}}_{TSKD}$ for the TSKD loss proposed in this article and ${\mathcal{L}}_{CE}$ for the original cross-entropy loss, which can be calculated as the formula below, (1)${\displaystyle {\mathcal{L}}_{overall}=\alpha {\mathcal{L}}_{TSKD}+\beta {\mathcal{L}}_{CE},}$ where α and β are tuned hyper-parameters to make a trade-off between the two losses, given the usual constraints that α + β = 1.

In the following two subsections, we will describe the two parts of TSKD: the teacher’s target class logit-swapped knowledge distillation (TKD) and the student’s target class logit-swapped knowledge distillation (SKD). On the one hand, it can enable the student model to concentrate on the distillation of knowledge from non-target classes; on the other hand, it can mitigate the potential overconfidence of the teacher model regarding the target class. Figure 2 illustrates the distinction between the conventional logits-based knowledge distillation loss function (Hinton, Vinyals & Dean, 2015) and that of the TSKD proposed in this article. In this way, the network will be optimized collaboratively to balance the two portions of enhanced loss functions, as calculated with the following formula: (2)${\displaystyle {\mathcal{L}}_{TSKD}={\lambda }_1{\mathcal{L}}_{TKD}+{\lambda }_2{\mathcal{L}}_{SKD}.}$

Teacher’s target class logit-swapped knowledge distillation (TKD)

With the acoustic features of phonetic fraction as input, the teacher model for speech recognition will generate the logits vector ${\mathit{z}}^{\mathcal{T}}=\left\{z_1^{\mathcal{T}},z_2^{\mathcal{T}},\dots ,z_V^{\mathcal{T}}\right\}$, where z_i represents the network’s predicted logit value for the i-th character and V represents the vocabulary size, i.e., the total number of categories. Correspondingly, the student model will produce the logits vector ${\mathit{z}}^{\mathcal{S}}=\left\{z_1^{\mathcal{S}},z_2^{\mathcal{S}},\dots ,z_V^{\mathcal{S}}\right\}$, where $\mathcal{T}$ and $\mathcal{S}$ stand for the teacher and the student model, respectively. Thus, the class probability distributions of the teacher and student models, namely ${\mathit{p}}^{\mathcal{T}}=\left\{p_1^{\mathcal{T}},p_2^{\mathcal{T}},\dots ,p_V^{\mathcal{T}}\right\}$ and ${\mathit{p}}^{\mathcal{S}}=\left\{p_1^{\mathcal{S}},p_2^{\mathcal{S}},\dots ,p_V^{\mathcal{S}}\right\}$, can be calculated with the following formula ¹ : (3)${\displaystyle p_i=\frac{exp\left(z_i\right)}{\sum _{j=1}^{V}exp\left(z_j\right)}.}$

To urge the student model to concentrate on non-target classes for knowledge distillation, inspired by DKD (Zhao et al., 2022), the logit output of the student model on the target class is substituted with that of the teacher model $z_t^{\mathcal{T}}$. Hence, the student model yields the logits vector ${\mathit{z}}^{\mathcal{S}}=\left\{z_1^{\mathcal{S}},z_2^{\mathcal{S}},\dots ,z_{t-1}^{\mathcal{S}},z_t^{\mathcal{T}},z_{t+1}^{\mathcal{S}},\dots ,z_V^{\mathcal{S}}\right\}$, and the student model’s class probability can be represented as ${\hat{\mathit{p}}}^{\mathcal{S}}=\left\{{\hat{p}}_1^{\mathcal{S}},{\hat{p}}_2^{\mathcal{S}},\dots ,{\hat{p}}_t^{\mathcal{S}},\dots ,{\hat{p}}_V^{\mathcal{S}}\right\}$, which can be calculated as follows, (4)${\displaystyle {\hat{p}}_t^{\mathcal{S}}=\frac{exp\left(z_t^{\mathcal{T}}\right)}{\sum _{j=1,j\ne t}^{V}exp\left(z_j^{\mathcal{S}}\right)+exp\left(z_t^{\mathcal{T}}\right)},}$ (5)${\displaystyle {\hat{p}}_i^{\mathcal{S}}=\frac{exp\left(z_i^{\mathcal{S}}\right)}{\sum _{j=1,j\ne t}^{V}exp\left(z_j^{\mathcal{S}}\right)+exp\left(z_t^{\mathcal{T}}\right)},}$ where $i\in \left\{1,2,\dots ,t-1,t+1,\dots ,V\right\}$. Given that the logits generated by the teacher model remain unchanged, its class probability distribution also keeps identical, namely ${\mathit{p}}^{\mathcal{T}}=\left\{p_1^{\mathcal{T}},p_2^{\mathcal{T}},\dots ,p_V^{\mathcal{T}}\right\}$. The student model’s modified class probability distribution ${\hat{\mathit{p}}}^{\mathcal{S}}$ and that of the teacher model ${\mathit{p}}^{\mathcal{T}}$ are employed for the calculation of the KL divergence loss, named the Teacher’s target class logit-swapped Knowledge Distillation (TKD), as the following formula: (6)${\displaystyle {\mathcal{L}}_{TKD}=\mathit{KL}\left({\mathit{p}}^{\mathcal{T}}\left|\right|{\hat{\mathit{p}}}^{\mathcal{S}}\right)=p_t^{\mathcal{T}}log\left(\frac{p_t^{\mathcal{T}}}{{\hat{p}}_t^{\mathcal{S}}}\right)+\sum _{i=1,i\ne t}^V{p}_i^{\mathcal{T}}log\left(\frac{p_i^{\mathcal{T}}}{{\hat{p}}_i^{\mathcal{S}}}\right).}$

Student’s target class logit-swapped knowledge distillation (SKD)

To alleviate the potential overconfidence (Ren, Guo & Sutherland, 2022) of the teacher model on the target class, the logit output of the teacher model on the target class is substituted with that of the student model $z_t^{\mathcal{S}}$. Thus, the teacher model produces the logits vector ${\mathit{z}}^{\mathcal{T}}=\left\{z_1^{\mathcal{T}},z_2^{\mathcal{T}},\dots ,z_{t-1}^{\mathcal{T}},z_t^{\mathcal{S}},z_{t+1}^{\mathcal{T}},\dots ,z_V^{\mathcal{T}}\right\}$, and the teacher model’s class probability distribution can be represented as ${\hat{\mathit{p}}}^{\mathcal{T}}=\left\{{\hat{p}}_1^{\mathcal{T}},{\hat{p}}_2^{\mathcal{T}},\dots ,{\hat{p}}_t^{\mathcal{T}},\dots ,{\hat{p}}_V^{\mathcal{T}}\right\}$, which can be calculated as follows, (7)${\displaystyle {\hat{p}}_t^{\mathcal{T}}=\frac{exp\left(z_t^{\mathcal{S}}\right)}{\sum _{j=1,j\ne t}^{V}exp\left(z_j^{\mathcal{T}}\right)+exp\left(z_t^{\mathcal{S}}\right)},}$ (8)${\displaystyle {\hat{p}}_i^{\mathcal{T}}=\frac{exp\left(z_i^{\mathcal{T}}\right)}{\sum _{j=1,j\ne t}^{V}exp\left(z_j^{\mathcal{T}}\right)+exp\left(z_t^{\mathcal{S}}\right)},}$ where $i\in \left\{1,2,\dots ,t-1,t+1,\dots ,V\right\}$. The teacher model’s modified class probability distribution ${\hat{\mathit{p}}}^{\mathcal{T}}$ and that of the student model ${\mathit{p}}^{\mathcal{S}}$ are employed for the calculation of the KL divergence loss, named the Student’s target class logit-swapped Knowledge Distillation (SKD), as the following formula: (9)${\displaystyle {\mathcal{L}}_{SKD}=\mathit{KL}\left({\hat{\mathit{p}}}^{\mathcal{T}}\left|\right|{\mathit{p}}^{\mathcal{S}}\right)={\hat{p}}_t^{\mathcal{T}}log\left(\frac{{\hat{p}}_t^{\mathcal{T}}}{p_t^{\mathcal{S}}}\right)+\sum _{i=1,i\ne t}^V{\hat{p}}_i^{\mathcal{T}}log\left(\frac{{\hat{p}}_i^{\mathcal{T}}}{p_i^{\mathcal{S}}}\right).}$

Dataset

Under the instruction of first-line ATCOs and tutors of ATC courses, a corpus was established, and qualified ATCOs were chosen to record the corpus in a quiet situation. Consequently, the built dataset follows the linguistic and acoustic feature distribution of air-ground communications regarding call content, voice control, pronunciation regulations, syntactic structure, and linguistic conventions. The Mandarin ATC communications dataset is a paraphrase dataset, with each sample consisting of the ATCO’s control instruction and the flight crew’s readback instruction successively. In this double-checked manner, the command could be carried out efficiently and precisely.

The Mandarin ATC communications dataset is stored as a WAV audio file with a sampling frequency of 16 kHz, 16 bits, and mono for each recording. Ten males and seven females, a total of seventeen certified ATCOs, participated in the recording. With 641 corpora and 10971 recorded voices, the total speech data has a duration of 1516 minutes. The dataset is randomly scrambled and split into the training set, validation set, and test set in the ratio of 7:1:2.

Implementation details

In this experiment, 80-dimensional FBank (Filter bank) features are taken as the model input, and frame window and frameshift size are 25ms and 10ms, respectively. The Kaldi toolkit (Povey et al., 2011) was employed to extract the acoustic features, and the speed factors of 0.9x and 1.1x were applied to scale the dataset for data augmentation. In the training procedure, the relative positional encoding of Transformer-XL (Dai et al., 2019) was utilized, and the Adam optimizer with the Noam learning rate (Vaswani et al., 2017) was adopted for efficient parameter optimization. All models were trained on the dataset through eight NVIDIA A100-SXM4-40G GPUs.

Concerning the effect of the teacher and student architectures on the efficacy of knowledge distillation, Cho & Hariharan (2019) reveals that an excellent teacher only sometimes produces qualified students due to their capacity gap. Hence, it is crucial to select a suitable network as the student model given the teacher model. In this experiment, the teacher models adopted the bidirectional long short-term memory (BiLSTM) network, Transformer (Vaswani et al., 2017), and Conformer (Gulati et al., 2020) models, which followed the configurations in our previous work (Shi et al., 2022). To choose student models with appropriate capacity, several speech recognition model variants with varying network sizes were constructed by modifying the encoder and decoder structures based on BiLSTM, Transformer, and Conformer, respectively. For easy identification, individual abbreviations were assigned to the corresponding student models in Table 1. Each abbreviation consists of the base model type, the number of layers of the encoder N, and the dimension of the decoder’s hidden units, which are connected by underscores.

The experiments employed characters as the fundamental modeling unit and the sentence error rate (SER) as the evaluation criterion with the following formula: (10)${\displaystyle SER=\frac{F}{M},}$ where F is the number of sentences with transcription errors, and M is the total number of sentences.

Table 2 represents the parameters, recognition error rate, and transcription time for all teacher and student model variants on the test set. For fairness, all the transcription results were calculated, given a beam search size of 10. Table 2 demonstrates that as model capacity decreases, the recognition performance of each of the three groups of models declines. Specifically, as the model capacity falls, the sentence error rate (SER) of the Transformer model rises sharply, whereas it goes up relatively gently for the LSTM and Conformer models. Notably, when the student model’s capacity drops to a quarter of the teacher model’s, the LSTM and Conformer models can still make reliable predictions. In contrast, the Transformer model can no longer make effective estimations. It verifies that the Conformer-based speech recognition model (Shi et al., 2022) can effectively extract global and local acoustic features, especially the long-distance context-dependent local similarity features in the dataset of Mandarin ATC communications. Overall, the Conformer-based knowledge distillation significantly lowers the model capacity with minimal accuracy loss.

Effectiveness of TSKD in homogeneous and heterogeneous architectures

Experiments were carried out on the Mandarin ATC communications dataset to investigate the effectiveness of the TSKD training strategy proposed in this article. As shown in Table 3, two distinct teacher-student network structures, including homogeneous and heterogeneous architectures, were employed. The left set used the Conformer teacher model to distill the Conformer student model within homogeneous architectures, whereas the right group made use of the Transformer teacher model to distill the Conformer student model within heterogeneous network structures. The findings suggest that the effectiveness of knowledge distillation can be boosted by employing either TKD or SKD, with TKD bringing greater benefits. It also supports that non-target class knowledge distillation (NCKD) is the primary reason why classical logits-based distillation is effective but severely restricted, which coincides with the efficiency analysis of DKD (Zhao et al., 2022). Furthermore, integrating two training strategies into one, namely TSKD, can further strengthen the performance of knowledge distillation.

In this article, TSKD focuses on the logit output exchange of the target class between the student and teacher models, which can effectively improve the distillation performance. Apart from the target class, the top-k information predicted by the teacher model for each segment of acoustic features is also a sort of valuable information (Reddi et al., 2021); it reflects which k characters the acoustic features input should be transcribed into, from the perspective of the teacher model. On the basis of this assumption, another comparative experiment was conducted to verify the effectiveness of the TSKD, with the predictive top-k output of the teacher model for each segment as the information exchange. As shown in Table 4, the experimental results show that exchanging top-k information is an effective knowledge distillation approach, nevertheless, its performance does not surpass that of the target class, i.e., TSKD.

To further validate the effectiveness of TSKD proposed in this article, several typical knowledge distillation methods were implemented in homogeneous and heterogeneous network structures. The experimental results are presented in Tables 5 and 6, respectively. The results demonstrate that under these six homogeneous or heterogeneous architectures, the TSKD enhanced the recognition accuracy of all student models by an average of 1.13%, achieving the best distillation results in most experiments.

Specifically, the TSKD dramatically boosts the Transformer student model under heterogeneous architectures, reducing the SER by 2.13%. A plausible explanation is that the Conformer teacher model may enable the Transformer student model to capture local similarity features more effectively, particularly the long-distance context-dependent local similarity features in the Mandarin ATC communications dataset. Notably, in the heterogeneous network employing the Transformer teacher model to distill the Conformer student model, the distillation performance of the student model exceeds that of the teacher; it suggests that Conformer is superior to Transformer in terms of the model architecture’s efficient feature extraction capabilities.

Parameter sensitivity experiment

To determine the relative importance between TKD and SKD, parameter sensitivity experiments were conducted to explore the influence of the proportion of the two components of TSKD on recognition accuracy. Table 7 presents the effect of varying ratios of the two parts of the loss functions in Eq. (2) on the SER of the student model, with Transformer and Con_12_512 serving as the teacher and student model, respectively. The experimental results show that the optimal knowledge distillation effect of 3.98% is achieved when the ratio of TKD to SKD is 1:1, indicating that the two parts of TKD and SKD are of equal value for effective knowledge distillation.

Effect of beam search size on recognition accuracy and transcription speed

The efficiency and accuracy of speech recognition depend on the beam search size as well as the model capacity. Table 8 details the effect of beam search size on recognition accuracy and transcription speed of the speech recognition student model variants. The results indicate that expanding the beam search size generally decreases SER, considering the models have a greater chance of predicting the optimal sequence of characters. However, this enhancement comes at the cost of a substantial rise in transcription time.

Specifically, when the beam search size is less than 3, the SER decreases dramatically as it increases. In contrast, when it is greater than 3, the SER decreases slowly and even increases in certain instances. When the beam search size is 10, transcription takes approximately twice as long as when it is 3. Nevertheless, the performance gain is limited to a maximum of 0.13% (Con_12_512) absolute recognition accuracy. In a word, by restricting the beam search size from 10 to 3, the average recognition speed can generally be doubled with comparable model accuracy.