Improving English-Chinese translation using the Kolmogorov-Arnold Transformer

Sequence-to-sequence framework

The most common method in NMT is the Sequence-to-Sequence (Seq2Seq) framework, as shown in Fig. 1. Here, the encoder processes an input sentence sequentially, mapping words to embeddings and fine-tuning them into contextualized representations. The encoded representations have the sentence’s meaning and are passed to the decoder. The decoder used to generate words, constructs new representations that determine the next output word. The encoder and decoder architectures can be recurrent neural networks (RNNs) and convolutional neural networks (CNNs).

In the Seq2Seq framework, the encoder processes an input sequence $S=(s_1,s_2,\dots ,s_N)$ and encodes it into a context representation $z$. The decoder then generates the corresponding output sequence $T=(t_1,t_2,\dots ,t_M)$ using $z$ as a reference. The encoder computes a sequence of hidden states as follows:

(3) $$h_n=\varphi (W_e{s}_n+U_e{h}_{n-1}+b_e),$$where $h_n$ represents the hidden state at step $n$, $W_e,U_e,b_e$ are trainable parameters, and $\varphi$ is an activation. In the decoder block, each output word is generated iteratively based on previous outputs and the context:

(4) $$d_m=\varphi (W_d{t}_{m-1}+U_d{d}_{m-1}+b_d)$$where $d_m$ denotes the decoder’s hidden state at step $m$. $W_d,U_d,b_d$ are trainable parameters, and $\varphi$ is an activation.

Attention mechanism

Recently, fully attention-driven NMT models have demonstrated exceptional performance, with attention mechanisms evolving from an auxiliary role to a primary means of extracting text features. A prime example of this evolution is the Transformer (Vaswani et al., 2017) architecture, which relies entirely on self-attention and feed-forward layers. Unlike RNNs or CNNs, the Transformer processes entire sentences simultaneously, rather than sequentially, resulting in more efficient and accurate translations. Stacking multiple Transformer layers further enhances translation quality. In self-attention layers, attention is computed across both the encoder and decoder. The alignment score, which determines how well different input positions align with a target position, is calculated as follows:

(5) $$e_{ji}=a(s_{j-1},h_i),a_{ji}=\frac{\exp (e_{ji})}{{\sum }_{k=1}^m\exp (e_{ki})},$$where $e_{ji}$ is the alignment score and $a$ is a function that calculates how well an input word at position $i$ aligns with an output word at position $j$. These calculations are influenced by the decoder’s previous step hidden state, $s_{j-1}$, and the encoder’s hidden state, $h_i$. The attention weights are applied to the encoder’s hidden states to generate a context vector:

(6) $$c_j=\sum _{i=1}^n{a}_{ji}{h}_i.$$

This context vector, along with the previous hidden state and generated word, helps compute the next decoder hidden state:

(7) $$s_j=g(s_{j-1},y_{j-1},c_j),$$where $g$ is an activation function, $y_{j-1}$ is the embedding of the previous word, and $s_{j-1}$ is the last decoder state. Finally, the next target word is predicted using a feed-forward and softmax layer:

(8) $$P(y_j\mid y_{<j},x)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(f(s_j,y_{j-1},c_j)).$$

Data preprocessing

The preprocessing of bilingual data follows a structured pipeline approach to ensure high-quality input for the translation model. This process involves multiple steps, including data loading, tokenization, vocabulary generation, sequence encoding, and batch preparation. The dataset consists of parallel English-Chinese sentence pairs, with each line containing an English sentence alongside its corresponding Chinese translation. To facilitate model training, tokenization is applied to each sentence, incorporating special tokens: BOS (Beginning of Sentence) and EOS (End of Sentence). English sentences undergo lowercasing before tokenization, while Chinese sentences are segmented at the character level.

To enhance neural network training, separate vocabularies are constructed for English and Chinese. The most frequent words, up to a maximum of 30,000, are retained, whereas less frequent words are substituted with a special UNK (unknown) token. Each token is then mapped to a unique numerical identifier, forming a word-to-index dictionary. A reverse index is also established to facilitate decoding. Sentences are subsequently converted into sequences of numerical IDs according to the constructed dictionaries. To improve training efficiency, sentences are sorted by length before batching, thereby minimizing unnecessary padding. For mini-batch training, sentences are grouped into fixed-size batches. As sentence lengths may vary within a batch, padding is applied to standardize their length. Additionally, a mask is generated to differentiate actual tokens from padding elements. This step ensures that the model processes data efficiently while reducing redundant computations on padded tokens. Finally, the dataset is split into 10,477 sentences for training, 1,177 for validation, and 2,763 for testing. This stratification ensures that the model has sufficient data to learn while allowing for performance tuning and unbiased evaluation on unseen examples.

Kolmogorov-Arnold networks

The Kolmogorov-Arnold network (KAN) is a neural network architecture inspired by the Kolmogorov-Arnold representation theorem. This theorem states that any multivariate continuous function defined on a bounded domain can be expressed as a finite composition of continuous univariate functions and addition. This leads to a novel approach in neural network design, where nonlinear transformations are performed through learnable univariate functions. In KAN, each layer consists of a set of learnable univariate functions acting as activation functions on the edges. Specifically, a Kolmogorov-Arnold layer with $d_{\mathrm{i}\mathrm{n}}$-dimensional inputs and $d_{\mathrm{o}\mathrm{u}\mathrm{t}}$-dimensional outputs is defined as:

(9) $$f(\mathbf{x})=\mathrm{\Phi }\circ \mathbf{x}=\left[\sum _{i=1}^{d_{\mathrm{i}\mathrm{n}}}{\varphi }_{i,1}(x_i)\cdots \sum _{i=1}^{d_{\mathrm{i}\mathrm{n}}}{\varphi }_{i,d_{\mathrm{o}\mathrm{u}\mathrm{t}}}(x_i),\right]$$where $\mathrm{\Phi }$ is a matrix of univariate functions ${\varphi }_{i,j}$, to enable a learnable nonlinear mapping of the input. A KAN is built by stacking several such layers, resulting in sequential compounding of transformations:

(10) $$KAN({\mathbf{x}}_0)={\mathrm{\Phi }}_{L-1}\circ {\mathrm{\Phi }}_{L-2}\circ \cdots \circ {\mathrm{\Phi }}_0\circ {\mathbf{x}}_0.$$

To enhance the flexibility of the learnable univariate functions, KAN parameterizes $\varphi (x)$ as a combination of the SiLU activation function and a B-spline function:

(11) $$\varphi (x)=w_b\cdot \mathrm{s}\mathrm{i}\mathrm{l}\mathrm{u}(x)+w_s\cdot \mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}(x),$$where

(12) $$\mathrm{s}\mathrm{i}\mathrm{l}\mathrm{u}(x)=\frac{x}{1+e^{-x}},\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}(x)=\sum _i{c}_i{B}_i(x).$$

Justification for model choice

Typical NMT models, such as Transformer-based architectures, have achieved remarkable success in English-Chinese translation. However, they perform poorly in capturing long-range dependencies, handling complex syntactic structures, and maintaining translation consistency across linguistically distant language pairs. Studies based on the self-attention mechanism allow models to capture global context; however, they do not effectively model hierarchical or compositional linguistic structures, which often results in inconsistent translations. In addition, most previous models still face challenges in computational efficiency as well as accuracy when translating long documents or operating under high-speed processing demands. To address these limitations, we employ the Kolmogorov-Arnold Transformer (KAT), a novel architecture whose purpose is to boost the expressiveness and efficiency of deep learning models in sequence-to-sequence tasks.

KAT is a hybrid architecture that integrates the strengths of Transformers and Kolmogorov–Arnold Networks, enabling the representation of complex functions through a structured decomposition process. This capability is particularly advantageous in machine translation, where capturing intricate relationships between source and target languages is essential. As a result, KAT can handle long sentences in both English and Chinese, enhancing the model’s ability to produce accurate and contextually coherent translations. This architecture improves parameter efficiency and reduces the risk of overfitting while maintaining high fidelity in translation. It is especially adept at capturing syntactic and semantic relationships in Chinese due to challenges such as word segmentation and character-based representation. By leveraging KAT, we aim to enhance translation quality through improved contextual understanding, reduced computational complexity, and greater fluency in both English and Chinese. This choice offers a robust, scalable, and interpretable approach to machine translation, advancing the capabilities of current NMT systems.

Model architecture

In this article, we propose the customed transformer architecture by replacing FFN with a KAN layer to enhance the expressiveness of the transformation. Overview of this architecture is described in Fig. 2.

Given an input sequence X consisting of word-tokenized representations, discrete tokens are initially mapped into dense vector spaces through an embedding layer. Positional encodings are incorporated to preserve order information, as the Transformer model does not inherently capture positional dependencies:

(13) $$X=\mathrm{E}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{L}\mathrm{o}\mathrm{o}\mathrm{k}\mathrm{u}\mathrm{p}(X)+\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}(X),$$where $X\in {\mathbb{R}}^{\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\mathrm{\_}\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}\times \mathrm{s}\mathrm{e}\mathrm{q}\mathrm{\_}\mathrm{l}\mathrm{e}\mathrm{n}\times \mathrm{e}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{d}\mathrm{\_}\mathrm{d}\mathrm{i}\mathrm{m}}$. The core component of the Transformer is the self-attention mechanism, which enables the model to capture long-range dependencies between words. The input X is projected into Query (Q), Key (K), and Value (V) representations. These representations are then used to compute the self-attention output:

(14) $$\alpha =\mathrm{S}\mathrm{e}\mathrm{l}\mathrm{f}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q,K,V),Q=X{W}_Q,K=X{W}_K,V=X{W}_V,$$where $W_Q,W_K,W_V$ are learnable weights. To facilitate stable training and prevent gradient vanishing, we incorporate residual connections followed by layer normalization:

(15) $$X_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(X_{\mathrm{a}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}+\alpha ).$$

Next, each token representation is then passed through a KAN that is explained in next subsection and a final residual connection and normalization step is applied:

(16) $$X_{\mathrm{h}\mathrm{i}\mathrm{d}\mathrm{d}\mathrm{e}\mathrm{n}}=\mathrm{K}\mathrm{A}\mathrm{N}(X_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}).$$

At the end of multiple encoder layers, the output representation $X_{\mathrm{o}\mathrm{u}\mathrm{t}}$ captures rich contextual information, which is then passed to the decoder for translation.

(17) $$X_{\mathrm{o}\mathrm{u}\mathrm{t}}=\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(X_{\mathrm{h}\mathrm{i}\mathrm{d}\mathrm{d}\mathrm{e}\mathrm{n}}+X_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}).$$

This structured processing allows the model to effectively encode syntactic and semantic dependencies, making it well-suited for NMT tasks.

Evaluation metrics

In this study, we use evaluation metrics including Bilingual Evaluation Understudy (BLEU), METEOR, Translation Edit Rate (TER), and Character-level F-score (ChrF) to assess the performance of our translation models. BLEU focuses on n-gram overlap between the translation and reference, making it suitable for overall evaluation but potentially overlooking semantic and contextual aspects. METEOR supplements this by incorporating synonym matching and sentence-level analysis, providing a better reflection of translation accuracy and fluency. TER measures the number of edits required to transform the system output into the reference translation, offering insight into post-editing effort and specific differences. ChrF, based on character-level F-score, enables finer-grained evaluation of minor errors and is especially suitable for languages with complex word structures. Comparing and combining these metrics offers a more comprehensive perspective on translation quality, thereby enhancing the accuracy of model performance assessment. They characterize their mathematical formulas as:

(18) $$\mathrm{B}\mathrm{L}\mathrm{E}\mathrm{U}=\exp \left(\sum _{n=1}^N{w}_n\log p_n\right)\times min(1,e^{1\text{-}r/c}),$$where $p_n$ represents the precision of n-grams, $w_n$ is the weight assigned to each n-gram, r is the reference translation length, and c is the candidate translation length.

(19) $$\mathrm{M}\mathrm{E}\mathrm{T}\mathrm{E}\mathrm{O}\mathrm{R}=F_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}\times (1-\mathrm{p}\mathrm{e}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{t}\mathrm{y}),$$where $F_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}$ is a weighted harmonic mean of precision P and recall R, and the penalty term accounts for the number of chunks that are not in the correct order.

(20) $$\mathrm{T}\mathrm{E}\mathrm{R}=\frac{\mathrm{E}}{\mathrm{R}},$$where E represents the number of insertions, deletions, substitutions, and shifts needed, and R is the total number of words in the reference translation. A lower TER score indicates a more accurate translation.

(21) $$\mathrm{C}\mathrm{h}\mathrm{r}\mathrm{F}=(1+{\beta }^2)\cdot \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{{\beta }^2\times \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}},$$where Precision and Recall are calculated over character-level n-grams, and $\beta$ is a parameter balancing the two measures.

Comparative analysis

Following the guidelines for comparative studies on various network models, the hardware and software environments, along with other experimental parameters, remain constant, with the network model being the only variable. This study employs a statistical machine translation model (Osborne, 2011) as the baseline system. In addition, various neural network architectures, including RNNs (Datta et al., 2020), long short-term memory (LSTM) networks (Ramaiah, Datta & Agarwal, 2022), and bidirectional LSTM (BiLSTM) (Tanvir et al., 2023) models, hybrid models such as RNN+Attention (Shi, Meng & Wang, 2019), LSTM+Attention (Wu & Xing, 2024), convolutional sequence to sequence learning (CSSL) (Gehring et al., 2017), BART (Lewis et al., 2019) and Transformer (Badawi, 2023) model are also analyzed and evaluated. These models developed under the same environment and parameter settings.

The training configurations for all baseline models were kept consistent to ensure fair comparison. Specifically, we used the Adam optimizer with a learning rate of 1e−4, a batch size of 128. The hidden dimension was set to 256, and for attention-based models (Transformer, BART), we used four attention heads. Encoder-decoder models (RNN, LSTM, CSSL) followed a 2-layer encoder and 1-layer decoder architecture. These settings were applied uniformly unless otherwise specified for a particular model.

Benchmarking KAN integration

To assess the impact of incorporating KAN into the KAT architecture, we conducted comparative experiments against a baseline Transformer model. The models were evaluated on the Bilingual MELD dataset using standard translation quality metrics, including BLEU-4, METEOR, TER, and ChrF. BLEU-4 measures n-gram precision against reference translations, METEOR evaluates alignment based on synonymy and word order, TER quantifies the number of edits needed to match the reference, and ChrF assesses character-level translation quality. Performance improvements across all four metrics were used to validate the effectiveness of the KAN integration.