Leveraging large language models for spelling correction in Turkish

Spell checkers tools with multilingual support

Early studies predominantly have employed rule-based models that relied on a predefined set of rules for spelling error detection and correction, such as the SPELL system (Pollock & Zamora, 1984). In the following years, studies (Norvig, 2007; Garbe, 2012; Atkinson, 2019) have generally focused on candidate generation for correction, employing an edit distance metric along with a lookup table. The introduction of n-grams, a core concept in natural language processing, has greatly improved the ability of models to execute context-sensitive corrections (Jurafsky & Martin, 2024). To incorporate the context of adjacent words, JamSpell (Ozinov, 2019) utilizes a 3-gram language model alongside a dictionary search for effective word-level correction. Moreover, morphology analyzers have been utilized to address misspellings by generating possible word forms for unrecognized tokens through the affix rules and dictionaries. Hunspell (Tron et al., 2005) generates suggestions by leveraging morphological analysis rules along with dictionary-based data. By providing language-specific dictionaries for these approaches, such systems enable support for multiple languages. In the study (Gupta, 2020), the Symmetric Delete algorithm (Garbe, 2012) is applied to predict candidates for comparison with dictionaries that utilize data structures such as Tries, DAWGs (Directed Acyclic Word Graphs), and BK Trees (Burkhard-Keller Trees) (Burkhard & Keller, 1973). Subsequently, misspellings can be corrected using the n-gram conditional probabilities of the candidate suggestions.

Spell checkers based on large language models for English

Relatively few studies have utilized large language models to address misspellings in English texts. Li, Liu & Huang (2020) involves training a vanilla transformer model (Vaswani et al., 2017) as a character-level encoder (Näther, 2020), while also integrating word-character and subword encoders to leverage character-level embeddings alongside subword representations. In Neuspell (Jayanthi, Pruthi & Neubig, 2020), a contextual spell correction tool is introduced that supports neural models such as Embeddings from Language Model (ELMO) (Peters et al., 2018), long short-term memory (LSTM) (Hochreiter, 1997), and pre-trained BERT, allowing for the consideration of surrounding word representations.

Turkish morphological spell checkers

With the aim of presenting the first spelling checker tool for Turkish, a morphological root-driven parser is introduced (Solak & Oflazer, 1993), specifically developed for Turkish words. This parser has also been designed for spelling correction in online texts. It incorporates complex word formations and various phonetic harmony rules for the correction task, including the determination of roots morphophonemic checks, and morphological parsing. Zemberek (Akın & Akın, 2007) was developed as an open-source NLP library for Turkic languages. It offers fundamental NLP functions, including spell checking, word suggestion, stemming, morphological parsing, word construction, conversion of ASCII-only text, and syllable extraction. The library relies on a root dictionary-based parser and aims to construct a specialized DAWG tree to represent the deformed states of root words as a dictionary. Based on these trees, the Zemberek spell checker corrects the given text if any matches are found by eliminating the need to search through all possible matches. In another study, a rule-based approach has been proposed to correct text by applying multiple normalization layers that address specific types of errors, aiming to obtain the canonical form of a noisy word (Torunoğlu-Selamet & Eryiğit, 2014). MUKAYESE (Safaya et al., 2022) has been introduced as a set of NLP benchmarks aimed at addressing the shortage of structured benchmarks for Turkish. Moreover, this study focuses context-free spell checking and correction for single-words, presenting a comparison among their dictionary-based Hunspell, another dictionary based Hunspell model, and the Zemberek, using their single-text correction dataset, TRSPELL10. In Koksal et al. (2020), #Turki$hTweets is introduced as a publicly accessible small-scale dataset of tweets collected to form a natural noisy data. This study has evaluated two morphology-based approaches, along with edit distance and rule-based models, to provide a preliminary analysis. A sequence-to-sequence model leveraging LSTM architecture has been proposed for text normalization, with inputs derived from a character-based tokenizer that processes the first three consonant letters of misspelled words (Büyük, 2020). Utilizing LSTM and bidirectional LSTM (BiLSTM) networks, the study (Aytan & Şankar, 2023) has focused on the detection of misspelled words rather than correction by evaluating the effect of different tokenizers. In a recent study (Oral et al., 2024), a Turkish spelling corrector designed for e-commerce search engines was proposed. This approach utilizes a character-level transformer to generate representation-based candidates, enhancing the accuracy of search queries.

Norvig algorithm

The Norvig algorithm (Norvig, 2007) relies on generating candidates based on possible edits that are one or two edits away from the word $w$. It determines the most likely spelling correction for a given word $w$ in a dictionary $\mathcal{D}$ by generating possible edits, ${\mathrm{E}}_w$, through deletion, insertion, substitution and permutation, as given in Eq. (1).

(1) $${\mathrm{E}}_w=\left\{\mathrm{I}(w),\mathrm{D}(w),\mathrm{P}(w),\mathrm{S}(w)\right\}.$$

Next, the algorithm generates candidate corrections, ${\mathrm{C}}_w$ based on distance to these edits using Levenshtein Distance ( ${\mathrm{d}}_{\mathrm{L}}$) (Levenshtein, 1966).

(2) $${\mathrm{d}}_{\mathrm{L}}(w,c)=\{\begin{array}{cc}|w|\hfill & \mathrm{i}\mathrm{f}|\mathrm{c}|=0,\hfill \\ |c|\hfill & \mathrm{i}\mathrm{f}|\mathrm{w}|=0,\hfill \\ {\mathrm{d}}_{\mathrm{L}}(\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}(c),\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}(w))\hfill & \mathrm{i}\mathrm{f}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}(\mathrm{c})=\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}(\mathrm{w}),\hfill \\ 1+min\left\{\begin{array}{c}{\mathrm{d}}_{\mathrm{L}}(\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}(c),w),\hfill \\ {\mathrm{d}}_{\mathrm{L}}(c,\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}(w)),\hfill \\ {\mathrm{d}}_{\mathrm{L}}[(\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}(c),\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}(w))\hfill \end{array}\right\}\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$

(3) $${\mathrm{C}}_w=\left\{c\mid min({\mathrm{d}}_{\mathrm{L}}(w,c)),c\in {\mathrm{E}}_w\right\}$$where the tail of a word $w$, denoted as $\mathrm{t}\mathrm{a}\mathrm{i}\mathrm{l}(w)$, is defined as the substring consisting of all characters except the first one. The head of a word $w$, $\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}(w)$, is the first character of $w$.

To rank these candidates, the algorithm employs a probabilistic model. The goal is to select the candidate correction $c$ that maximizes the probability of $c$ being the intended correction, given the original word $w$. This is achieved by applying Bayes’ Theorem, which simplifies to maximizing the product of two probabilities: the language model probability, $P(c)$, representing the frequency with which $c$ appears in text, and the error model probability, $P(w\mid c)$, which quantifies the likelihood that $w$ is an error for $c$. As given in Eq. (4), an $\arg max$ function is employed to choose the candidate with the highest combined probability, considering a set of possible corrections and using these models to identify the most contextually appropriate correction $\hat{w}$ for the input word $w$.

(4) $$\hat{w}=\arg \underset{c\in {\mathrm{C}}_w}{max}P(c\mid w)=\arg \underset{c\in {\mathrm{C}}_w}{max}P(c)P(w\mid c).$$

Aspell

Aspell is a spell checker and automatic correction tool that is specifically designed for use with the GNU operating system. It generates suggestions for misspelled words by leveraging both edit distance metrics and contextual analysis, with frequency-based ranking employed to ensure the accuracy of corrections. The Aspell scoring function, $S(c)$, is based on the weighted average of edit distances between the misspelled word and entries in the lexicon, which includes a collection of words, morphological information about word forms, and additional data such as frequency and part-of-speech tags. Equation (5) computes the weighted average edit distance, $d_{wL}$, using weights and the corresponding edit distances.

(5) $$d_{wL}(w,c)=\frac{1}{n}\sum _{i=1}^n{\mathrm{W}}_i\cdot d_L(w,c_i)$$where ${\mathrm{W}}_i$ represents the weight of $i$th edit operation and $c_i$ denotes the candidate resulting from the $i$th edit. The final suggestions come from the $\arg min$ of Aspell scoring, $\arg \underset{c_i\in C}{min}S(c_i)$.

Hunspell

Hunspell (Tron et al., 2005) is an autocorrect spell checker and morphological analyzer widely utilized in numerous applications and platforms, including Mac OS X, Safari, OpenOffice, Firefox, Chrome, and Adobe products. In addition, it is integrated into various tools such as the XML Copy Editor and LyX. Hunspell supports a wide range of languages with complex morphological features. This spell checker is based on a two-part dictionary system, consisting of a dictionary $\mathcal{D}$ and an affix file. Hunspell’s strength lies in its use of affix compression and morphological rules to effectively generate variations. The algorithm begins with tokenization, followed by a comparison of each token against the entries in the dictionary. When a token is not found, Hunspell generates possible word forms, $w^{\prime }$, applying affix rules contained in the affix file that add or remove prefixes and suffixes after finding the root. In agglutinative languages like Turkish, affix files consist of possible suffixes to generate candidate words. For example, the root word “gel”, {“iyor”, “di”, “ecek”} suffixes can generate candidates like “geliyor, geldi, gelecek”. It also supports compound words by breaking them down into valid subwords on the basis of its dictionaries and affix rules. Then, it performs a morphological analysis to match these generated forms with valid entries in the dictionary.

Symmetric delete spelling correction algorithm (SymSpell)

SymSpell (Garbe, 2018) is an exceptionally fast spell checker library that offers performance up to 1 million times faster than traditional methods through its Symmetric Delete Spelling Correction Algorithm. It improves the Norvig algorithm by generating all potential candidates $c$ exclusively through delete operations, $c\in \left\{\mathrm{I}(w)\right\}$. Instead of directly addressing transpositions, replacements, and insertions within the input phrase, SymSpell transforms these operations into deletions of dictionary terms. This method has proven to be significantly more efficient, as replacements and insertions are both computationally expensive and language-dependent. The candidate word set $C_w$ and the candidate dictionary set $C_d$ can be derived from the candidates $D(w)$ and ${\cup }_{d\in \mathcal{D}}\mathrm{D}(d)$ that have the minimum edit distance $min({\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w,c))$, obtained by deletion.

This technique optimizes the process of generating edit candidates and searching the dictionary for a given Damerau-Levenshtein distance ( ${\mathrm{d}}_{\mathrm{D}\mathrm{L}}$) (Damerau, 1964) which can be computed by dynamic programming recurrence for $0\le i\le |w|,0\le j\le |c|$, ${\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w[i],c[0])=i$ and ${\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w[0],c[j])=j$:

(6) $${\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w[i],c[j])=min\{\begin{array}{c}{\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w[i-1],c[j-1])+{\mathbf{1}}_{w=c}(w,c)\hfill \\ {\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w[i],c[j-1])+1\hfill \\ {\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w[i-1],c[j])+1\hfill \\ {\mathrm{d}}_{\mathrm{D}\mathrm{L}}(w[k-1],c[l-1])+(i-k-1)+1+(j-l-1)\hfill \end{array}$$

Here, the indicator function ${\mathbf{1}}_{w=c}(w,c)$ is 1 if the condition $w=c$ holds. The $\arg max$ of candidate intersections, $\arg \underset{d\in \mathcal{D}}{max}\left|{\mathrm{C}}_w\cap {\mathrm{C}}_d\right|$, yields the predicted correction.

JamSpell

JamSpell (Ozinov, 2019) employs a variant of the SymSpell algorithm integrated with a 3-gram language model to incorporate contextual information, thereby enhancing performance. JamSpell is also based on the principles of Norvig. The algorithm leverages SymSpell’s efficient edit candidate generation and dictionary search methods, thereby improving the performance of SymSpell. Moreover, the Bloom filter is integrated with SymSpell to store the index in an efficient manner. Deletions from the dictionary are stored in the Bloom filter. When searching for candidates, deletions are first applied to the original word up to the desired depth, similar to the method used by SymSpell. However, unlike SymSpell, insertions are performed for each deletion, and the resulting words are subsequently checked against the original dictionary. The Bloom filter is utilized to access the index of deletions and to skip insertions for deletions that are not present.

In addition, it integrates an n-gram language model to further refine and filter word-level corrections based on contextual information. Rather than employing a naive approach that calculates edit distances against a corpus-based dictionary, the use of n-gram dictionaries enables the spelling corrector to consider the frequency of sequences of $n$ words occurring together, thereby enhancing predictions for out-of-vocabulary words. This involves constructing comprehensive collections of n-gram sequences, consisting of $n$ words or characters, to analyze word co-occurrences for predicting the correct words. JamSpell employs trigrams (3-grams) which are sequences of three consecutive elements from a given text to account for the context. In a trigram model, the conditional probability of the word $w_i$ can be defined by the frequencies of $w_{i-1}\mathrm{a}\mathrm{n}\mathrm{d}{w}_{i-2}$:

(7) $$P(w_i\mid w_{i-2},w_{i-1})=\frac{f(w_{i-2},w_{i-1},w_i)}{f(w_{i-2},w_{i-1})}$$where $f(w_{i-2},w_{i-1},w_i)$ is the frequency of the trigrams $(w_{i-2},w_{i-1},w_i)$ in the corpus. For spelling correction, the context score S is computed and suggestions are ranked in decreasing order of this score, returning the top $k$ suggestions. The context score of n-gram $S_n$ is given:

(8) $$S_n=W_n\sum _{j=1}^{n-1}P(w_i\mid w_{i+j-1})$$where $W_n$ is the weight for the $n$-gram score. For this case, trigram context score ( $S_3$) is computed as a context score. If some n-grams $w_{i-n},\dots ,w_{i-1},w_i$ were not found in the text, the frequency of occurrence of the n-gram $f(w_{i-n},\dots ,w_{i-1},w_i)$ will be zero and the n-gram will immediately receive a zero probability. To address this issue in JamSpell the probability of each token in a trigram, $P(w_i\mid w_{i-1},w_{i-2})$ is considered as the product of token probabilities of all orders, $P(w_i\mid w_{i-2},w_{i-1})P(w_i\mid w_{i-1})P(w_i)$ for smoothing the trigram probability. It also addresses the issue of high memory consumption by employing a perfect hash (Guthrie, Hepple & Liu, 2010) based on the Hash, Displace, and Compress (CHD) algorithm (Belazzougui, Botelho & Dietzfelbinger, 2009) to store information about n-grams. All of these allow JamSpell to balance computational efficiency with the accuracy of contextually appropriate spelling suggestions.

Zemberek

Zemberek (Akın & Akın, 2007) is an open-source natural language processing library specifically developed for Turkish. It offers tools for various tasks, including tokenization, stemming, named entity recognition (NER), language identification, and grammatical analysis. It is designed to accommodate the suffix-based nature of Turkish. It also provides normalization as a basic spell-checking feature. Since Zemberek relies on a root word dictionary, it requires a root-finding approach. A special DAWG tree is constructed, where the roots of the tree correspond to the root words, which are used as the dictionary. As a root selector, three root selectors are employed in Zemberek. For a given word, the first selector is based on a standard, strict root selection approach, functioning as a dictionary-based top-down parser. The second, an error-tolerant parser, applies the Damerau-Levenshtein (a default similarity algorithm) or the Jaro-Winkler string similarity algorithm to select candidate roots. The final, ASCII-tolerant parser generates the root by considering letters that are not included in the ASCII character set. To provide the best suggestions, the results are ranked based on the frequency of root word usage in Zemberek.

Background

Bidirectional encoder representations from transformers (BERT)

BERT (Devlin, 2018) is an encoder-only model built with stacked transformer blocks. Each transformer block includes two main sublayers: a multi-head attention mechanism and fully connected feed-forward layers. Each transformer layer starts with a multi-head attention mechanism which allows the model to simultaneously focus on different parts of the input sequence. This is followed by a residual connection (He et al., 2016) and layer normalization where the residual connection adds the initial input of the attention mechanism to its output. The next step involves processing this output through a feed-forward network, which includes two linear transformations with a ReLU activation function in between. Then, another residual connection and layer normalization are applied after the feed-forward network. The final output is then passed to the next transformer layer to further enhance the representations at each level, which are subsequently utilized for various downstream NLP tasks.

BERT is designed to serve as both a pre-training model for unsupervised tasks and a fine-tuning model for various downstream applications. During the pre-training phase, BERT was trained on unlabeled data using two unsupervised learning objectives: Masked language modeling (MLM) and next sentence prediction. There are two primary versions of the BERT model: $\mathrm{B}\mathrm{E}\mathrm{R}{\mathrm{T}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}$ and $\mathrm{B}\mathrm{E}\mathrm{R}{\mathrm{T}}_{\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}}$. The $\mathrm{B}\mathrm{E}\mathrm{R}{\mathrm{T}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}$ model consists of 12 transformer layers, each with a hidden size of 512. The block representation of the $\mathrm{B}\mathrm{E}\mathrm{R}{\mathrm{T}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}$ model, shown in Fig. 1, illustrates the components of each transformation layer. The $\mathrm{B}\mathrm{E}\mathrm{R}{\mathrm{T}}_{\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}}$ model, however, doubles the number of layers and hidden size of each layer, resulting in a total of 340 million parameters.

Masked language modeling based model

In the MLM task, 15% of the tokens were randomly masked allowing the model to predict the masked words, which were replaced with a [MASK] token. For MLM, the objective is to predict these missing words based on the context provided by the surrounding words. Pre-training on masked word prediction enables models to gain a deep understanding of language context. This understanding can then be leveraged for other NLP tasks through fine-tuning.

In this study, misspelling correction is treated as a masked prediction, utilizing a pre-trained BERT-based MLM which is demonstrated in Fig. 2. As seen in the given figure, [MASK] tokens should be generated before feeding the inputs into the BERTurk encoder model. To generate [MASK] tokens for out-of-vocabulary (OOV) words, the study utilizes the Turkish transformer model from spaCy (https://spacy.io/, https://huggingface.co/turkish-nlp-suite) (Altinok, 2023). This work adapts the BERTurk model (Schweter, 2020) into the ContextualSpellCheck (https://github.com/R1j1t/contextualSpellCheck) (Goel, 2021) as a MLM. The input sequence $S=(w_1,w_2,\dots ,w_n)$ is masked in accordance with the existence of the word $w_i$ in the dictionary $\mathcal{D}$ using spaCy:

(14) $$S[i]=[\mathrm{M}\mathrm{A}\mathrm{S}\mathrm{K}],\mathrm{\forall }{w}_i\in S\mathrm{s}\mathrm{u}\mathrm{c}\mathrm{h}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{t}{w}_i\notin \mathcal{D},0\le i\le |S|$$

(15) $$S=(w_1,w_2,\dots ,[\mathrm{M}\mathrm{A}\mathrm{S}\mathrm{K}],\dots ,w_n)$$

By adapting the ContextualSpellCheck pipeline, input sequences with [MASK] tokens are provided to a MLM, referred to as TransformerMLMEncoder for the remainder of the article, yielding embeddings for the correction task. Subsequently, a FFNN follows the TransformerMLMEncoder. This network acts as a linear layer to generate the logits for the sequence tokens. The softmax function transforms the logits into probabilities. Candidates ${\mathrm{C}}_w$ are the top $k$ predictions sorted by their highest probabilities for correction. Accordingly, ContextualSpellCheck is revised to suggest a word if there is a candidate ${\mathrm{C}}_i$ with an edit distance of 1. In the other case, the model outputs the given input token to prevent false positive predictions. The model based on the masked language encoder is presented below:

(16) $$h=\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{M}\mathrm{L}\mathrm{M}\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}(S)$$

(17) $$\mathrm{F}\mathrm{F}\mathrm{N}\mathrm{N}(h)=W_{s}h+b_s$$

(18) $$P(S)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\mathrm{F}\mathrm{F}\mathrm{N}\mathrm{N}(h))$$

(19) $$C_{w_i}=\mathrm{S}\mathrm{o}\mathrm{r}\mathrm{t}(P(S_i){{,}^{\mathrm{\text{'}}}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}}^{\mathrm{\text{'}}})[:k]$$

(20) $${\hat{w}}_i=\{\begin{array}{cc}c\mid min(d_{DL}(w_i,c))\mathrm{\forall }c\in C_{w_i},\hfill & \mathrm{i}\mathrm{f}min(d_{DL}(w_i,c))\le 1\hfill \\ w_i,\hfill & \mathrm{e}\mathrm{l}\mathrm{s}\mathrm{e}\hfill \end{array}$$where $W_s$ and $b_s$ are the weights and bias for the linear layer. $P(S_i)$ denotes the probability of the $i$th logit.

Subword encoder based model

Encoder-only LLMs are also utilized with a FCNN layer to correct sentences by regenerating whole subwords. The comparison of this model with MLM based correction model is visualized in Fig. 3. Rather than predicting [MASK] tokens, the encoder predicts each subword as a token in the given sequence. The encoder of this model will be referred to as TransformerEncoder. TransformerEncoder encodes a sequence of subword tokens S to $h_e$ embeddings. Dropout (Srivastava et al., 2014) is applied within the TransformerEncoder for regularization. Thereafter, subword embeddings, $h_{\mathrm{s}\mathrm{u}\mathrm{b}}$, are obtained by concatenating the embeddings of the subwords $w_{\mathrm{s}\mathrm{u}{\mathrm{b}}_k}^i$ in the given sequence S if a token is a subword of $w_i$, as follows:

(21) $$h_e=\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}(S)$$

(22) $$h=\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(h_{{\mathrm{e}}_0},\dots ,h_{{\mathrm{e}}_k})$$where $h_{{\mathrm{e}}_k}$ denotes the embeddings of $w_{{\mathrm{e}}_k}$.

Subsequently, the FFNN, a single linear layer, predicts the logits of the merged token embedding. These logits are converted to probabilities, $P(S)$, by applying the softmax function to generate predictions for the words $w_i$ in the sequence S. The corrected sequence $\hat{S}$ is derived from the argmax of $P(S)$.

(23) $$\mathrm{F}\mathrm{C}\mathrm{N}\mathrm{N}(h)=W_{s}h+b_s$$

(24) $$P(S)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\mathrm{F}\mathrm{C}\mathrm{N}\mathrm{N}(h))$$

(25) $$\hat{S}=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}(P(S))$$

Encoder-decoder LLM based model

The sentence spelling correction task will also be treated as a sequence-to-sequence generation task using pairs of noised and correct sentence pairs. Consequently, encoder-decoder LLMs pre-trained for downstream NLP tasks, such as machine translation, question answering, and next sentence prediction, are also included in this study.

The encoder-decoder LLMs have architectures derived from the framework proposed by Vaswani et al. (2017). The encoder consists of multiple blocks, each containing two components: a self-attention layer and a feed-forward neural network. Self-attention layers are bidirectional, similar to those in the BERT model. Therefore, the encoder model can be referred to as a bidirectional encoder. The decoder-only model can function as a LLM, specifically a model trained for next sentence prediction like OpenAI GPT (Radford, 2018). GPT relies on a left-to-right architecture, enabling it to attend only to the tokens from previous steps within the self-attention layer. In an autoregressive decoder like GPT, a token is sampled from the predicted distribution of the decoder, and then this token is fed back into the decoder to generate the prediction for the subsequent output at each time step. An encoder-decoder model combines left-to-right autoregressive language model and bidirectional encoder with MLM objective as given in Fig. 4. The decoder in such a sequence-to-sequence model has a similar structure to that of the encoder. In the default small-sized encoder-decoder LLM, both the encoder and decoder consist of six blocks. Furthermore, they have similar sizes and configurations as well.

In a sequence-to-sequence model designed for sentence spelling correction, the encoder, referred to as BidirectionalEncoder in Eq. (26), takes a sequence S that may contain erroneous words as input while the decoder, namely AutoregressiveDecoder, produces a corresponding denoised sequence in an autoregressive manner. Initially, the bidirectional encoder maps an input sequence of tokens to the to the sequence of embeddings $h$:

(26) $$h=\mathrm{B}\mathrm{i}\mathrm{d}\mathrm{i}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}(S)$$

(27) $$\widehat{S_t}=\mathrm{A}\mathrm{u}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{D}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}(S_{<t},h)$$where $S_{<t}$ means a subsequence of S at time step $t$ and $\widehat{S_t}$ is the predicted subsequence. In AutoregressiveDecoder, $\widehat{S_t}$ is generated by applying the softmax function to the logits at each time step $t$.

Evaluation metrics

The comparisons are presented in terms of word correction, word-level accuracy, and Levenshtein similarity, considering the input sentence S, the predicted sequence $\hat{S}$, and the target sequence T. For the noisy word $\widetilde{w_k}$ in a sentence S and the corresponding denoised word and target pair ( ${\hat{w}}_i,t_i$), the word correction is defined as:

(28) $$\mathrm{W}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{{\sum }_k\mathbf{1}(\widetilde{w_k})}{|\tilde{W}|},\mathrm{\forall }\widetilde{w_k}\in \tilde{W}$$where $\tilde{W}$ is a set of noisy words in the sentence S. The correction indicator $\mathbf{1}(\widetilde{w_k})$: where $\widehat{w_i}\in \hat{S}$ and $t_i\in T$.

Word-level correction focuses on whether a noisy word is corrected, which may lead to a lack of consideration for the overall sentence. Therefore, sentence accuracy is also considered to evaluate the entire sentence by assessing a model’s ability to retain error-free words without making any modifications. For $\mathrm{\forall }{w}_i\in S$, accuracy can be computed by counting all correct words, regardless of whether they are erroneous, using the indicator function:

(29) $$\mathcal{C}(w_i)=\{\begin{array}{c}1,\mathrm{i}\mathrm{f}\mathbf{1}({\mathrm{w}}_{\mathrm{i}})=1\hfill \\ 0,\mathrm{e}\mathrm{l}\mathrm{s}\mathrm{e}\hfill \end{array}$$

(30) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{{\sum }_i\mathcal{C}(w_i)}{|S|}$$

For considering the similarity between predicted sentences and targets, even if without correcting full word-level misspellings, Levenshtein similarity, which depends on the Levenshtein distance ( ${\mathrm{d}}_{\mathrm{L}}$), is also provided:

(31) $$\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{h}\mathrm{t}\mathrm{e}\mathrm{i}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}=1-\frac{{\mathrm{d}}_{\mathrm{L}}(S,T)}{|T|}$$

Experimental results

In this section, a comparative evaluation of various spelling correction tools and LLMs for the Turkish spelling correction task is presented. The performance of these models is assessed based on word-level accuracy and word correction metrics. The evaluation is conducted on the NoisyWikiTr dataset with two distinct noising strategies: uniformly generated noising and realistic noising, providing a comprehensive analysis of their effectiveness in handling different types of noise.

Table 5 presents the comparative evaluation of spelling correction tools and LLMs for the Turkish spelling correction task, based on word-level accuracy and word correction metrics, evaluated on both the uniformly generated noisy and realistically noisy sets.