Slovak morphological tokenizer using the Byte-Pair Encoding algorithm

Segmentation and extraction of word roots

To create our own morphological tokenizer capable of preserving the integrity of roots within individual tokens, it is necessary to first obtain roots for as many words as possible from the Slovak language.

For root segmentation, we utilized two lexical resources. The first dictionary from 2015, titled Retrograde Morphematic Dictionary of Slovak (Ološtiak, Genči & Rešovská, 2015), comprises over 70,000 morphemically segmented lexical units (lemmas), arranged in retrograde order. Root morphemes are bolded in the dictionary, grammatical morphemes are italicized, and all other types of morphemes are marked in standard font. The dictionary also includes additional annotations such as morphemic boundaries, information on derivational motivation, various markers capturing grammatical information, and many others, which are irrelevant for our research. An example of this dictionary is shown in Fig. 3. The English translations are indicated in blue.

The second dictionary we utilized was the Dictionary of Root Morphemes of Slovak (Sokolová, Ološtiak & Ivanová, 2012), which additionally includes some derived and alternative roots that were not present in RMSS.

After segmenting the roots for basic word forms (lemmas), we utilized the Morphological Database of the Slovak National Corpus (Slovak National Corpus, 2024), which contains over 98,000 unique lemmas with their corresponding full word paradigms (all inflected forms). In total, the database contains up to 1.19 million unique full word paradigms, along with tags capturing grammatical categories. Since we knew the roots for the lemma forms as we segmented them in the previous step, we were able to extract roots for other word forms from this morphological database. In total, we managed to extract roots for over a million words.

Text corpus

The aim of subword tokenization is to divide continuous text into smaller text units known as tokens. Current tokenization models utilizing BPE (Sennrich, Haddow & Birch, 2016) or WordPiece (Wu et al., 2016) algorithms are often trained on extensive text corpora. We decided to train the tokenizer using the OSCAR (Open Super-large Crawled Aggregated coRpus) text corpus (Ortiz Suárez, Sagot & Romary, 2019), which includes texts for up to 166 languages, including Slovak. We specifically used the 2019 version of OSCAR, which is derived from the Common Crawl corpus. We downloaded the Slovak part of this corpus using the HuggingFace library in Python.

The preprocessing of this corpus consisted of several steps. In the first step, we removed all text sections that were not in the Slovak language, as we aimed to train a purely Slovak tokenizer focusing on the morphology of the Slovak language. In the second step, various non-Slovak characters were removed, including mathematical symbols, pictograms, emoticons, and consecutive duplicate special and whitespace characters. In the third step, all duplicate and very similar sentences were removed, and URL addresses were replaced with the string “<url>”. After cleaning, the total size of the text corpus is 4.2 GB and it consists of a total of 589 million words, 3.7 million unique words, and contains only 222 unique characters.

Design and training of a morphological tokenizer

To ensure that word roots are not divided into multiple tokens and to preserve their integrity within individual tokens, it is essential for the roots to enter the BPE algorithm as indivisible units. There exists an implementation of the BPE training algorithm using the Tokenizers library from Hugging Face, which provides a powerful and flexible tool for text tokenization with support for various models and languages. However, it is not possible to enter the roots of words into this library's algorithm and indicate which roots should not be divided further. The implementation only allows input data in the form of text or file paths. Therefore, we decided to reprogram the BPE algorithm from scratch.

We started by loading the text corpus, which we normalized by converting all letters to lowercase. Then, we pre-tokenized the text using the GPT-2 tokenizer, which divides the input text into words with spaces and punctuation. Additionally, the GPT-2 tokenizer transforms the encoding into bytes, ensuring that the size of the basic vocabulary will be maximum 256 tokens and ensuring that every basic character will be included in the vocabulary.

Subsequently, we obtained the frequency of occurrence for all words and punctuation in the entire corpus (we created a dictionary called word_freqs). Finally, we added words to the list that were not found in the corpus but for which we had recorded roots.

When examining tokenization using the SlovakBERT tokenizer, we found that some words, especially the first words in a sentence without preceding whitespace, are often tokenized differently than the same words appearing within the sentence, i.e., with preceding whitespace. We decided to address this issue of ambiguous tokenization in our morphological tokenizer by adding whitespace before each valid word if it was not already present. We have already implemented this adjustment in the pre-existing word_freqs dictionary.

In the next step, we created the basic vocabulary, consisting of all characters used in the corpus (at the byte level), and added special tokens <s>, <pad>, </s>, <unk>, <mask>, which will be necessary during the training of the language model.

Next, it is necessary to prepare words for training the BPE tokenizer. In the original implementation, each word is divided into characters, but we divided each word into characters and roots if roots were recorded for that word. An example of word division (split dictionary) along with the frequency of occurrence can be seen in the following table (Table 1).

The training process itself involved iteratively iterating through individual words (in the word_freqs dictionary) and counting the frequencies of all pairs of adjacent tokens in the specific words from the splits dictionary. From these frequencies, we selected the pairs of tokens that were most commonly repeated and merged them into a single token. This merged token was added to the dictionary, and we replaced all occurrences of the corresponding pair of tokens in the splits dictionary. We repeated this process until we reached the maximum number of tokens, which we set to 50,264 (the same number as the SlovakBERT tokenizer).

The training of the tokenizer took approximately 180 h on an AMD EPYC 7542 32-Core Processor, which is several times longer compared to the optimized version of the implementation from Hugging Face, where it only took a few minutes. With the Tokenizers library, we also trained another tokenizer (pureBPE) on the same text as our morphological SKMT.

Tokenization performance evaluation

To evaluate the tokenization results of our method, we decided to compare the outcomes with tokenization using the SlovakBERT (SB) model tokenizer and with our custom pureBPE tokenizer, trained on the same text as our morphological tokenizer (SKMT). Even the SB tokenizer was partially trained on the same corpus.

For tokenization testing, we employed the OSCAR 2201 text corpus (Abadji et al., 2022). The entire Slovak portion of this corpus comprises 16.5 GB of data and contains 1.6 billion words. We randomly selected 500,000 text paragraphs from this corpus, ensuring that we chose paragraphs on which our tokenizers had not been trained. Each paragraph consists of approximately a few sentences. This sample seemed sufficiently representative and should ensure reliable tokenization results.

We evaluated several characteristics of tokenization, including the number of unique valid words (Word Count), the number of unique words for which roots were recorded (Word Root Count), the total number of tokens in the text (Total Tokens, TT), the average number of tokens per word (AT/W), the average number of tokens per rooted word (AT/RW), the average token length in characters for all valid unique words (Average Token Length, ATL), and the average token length in characters for unique words with recorded roots (Average Rooted Token Length, ARTL). Additionally, we assessed the percentage of correctly tokenized words with respect to roots (Correct Tokenization, CT).

For all acquired characteristics, we calculated basic statistics, which are provided in Table 2.

From the results of basic statistics, it is evident that the pureBPE tokenizer achieved the lowest values for characteristics such as TT, AT/W, and AT/RW, while also obtaining the highest values for the average token length in characters for both ATL and ARTL. Conversely, the SB tokenizer exhibited the highest values for TT, AT/W, and AT/RW, and the lowest values for ATL and ARTL, highlighting a clear contrast in performance between the two tokenizers. Additionally, while the SKMT tokenizer achieved the best results for the percentage of correctly tokenized words with respect to root integrity (CT), the SB tokenizer had the lowest performance in this characteristic.

For more detailed verification of the results, we decided to use box plots for all three tokenization methods: SB, pureBPE and SKMT.

Visual representations of the number of tokens can be observed in the form of box plot in Fig. 4. From the graph, it is clear that both tokenizers (pureBPE and SKMT) tokenize the text into a smaller number of tokens (TT characteristic) compared to SB, as the median for pureBPE (Q2 = 88) and SKMT (Q2 = 91) is smaller than for SB (Q2 = 93). Based on the minimum, lower quartile (Q1), median (Q2), upper quartile (Q3), and maximum, it can be confirmed that pureBPE tokenizes into the smallest number of tokens.

A smaller number of tokens can potentially enhance the efficiency of a language model by reducing the computational operations required for text processing, thereby leading to faster training and inference. It can also improve the model’s understanding, as longer sequences may distract the model’s attention and hinder context capture. However, a smaller token count may occasionally be disadvantageous, resulting in loss of textual nuances and details that could affect model accuracy. The SKMT tokenizer strikes this balance, positioning itself in token count between pureBPE and SlovakBERT tokenizers.

Another characteristic we examined was the average number of tokens per word for all unique words (All Words-AT/W) and for unique words for which we know the roots (Only Rooted Words-AT/RW). The results from Fig. 5 suggest that both tokenizers (pureBPE and SKMT) achieve a lower average number of tokens per word compared to the SB tokenizer. Interestingly, the difference is even more pronounced for rooted words in both observed groups. Overall, it can be observed that pureBPE and SKMT consistently produce fewer tokens per word than SB, with SB showing the highest token count in this regard.

Since the average number of tokens correlates with the average token length, it is not surprising that the results, depicted in Fig. 6, confirm the same trend as we found in the previous graph (Fig. 5). In this graph, the larger differences between the groups All Words (ATL) and Only Rooted Words (ARTL) can be seen even more clearly.

The most important results of our research can be observed from Fig. 7, which displays the percentage correctness of tokenized words concerning root integrity in tokens (CT characteristic). The correctness of a tokenized word concerning root integrity means that all roots within a single word will be segmented into tokens such that none of the roots are split across multiple tokens, but the entire root will be part of one token. This means that one token can contain not only the root but also other characters of the word that are before or after this root. It was already evident from Table 2 that the SB tokenizer yields the worst results (mean = 0.905, Q2 = 0.914), followed by pureBPE (mean = 0.931, Q2 = 0.939), with SKMT performing the best (mean = 0.997, Q2 = 1.000). These results are confirmed in this graph (Fig. 7).

For more thorough verification of the results, we decided to use residuals for this characteristic. We compare the tokenization results of pureBPE and SlovakBERT with the tokenization of SKMT. This comparison differs from the previous one in that we now take the percentage difference between SB and SKMT (denoted as R1) and pureBPE and SKMT (denoted as R2) for specific tokenized texts, not just calculated statistics for individual tokenizers.

The results were confirmed by this approach as well (Fig. 8), as Q1 (−0.128), Q2 (−0.083), and Q3 (−0.047) for the residuals R1 (SB-SKMT) are less than 0, indicating that the SB tokenizer tokenizes words with lower percentage correctness concerning root integrity compared to SKMT. A similar pattern is observed for the residuals R2 (pureBPE-SKMT). Since Q1 (−0.095), Q2 (−0.059), and Q3 (−0.029) are also less than 0, it is clear that the SKMT tokenizer tokenizes words with higher percentage correctness concerning root integrity compared to the pureBPE tokenizer. Based on the measured values of R1 and R2, it can be concluded that the SKMT tokenizer is more accurate in preserving root integrity than pureBPE and SB.

Tokenization evaluation using language modeling

Since our new tokenization method, SKMT, achieved the best results based on the average root integrity value in the resulting tokens, we decided to validate this tokenizer on a language model. We used the RoBERTa architecture (Liu et al., 2019), which is based on the Transformer architecture (Vaswani et al., 2017), for pretraining the models. One model was trained using the pureBPE tokenizer (named SK_BPE_BLM – SlovaK BPE Baby Language Model), and the other was trained using our SKMT (named SK_Morph_BLM – SlovaK Morphological Baby Language Model). Our model consisted of six layers of encoders, 12 attention heads, a hidden size of 576, and a total of 58 million parameters. The maximum position embeddings were set to 258, corresponding to the maximum sequence length. The model was trained with a batch size of 128 for approximately 10 epochs. We used the Adam optimizer with a learning rate of $1\times {10}^{-4}$, a dropout rate of 0.1, and a weight decay of 0.01. We pretrained these models on the same text used for evaluating the tokenization. It is important to emphasize that our goal was not to create a model that surpasses current models containing billions of parameters, but to demonstrate that our proposed tokenization method (SKMT) improves language modeling compared to the currently used BPE method.

We evaluate the language modeling itself using the perplexity metric, which measures how well the model predicts a masked token in a sequence. It is calculated as the exponential value of the cross-entropy (L-average logarithmic loss) of the model. The lower this value, the better the model is at predicting the correct tokens, indicating a better understanding of the text (Le Bronnec et al., 2024).

(1) $$Perplexity=e^L$$

The data (Fig. 9) indicate that the model trained with the SKMT tokenized text (SK_Morph_BLM) outperformed the model trained with the BPE tokenized text. Specifically, the SKMT model achieved a lower evaluation loss with a median of 0.187 and a corresponding lower perplexity with a median of 1.206 at the conclusion of the pretraining phase (epoch 9). In contrast, the BPE model demonstrated an evaluation loss with a median of 0.251 and a perplexity with a median of 1.285. These results strongly suggest that the SKMT method facilitates a more nuanced understanding of linguistic structures compared to the traditional BPE approach. Further validation of these findings will be conducted through an NLP task focused on sentiment classification.

For fine-tuning the language models on the NLP task of sentiment classification, we utilized the Reviews dataset, which was created by the authors of the SlovakBERT model (Pikuliak et al., 2022) for the purpose of fine-tuning and evaluating this model. This dataset comprises reviews from various domains: accommodation, books, cars, games, mobiles and movies. In total, the dataset consists of 677 entries labeled with sentiments: negative (315 entries), neutral (57 entries), and positive (305 entries). Preprocessing involved manual correction of diacritics, which were missing in some entries. The text from the dataset was tokenized using both tokenizers, and the models were evaluated through fine-tuning using stratified 10-fold cross-validation. We reduced the learning rate to $1\times {10}^{-5}$ and set the batch size to 16. A classification layer was added to the model, which determines the classification into one of the three sentiment categories.

Fine-tuning was conducted over 10 epochs across all 10 folds. After each epoch, basic evaluation metrics were calculated: accuracy, precision, recall, and F1-score. The result of each fold was the highest F1-score achieved. In Fig. 10, we present the results of this validation, showing the metrics for accuracy and F1-score. The primary metric for evaluation will be the F1-score, due to the imbalance in the dataset. From the graph, it is evident based on the F1-score metric that the SK_Morph_BLM model (median = 0.787) was able to classify sentiment better than the SK_BPE_BLM model (median = 0.753) by nearly 3.5%.

To examine the results in greater detail, we generated a confusion matrix (Fig. 11) that displays the correct and incorrect predictions for each category, combined in a single graph across all folds. For negative sentiment, the SK_Morph_BLM model performs 1% worse compared to the SK_BPE_BLM model. However, for neutral and positive sentiments, SK_Morph_BLM significantly outperforms SK_BPE_BLM, specifically by 14.1% for neutral sentiment and by 3.9% for positive sentiment.

Following the methodology outlined in the thesis “Distilling the Knowledge of SlovakBERT” (Agarský, 2022), we focused on the Semantic Textual Similarity (STS) task after fine-tuning our language models on sentiment classification. For this task, we utilized the Semantic Textual Similarity Benchmark (STS-B) dataset (Cer et al., 2017), which we translated from English into Slovak. This benchmark consists of sentence pairs from various domains and genres, rated for their similarity on a scale from 0.0 to 5.0. In the original version of the dataset, these sentence pairs were divided into training, development, and test sets. We combined the training and development sets, excluding the test set that did not contain similarity ratings, resulting in a total of 7,163 sentence pairs with similarity ratings. The similarity ratings were normalized to a scale of 0.0 to 1.0.

To fine-tune and evaluate the models for this task, we again employed stratified 10-fold cross-validation. Each fold was trained for a maximum of 15 epochs, implementing an early stopping mechanism based on the mean squared error (MSE) metric to prevent overfitting when performance no longer improved. We set the learning rate to $1\times {10}^{-5}$ and the batch size to 32. A regression layer was added to the model to predict the continuous value of semantic similarity.

For evaluating text similarity, we used the Pearson and Spearman correlation coefficients, which provide a comprehensive view of the model’s quality in measuring the similarity between sentence pairs. In Fig. 12, we present the STS evaluation results, showing the Pearson and Spearman coefficients. The graph clearly indicates that the SK_Morph_BLM model achieved higher median values of Pearson (Q2 = 0.612) and Spearman (Q2 = 0.586) coefficients compared to the SK_BPE_BLM model, which achieved median values of Pearson (Q2 = 0.512) and Spearman (Q2 = 0.501) coefficients. These results suggest that the SK_Morph_BLM model, which is based on morphological tokenization, better captures the similarity between sentence pairs than the SK_BPE_BLM model.