Machine learning based framework for fine-grained word segmentation and enhanced text normalization for low resourced language

Removing diacritics

In NLP models, the diacritics can reduce the output to some extent. The word with a diacritic and the same without a diacritic would be considered different. Therefore, to remove the diacritics from Urdu text, we brought the characters in the range 0x0600–0x06ff (Khan et al., 2021). Equations (1)–(2), presents the rule for removing the diacritics. (1)${\displaystyle text=\left\{sentencefromcorpus\right\}}$ (2)${\displaystyle new\text{\_}text{=}^{\prime }.join\left[tfortintextifftnotin\left[unichr\left(x\right)\right]\right]forxinrange\left(0x0600,0x06ff\right)iffunicodedata.category\left(unichr\left(x\right)\right)=={}^{{}^{\prime }}M{n}^{{}^{\prime }}}$

Separating punctuation and sentence ending characters

For this purpose, we utilized the string library and downloaded the set of punctuation characters (Martín-del Campo-Rodríguez et al., 2019). To standardize Urdu text, we introduced a white space after each punctuation mark and removed the white space before punctuation. The rule for separating punctuation and sentence ending characters is given in Eqs. (3) and (4). (3)${\displaystyle s=\left\{setofUrdupunctuation/sentenceendingcharacters\right\}}$ (4)${\displaystyle \forall x\in s,iffxintext;new\text{\_}text=text.replace\left(x,x{+{}^{{}^{\prime }}}^{{}^{\prime }}\right)}$

Similarly, the a set of ending sentence ending characters was created and if that specific character occurred in text space before it was removed and single space after such character was introduced.

Separating digits

The digits should be separated from the text; otherwise, the natural language processing models would consider the digits and joined text as a single word. Therefore, we placed a white space before and after the digits. We have also incorporated the ‘.’ symbol between the digits. If it appears between the digits, no white space will be placed. (5)${\displaystyle new\text{\_}text=re.sub\left(r^{{}^{\prime }}{\left(\left[0-9\right]+\left(.\left[0-9\right]+\right)?\right)}^{{}^{\prime }},r^{{}^{\prime }}1”,text\right)}$

In Eq. (5), regular expression has been provided for separating digits from other text.

Separating special, mathematical and greek symbols

In Urdu text, mathematical and Greek symbols are also widespread. To separate these symbols from the text, we introduced white spaces before and after such characters as presented in Eqs. (6) and (7). A rule was also developed to separate special that are explicitly used in the poetic text. (6)${\displaystyle s=\left\{setofspecial/mathematical/greekcharacters\right\}}$ (7)${\displaystyle \forall x\in s,iffxintext;new\text{\_}text=text.replace\left(x{{,}^{{}^{\prime }}}^{{}^{\prime }}+x{{+}^{{}^{\prime }}}^{{}^{\prime }}\right)}$

Separating English words

In Urdu texts, English words are commonly observed. These words can convey meaning whether they are separate from Urdu text. However, in text processing, this can affect the final results. Therefore, white spaces were placed before and after English words. (8)${\displaystyle new\text{\_}text=re.su{b}^{{}^{\prime }}\left[\mathbf{A}-\mathbf{Za}-\mathbf{z}\right]{+}^{{}^{\prime }},lambdaele{{:}^{{}^{\prime }}}^{{}^{\prime }}+ele\left[0\right]{{+}^{{}^{\prime }}}^{{}^{\prime }},text}$

The Eq. (8) represents the regular expression for separating the English words from other text.

Standardizing characters

Urdu writing script is complex to analyze. A single character can have different shapes of writing, such as are various shapes of the same character. This can negatively affect the results. We converted the variations into standardized forms by developing the appropriate rule presented in Eqs. (9)–(11). (9)${\displaystyle s\text{\_}set=\left\{standardizedUrducharacters\right\}}$ (10)${\displaystyle v\text{\_}set=\left\{setsofUrducharactervariations\right\}}$ (11)${\displaystyle \forall s\in s\text{\_}set,v\in v\text{\_}setiffv\left[i\right]intext:replace\left(vwiths\right)}$

Removing extra spaces

Urdu text can contain extra white spaces during composition; similarly, while implementing prior rules and regular expressions, extra spaces are possible in the text. To tackle this issue, we developed a regular expression to remove extra white spaces. Using regular expression as presented in Eq. (12), more than white spaces would be removed from text. (12)${\displaystyle new\text{\_}text{=}^{{}^{\prime }}re.sub\left(\setminus s\setminus s{+}^{{}^{\prime }}{,{}^{{}^{\prime }}}^{{}^{\prime }},text\right)}$

The proposed rules and regular expressions with possible outcomes are presented in Table 4. The regular expressions and rules were created to normalize the text and convert it into standard Urdu text. The rules were manually extracted (Shafi et al., 2022) with the help of domain experts. These expressions can significantly increase the effective manipulation of text while reducing the computation complexity if we do not remove the diacritics. The model will consider both words differently.

Table 4:

Proposed rules and regular expressions for Urdu text normalization.

DOI: 10.7717/peerjcs.1704/table-4

Defining key features

After performing manual annotation, the next phase would be based on feature extraction. These tasks need input written text with clear boundaries. Sentence splitting is separating sentences based on punctuation in the text. Tokenization and sentence splitting are complex tasks when it comes to the Urdu language, as there are irregular white spaces between words. The features useful for tokenization would be extracted from the text corpus and forwarded to the machine learning algorithm. The features can be the following:

1. Current character of text

2. Previous seven characters from the current character

3. Next seven characters from the current character

4. Identification, if the current character is non-joiner

5. Identification, if the current character is a joiner

6. Identification, if the current character is a digit

7. Identification, if the current character is a Greek letter

8. Identification, if the current character is a mathematical character

9. Identification, if the current character is a symbol

10. Identification, if the current character is from the English language

11. Unicode class of the current character

Feature classification

The native speaker can identify the word boundaries. However, for machines, it is difficult. The characters in Urdu text can adopt different shapes when joined based on context. A character can be categorized as (1) starting, (2) ending, (3) middle, and (4) separated. Such characters are termed a joiner, while on the other hand, if a character can make a posse, only two forms, ending and separation, are referred to as non-joiners. When two morphemes join while forming a word, the writer would like inert a space for better visualization if the initial morpheme ends with a joiner. On the other hand, if the first morpheme ends with a non-joiner, the writer would not inset white space because there would be no effect on shape or word. The sub-words are separated with zero-width non-joiner.

For accurately performing the tokenization of Urdu text, we utilized the conditional random field (CRF) algorithm (Lafferty, McCallum & Pereira, 2001), which used different linguistic features for the identification of white space as the boundary of a word and zero-width non-joiner (ZWNJ) as a boundary of sub-word. CRF is a type of probabilistic graph model that considers the neighboring context for performing different tasks such as classification. In CRF the dependencies are implemented between the predictions. CRF was considered to specify the word boundaries. However, We have also developed rules for separating punctuation, mathematical symbols, special characters, and English words and removing more than single spaces. The linear chain CRF can be expressed with Eq. (13). (13)${\displaystyle P\left(y|x\right)=\sum _{h}P\left(y|h,x\right)P\left(h|x\right)}$

where the observations are x = x₁, x₂.... x_n and labels are y = y₁, y₂, …. y_n. The set of latent variables is presented by h. The complete code of normalization and tokenization with dataset have been uploaded on GitHub (https://github.com/Shahzad-Nazir/Normalization-and-Tokenization). The research community popularly uses this model for conducting NLP experiments (Geng, 2021). For training, the model, 60% dataset, and for testing, 40% dataset was utilized (Nazir et al., 2020).

Predicting sequence of character

To accurately perform the tokenization, the CRF linear model was trained on a given set of features against each character of text. The stream of characters was converted into 0s and 1s. If the current character is the last character of a word, it was assigned a label 1, and for other characters, 0 was assigned. The label 1 denotes that the next would be white space, while 0 indicates that the next would be a non-space character. After training, the model could predict the appropriate sequence of characters of words.