Detection of offensive terms in resource-poor language using machine learning algorithms

Data gathering

For the purpose of this study, two cross-platform datasets have been gathered for detecting offensive language, one from Twitter platform (Amjad et al., 2022) (referred to as D₁) and the other from YouTube (referred to as D₂). The type of text on both platforms is different so these datasets are ideal for answering the research question discussed in the introduction.

The D₁ dataset comprised of 2,400 tweets containing offensive language in Urdu. The statistics of the datasets used in this study are reported in Fig. 3. The D₁ is a balanced dataset with 1,187 offensive tweets and 1,213 non-offensive tweets. Likewise, D₂ is also a balanced dataset with 1,109 offensive comments and 1,062 non-offensive comments.

A sample representation of datasets D₁ and D₂ are shown as word clouds in Figs. 4 and 5 respectively displaying the most prominent words. The word cloud is created using Matplotlib and WordCloud package in Python. The words in the word cloud are from text columns from both datasets. The parameters defined for the word cloud are the usage of available stopwords, size of fonts, number of most frequent words and colourmap to have a consistent colour theme for both word clouds.

The collected datasets are pre-processed in order to transform them into a suitable format for further processing. This study proposes a combinatorial pre-processing approach using standard pre-processing techniques. The details are described in the following subsections.

Pre-processing

Data pre-processing steps in this study consist of two main operations: (i) data cleaning data and (ii) feature engineering. To clean data, stopword removal (SR), punctuation removal (PR) and accent removal (AR) are performed.

Stopword removal (SR)

Stopword are usually in large quantities but have the least significance in the text classification. For instance, is, are, am, was, were, the, a, an etc are examples of stopwords in English and Fig. 6 shows the example of stopwords in Urdu. These words are removed in the process of stopword removal. An example of stopword removal is given in Fig. 7.

Training testing sets

The considered cross-platform datasets D₁ and D₂ have been used for the training and testing using the split percentage method of validation. The study proposes the use of datasets from different platforms for training and dataset from entirely different platform testing, thus, the training set can be taken at different percentages from one dataset and tests can be performed on the entirety of the other dataset as shown in Table 3. The four experiments with 100%, 80%, 70% and 60% are taken as training sets from D₁. The developed model is used for testing over a D₂. Likewise, four experiments with 100%, 80%, 70% and 60% were taken as training sets from D₂ and tested on complete D₁.

Modelling and evaluation

The machine learning algorithms used in the study are well established, namely, decision tree (DT), logistic regression (LR), support vector machine (SVM), naive Bayes (NB), and random forest (RF). The selection of machine learning algorithms in this study has been made carefully keeping several factors as the selection criteria such as the availability of data, the research questions, and the complexity of the problem. Besides, due to the small or limited number of instances in the considered datasets, these selected algorithms deem suitable for this study. The considered machine learning algorithms usually perform well when applied to small datasets. The details of the machine learning algorithms applied in the experimental set-up are described in the following.

Decision tree

A decision tree (DT) is a general-purpose predictive modelling method with applications in a variety of fields. It is among the most commonly used and practical supervised learning methods. DT is a tree-like graph in which nodes represent the points at which a feature is selected and pose a question; edge denotes the answers to the question, and leaves show the actual class label. They are employed in nonlinear decision-making. The DT method has several advantages over other methods: (a) It is simple to interpret and understand. (b) It is simple to depict graphically. (c) It can handle both quantitative and qualitative data. (d) It necessitates minimal data preparation. (e) It handles large datasets well. Figure 8 shows a simple example of a decision tree employed for weather prediction (Matzavela & Alepis, 2021).

Logistic regression

Logistic regression (LR) is also another effective supervised machine learning approach for binary classification. Logistic regression fits well as a linear regression for classification tasks. To predict a binary output variable, logistic regression employs the logistic function shown in Eq. (2). The major distinction between logistic regression and linear regression is the range of logistic regression, which is limited to 0 and 1. Furthermore, logistic regression unlike linear regression doesn’t really require a linear connection between input and output variables. This is because the odds ratio is transformed using a nonlinear log transformation (Belyadi & Haghighat, 2021). (2)${\displaystyle LR=\frac{1}{1+e^x}.}$ The input variable in the sigmoid function Eq. (2) is x. Consider feed values ranging from −20 through 20 into logistic function as shown in Fig. 9. The inputs have been converted to values ranging from 0 to 1.

Support vector machine

The support vector machine (SVM) is a prominent supervised learning technique that is used for both classification and regression problems. However, it is mostly utilised for classification tasks. The goal of the SVM algorithm is to find the optimum decision or line boundary for categorising n-dimensional space such that fresh data points be placed at the proper category in future. A hyperplane is the optimal choice boundary. SVM selects the extreme vectors/points that aid in the creation of the hyperplane. Consider Fig. 10 that shows two distinct categories separated by a hyperplane or decision boundary (Mohammadi et al., 2021).

Naive Bayes

Bayesian classifiers are a type of classification method that is based on Bayes’ Theorem. It is a group of algorithms that all share a similar proposition, namely that every pair of characteristics being categorised is independent of one another. Bayes’ theorem, often called the Bayes rule, is a mathematical formula used to calculate the likelihood of a hypothesis given past knowledge. It is determined by the conditional probability (Bhavani & Kumar, 2021). The formula for the Bayes rule is shown in Eq. (3). (3)${\displaystyle P\left(A\mid B\right)=\frac{P\left(B\mid A\right)\ast P\left(A\right)}{P\left(B\right)}.}$

Random forest

Random forest is a well-known machine-learning algorithm. It may be applied to both regression and classification problems. It is built on the notion of ensemble methods, which is the process of merging numerous classifiers to resolve a complicated issue and enhance the model’s performance. It comprises decision trees on different subsets of the provided dataset and chooses the average to enhance the predicted accuracy of that dataset (as the name implies—random forest). Instead of depending on a decision tree classifier, the random forest collects the forecasts from each tree and predicts the output depending on the voting of predictions (Sumathi & Pugalendhi, 2021). Figure 11 shows the working principle for the random forest as discussed.

After modelling the next step is to evaluate the performance of different machine learning algorithms with experimental set-up. The performance evaluation metrics used in this study are accuracy, precision, recall and F-measure. Details of the considered evaluation metrics are described as follows.

Accuracy

Accuracy metric of a model is shown as the ratio of correctly predicted text labels to the total number of instances in the dataset. Accuracy is computed as reported in Eq. (4). (4)${\displaystyle Accuracy=\frac{T_p+T_n}{T_p+T_n+F_p+F_n}}$ where T_p, is true positive, Tn refers true negative, F_p shows false positive and F_n means false negative.

Precision

An algorithm’s precision is given as the ratio of correctly predicted value with the total number of predicted values. It is calculated as shown in Eq. (5). (5)${\displaystyle Accuracy=\frac{T_p}{T_p+F_p}.}$

Recall

Recall of a model is a metric that is represented as the ratio of correctly classified values divided by the total number of values in the dataset. It is computed by Eq. (6). (6)${\displaystyle Accuracy=\frac{T_p}{T_p+F_n}.}$

F-measure

F1 score also called F-measure is the harmonic mean of the values. It is considered a a reliable metric, which includes both precision and recall in computing its value. F1 score is harmonic or precision and recall. F1 score provides a balance between precision and recall. Further, F1 score relatively suits better when the distribution of class labels is imbalanced. It is computed by Eq. (7). (7)${\displaystyle F-measure=\frac{2\ast precision\ast recall}{precision+recall}.}$

Scenario 1

The dataset D₁ is used for the training (model trained on tweets) and dataset D₂ is used for the purpose of testing. Split percentage is used for the validation because cross-validation method does not support the aim of the study, which uses different datasets for the training and testing purposes. D₁ is split into 100%, 80%, 70% and 60% for training as reported in Table 3.

Table 4 shows the performance evaluation of considered machine learning algorithms when recursive pre-processing uses merely the stopword removal (SR) method. It is observed that DT performed with an accuracy of 83.27%, precision of 85.13%, F1 Score of 83.11% and Recall of 83.52% when 100% of dataset D₁ is used for training and 100% D₂ as testing set. NB performance remained bad amongst the considered algorithms for different experiments. For instance, while a 60% training set is used, naive Bayes produced an accuracy of 55.65%. It is obvious from the results that when more data is used in training, the developed models perform better. It may be noticed that it is not necessary that more cleaning operation leads to a generalised model. Instead, it depends on the pre-processing operations which lead to producing more matching vocabulary for both datasets.

Machine learning algorithms, inherently, tend to behave differently with a different dataset. Besides, a combination of more than one pre-processing technique may also cause different performances in terms of outcomes. It is evident from Fig. 12 that RF performed better with F-measure when 100% split training is used and achieved the least F-measure at 60% split. NB performance remained bad in comparison with the other four considered algorithms. The NB achieved the highest F-measure of 56.88% (100% split training set) and the least value of 53.68% (60% split training set) is used.

The three pre-processing techniques are combined (i.e., SR + PR + AR) in recursive pre-processing, the results of this experiment are shown in Fig. 13. RF outperformed other considered algorithms 75.19% (at 100% split). The poor performance was again produced by NB with an F-measure of 53.75% (at 60% split).

The experimental results showed that DT outperformed when less number of pre-processing techniques are applied. However, when the number of pre-processing techniques has increased the performance of DT decreases because of its reliance on a number of features which decrease with an increasing number of cleaning operations. The ensemble nature of RF produced better results with increasing the pre-processing operations. NB produced poor results for the experiments, the possible reason is its nature of probabilities. The dataset is unstructured data i.e., text data, thus, the probabilistic approach may not perform well in these situations.

Scenario 2

This scenario uses dataset D₂ as training and dataset D₁ for testing. Table 5 shows the results of this scenario when SR and PR pre-processing operations are performed. RF outperformed in this scenario also with accuracy, precision, recall and F1 score of 74.54%, 76.10%, 74.08% and 74.41% respectively at 100% split of training data. Naive Bayes performed very poorly with almost 60% of accuracy around all the splits (i.e., 100%, 80%, 70% and 60%). The increase in pre-processing operations for RF tends to perform better than DT.

Figure 14 shows F1 score for the considered algorithms at different splits when a single pre-processing operation SR (stopword removal) is applied. The outperforming algorithm, in this case, is DT with F1 score of 73.12% at 100% split. It is observed that the value of F1 score decreases with the decrease in training data. NB performed poorly with the F1 score of 57.55% at 100% split. The difference in F-measure scores varies around 2% to 3% for NB at different splits.

Figure 15 reports three pre-processing techniques (SR, PR and AR) are applied together; the performance of DT decreases from 73.12% to 72.75% and RF increases from 71.50% to 72.75% at 100% split. The performance of the model decreases with a decrease in training percentage. The very poor-performing model throughout the study is naive bayes.

Comparison with baseline

We compared models created using suggested pre-processing techniques from the experiments in scenarios 1 and scenario 2. We trained classification models based on datasets D₁ and D₂ separately with a stratified split of 15%. The comparative results using dataset D₁ with existing studies, our ML models outperformed the XGBOOST (eXtreme Gradient Boosting) and LGBM (Light Gradient Boosting Machine) models. However, the proposed combinatorial pre-processing techniques with ML algorithms could not achieve better results than that of transfer learning models which use pre-processing as well as neural networks, such as mBERT (multilingual Bidirectional Encoder Representations from Transformers) and dehatebert-mono-arabic. The experimental results, in which dataset D₂ is applied, show that the ML models outperformed all the models reported in Akhter et al. (2020) for both scenario 1 in section ‘Scenario 1’ and scenario 2 in section ‘Scenario 2’. These results suggest that an objective approach in selecting pre-processing steps can improve the performance of a simple ML model compared to classical and boosting techniques.

Answering the research questions (RQs)

The experimental results provide the answers to the research questions (RQs). To answer RQ1, the results in scenario 1 in section ‘Scenario 1’ and scenario 2 in section ‘Scenario 2’ clearly suggest that pre-processing techniques improve the performance of the classification model based on cross-platform data. However, the performances of models depend upon pre-processing approaches and their combinations. The answer to RQ2, it can be observed from the results that in scenario 1 in section ‘Scenario 1’ where the D₁ is used for training, the most effective pre-processing technique is SR, because data in tweets is limited by characters and also the language used in tweets is substandard. Thus, other pre-processing techniques would not affect it. In scenario 2 in section ‘Scenario 1’, the combination of SR and PR performed well because the language used in the comment is standard and has no limit of characters. AR requires the use of sophisticated language, which is usually unavailable on social media platforms. Table 6 reports the answer to RQ3, that is, pre-processing affects significantly over classical machine learning models. However, pre-processing techniques do not make a significant impact on NLP models such as BERT. Thus, it is evident from experimental results that pre-processing techniques can improve the performance of the classification model at a certain level.

This study focused on developing a model using one dataset and evaluating the model using entirely another dataset. Therefore, the comparison of the results of scenario 1 (details in section ‘Scenario 1’) and scenario 2 (details in section ‘Scenario 2’) to the existing literature would be biased. However, experiments have been performed to compare the developed model by applying combinational pre-processing techniques using dataset-1 (d₁) as well as dataset-2 (d₂) separately. These experiments used the d₁ for the training as well as testing in order to compare the results with the existing literature. Similarly, experiments were performed using d₂ using the proposed approach of developing the model. These experimental results are reported in Table 6 and their confusion matrix for the developed model using the proposed approach is presented in Fig. 16.

The confusion matrices reported in Fig. 16 show the effectiveness of the proposed approach in which combinational pre-processing techniques are used to develop the model. It is observed that using the proposed approach to develop the model outperformed the models available in the existing literature (see details in Table 6). Thus, it is inferred from experimental results shown in section ‘Scenario 1’, section ‘Scenario 2’, and Table 6 the proposed model that adopts combinational pre-processing techniques provides worth mentioning results. The effectiveness of the proposed approach is observed when the same dataset is used for the training as well as testing (see Table 6). Besides, it also provided significant results when one dataset is used for training and an entirely disjoint dataset is used for the testing (see section ‘Scenario 1’ and section ‘Scenario 2’).