Addressing cyberbullying in Urdu tweets: a comprehensive dataset and detection system

Data scraping

The initial step in developing the research corpus involved scraping Urdu tweets, specifically focusing on tweets originating from Pakistan. For this purpose, the Twitter-Scraper module from the SNS library in Python was utilized.

The data collection process was carried out at the district level, focusing on the target region of Pakistan. To identify the specific districts, geo-coordinate locations were extracted from reliable sources such as Google Answers and LatLong.net (2022). The districts were ranked based on population to ensure a representative sample encompassing various regions. The list of all districts was obtained from the official website of the Pakistan Bureau of Statistics (PBS, 2020). A maximum of 6,000 tweets were collected for each district, aiming to gather a substantial volume of data for analysis and ensuring diverse representation from different regions within Pakistan.

To capture a wide range of tweets and observe potential variations over time, the data collection timeframe spanned from January 2022 to July 2022. This extended duration allowed for the collection of a significant number of tweets and facilitated the analysis of temporal patterns and content changes. To ensure an even distribution of tweets throughout the collection period, approximately 1,000 tweets were targeted for each month. This systematic approach ensured the acquisition of a balanced dataset, covering different months within the specified timeframe.

To enhance the dataset’s coverage and address potential imbalances, tweets containing cyberbullying terms were selectively included for data augmentation. Careful consideration was given to choosing terms that would capture a broad range of content relevant to the research focus. By incorporating tweets specifically mentioning these terms, the coverage and representation of the dataset were improved, enabling a more comprehensive analysis. Furthermore, this approach helped balance the dataset by ensuring adequate representation of various topics and aspects, thereby mitigating potential bias stemming from an uneven distribution of tweets across different themes or subjects.

In conjunction with data augmentation, the training dataset underwent a process of shuffling. The order of tweets was randomized to introduce variability into the model's exposure during each training epoch. This strategy aimed to prevent the model from learning patterns associated with the order of the data and, consequently, reduced the risk of overfitting.

By following the outlined data collection steps, a diverse and representative dataset encompassing tweets from various districts in Pakistan was acquired. The inclusion of the timeframe, shuffling of tweets, and data augmentation (fetching tweets against cyberbullying specified terms) ensured the incorporation of a wide range of tweets, ultimately enhancing the overall quality of the dataset.

Guidelines development

Annotation guidelines development process was an iterative cycle that involves repeating the steps to achieve a high-quality annotated dataset. The complete cycle includes defining annotation guidelines, annotating tweets, conducting completeness and correctness checks, inter-annotation comparison for diversity and cross-check, reviewing conflicting cases, refining rules, and repeating the process as necessary. This iterative approach helped to improve the accuracy and reliability of the annotated data with each iteration.

The annotation guidelines were developed through an iterative process. Initially, a random sample of 10% from the collected tweets was selected. Three native Urdu annotators were then assigned the task of independently labeling each tweet as “insult,” “offensive,” “name-calling,” “threat,” “curse,” “profane,” or “None.” Simultaneously, the annotators were instructed to create an initial set of guidelines based on their individual annotations.

The annotated results were compared and discrepancies in the labeled data sample were discussed among the annotators. This collaborative discussion helped refine and finalize the guidelines. Through this iterative process, a comprehensive set of guidelines for annotation was successfully developed. Guidelines developed for data annotation are presented in Table 3.

Data annotation

The data annotation process is pivotal in generating a dataset of utmost quality. It encompasses a systematic procedure for meticulously assigning precise labels to each tweet within the corpus, ensuring the creation of a dataset of high quality and accuracy. The annotation process ensures that the dataset is appropriately labeled to facilitate subsequent analysis and model training. The annotation process was conducted with three native Urdu annotators. As a result of this process tweets were annotated with label of “insult”, “curse”, “name-calling”, “profane”, “offensive”, “threat” or “none”.

The following section provides further details regarding the annotation process employed in this research.

Tweets annotation: Once the annotation guidelines were established, the tweets were annotated accordingly. Each tweet was carefully reviewed and labeled based on its content and relevance to the research topic. The annotation process involves assigning appropriate label to each tweet.

Complete and correctness check: After annotating the tweets, a completeness and correctness check was performed to check the quality of annotated data. This step involves reviewing the annotated dataset to identify any tweet with missing label or incorrect label (label other than defined in guidelines). It was crucial to address any errors or discrepancies at this stage to maintain the quality of the annotated dataset.

Review conflicts: Similar to the initial level, conflicts arising from the annotators’ decisions were identified, and consensus was reached through discussion. In cases where conflicts persisted on the labeling of a tweet during the annotation process, a third annotator was brought in for discussion. The annotators collectively deliberated on the differing opinions and arrived at a consensus to assign the appropriate label based on a majority vote. This approach ensured that the final label for each tweet accurately reflected the agreed-upon decision among the annotators.

Specification of developed Corpus

The specification of developed corpus is presented in Table 4. Around 12,500 Urdu tweets were finalized with different type of cyberbullying for cyberbullying detection system. The majority of tweets fall into the ‘None’ category (8,598), followed by ‘Insult’ (1,532), ‘Offensive’ (995), ‘Name-calling’ (738), ‘Profane’ (397), ‘Threat’ (92), and ‘Curse’ (76). The total number of tweets in the dataset is 12,428.

Example tweets from few categories

Table 5 presents a collection of annotated tweets written in Urdu, along with their corresponding Roman equivalents and English translations. This subset specifically showcases tweets annotated against categories such as ‘Name-calling’ and ‘Threat’. Examining these examples provides a nuanced exploration of the labeled data, offering valuable insights into the contextual aspects that led to the assignment of these particular labels. This diverse representation not only enhances comprehension of the labeled dataset but also provides a detailed examination of the language nuances and cultural factors influencing the cyberbullying detection process.

Table 5:

Example tweets from developed corpus.

Name-calling
1	Nastaliq Urdu	دو کوڑی کے ڈنگر عمران خان کو مناظرے کا چیلنج کر رہے ہیں #امپورٹڈ_حکومت_نامنظور
	RU Urdu	Doo kori kay dangar Imran khan ko manazrey ka challenge ker rehey heen. Imported hakoomat namanzoor
	English translation	Two bit as*es are challenging Imran Khan for a debate
2	Nastaliq Urdu	بھائی موٹے مبین کی طرف سے یوتھیوں سے اظہار تعزیت
	RU Urdu	Bhai motey Mubeen ki taraf sey youthyon sey izhar taziyat
	English translation	Condolences from Fat Mubeen to youthia
Threat
1	Nastaliq Urdu	میں تمہیں نسل کے ساتھ مٹانے کے لئے آنے والا ہوں
	RU	Mein tumeen nasal ky saath mataney ky liyey aney wala hon
	English translation	I am coming to wipe you out with the generation
2	Nastaliq Urdu	اب تک میں تمہیں چھوڑ رہا تھا، لیکن اب میں تمہیں قتل کر دوں گا
	RU	Ub tak mein tumeen choor reha tha, lakn ub mein tumeen qatal kr don ga
	English translation	Until now I was leaving you, but now I will kill you
Curse
1	Nastaliq Urdu	ملک کی بدنامی کرنے والے کو اللہ ننگی موت دے۔
	RU	Mulk ki badnami kerney waley ko Allah nangi moot dey
	English translation	May Allah give a disgraceful death to those who defame the country.
2	Nastaliq Urdu	خدا کرے سارے دانت ٹوٹ جائے اس کے مسکرا کر جب وہ کسی اور لڑکی کو دیکھے
	RU	Khuda kerey sarey daant toot jay us ky muskrat kr jab who kissi aur lerki ko dhekhey
	English translation	May God break all his teeth if he smiles when seeing a girl
Offensive
1	Nastaliq Urdu	اپنے فحش بے حیا بھائی سے پوچھ لیں۔
	RU	Apney fehash bey haya bhai sey pooch len
	English translation	Ask your lewd brother.
2	Nastaliq Urdu	چور کی بیٹی۔نس نس مین غریب قوم کاا خون بہ رہا @MaryamNSharif
	RU	@MaryamNSharif choor ki beti. Nas nas mein Ghareeb qoom ka khoon beh reha
	English translation	@MaryamNSharif The daughter of a thief. The blood of the poor nation was flowing in the veins

DOI: 10.7717/peerj-cs.1963/table-5

Preprocessing

Preprocessing is particularly important for tweets processing due to their unique characteristics, such as limited length, informal language, and specific conventions like hashtags and mentions. Therefore, before model construction raw data is cleaned up and transformed into a format that is appropriate for the following steps using a variety of approaches. Following preprocessing steps are applied.

Text cleaning: cleaning techniques are applied to standardize the textual data. This includes removing URLs, emails, and hashtags, as well as handling diacritics (if applicable). By removing irrelevant or noisy elements, the text becomes more manageable and consistent.

Tokenization: it involves breaking down the text into individual units, typically words or sub words. This process aids in creating a structured representation of the text, facilitating further analysis and feature extraction.

Stop words removal: common words are words that do not carry significant meaning in the context of the research study. Removing these words, such as articles, prepositions, and conjunctions, helps to reduce noise and improve the efficiency of subsequent analysis.

Lemmatization: Reducing words to their most basic or dictionary form is known as lemmatization. Words can be considered as a single entity by reverting them to their root forms, which allows for more precise analysis and model training.

Feature extraction

The effectiveness of classification techniques relies on the choice of input feature set. This study employs three types of features: the classical bag of words (BoW) model, specifically term frequency inverse document frequency (TF-IDF), and state-of-the-art representations known as word embeddings. The BoW approach disregards grammar and identifies offensive sentences by checking for the presence of offensive, profane, threat, or abusive words. In this study, both unigram and bigram are explored as feature vectors for the bag of words model. Additionally, the significance of TF-IDF, a well-known information retrieval statistical technique, is investigated. The TF-IDF value of a word indicates its importance for classification within the corpus. For instance, cyberbullying annotated instances are expected to have a higher occurrence of offensive or abusive words. Word embedding is a dense vector representation method employed in deep learning models to capture word meanings based on contextual relationships. The embedding layer, which serves as the initial layer of the deep learning model, learns word representations from training data by considering adjacent words. When two words appear in similar contexts, their dense vector representations become more similar. Word embeddings effectively capture word semantics based on their contextual usage. In this study, pretrained Word2Vec embeddings are utilized to represent tweet-level features. We utilized 100-dimensional embeddings with a window size of 5, and the LSTM model utilizes this embedding layer for word representation. For fastText word embedding of 300 dimension is utilized for unigram, bigram and trigrams.

Classification techniques

In this section, we provide a comprehensive overview of both the traditional supervised machine learning techniques and the state-of-the-art deep learning techniques employed in this study for the detection of cyberbullying from tweets.

For traditional supervised machine learning methods, we utilized support vector machine (SVM), naïve Bayes (NB), and logistic regression. SVM is a popular algorithm that effectively separates instances of different classes by maximizing the margin between them. Naïve Bayes is a probabilistic classifier based on Bayes' theorem, assuming independence between features. Logistic regression is a widely-used linear model that estimates the probability of a certain class.

To leverage the power of deep learning in our cyberbullying detection task, we incorporated two state-of-the-art techniques: long short term memory (LSTM) and fastText. LSTM is a recurrent neural network (RNN) architecture that captures temporal dependencies in sequences, making it suitable for analyzing sequential data such as tweets. The LSTM cell equations for the input, output, and forget gates, as well as the cell state, are defined as follows:

$$i=\sigma \left(x_t{U}^i+s_{t-1}{W}^i\right)$$

$$f=\sigma \left(x_t{U}^f+s_{t-1}{W}^f\right)$$

$$o=\sigma \left(x_t{U}^o+s_{t-1}{W}^o\right)$$

$$g=\tanh \left(x_t{U}^g+s_{t-1}{W}^g\right)$$

$$c_t=c_{t-1}\circ f+g\circ i$$

$$s_t=\tanh \left(c_t\right)\circ o$$where:

• x_t is the input at time t,

• h_t is the hidden state at time t,

• c_t is the cell state at time t,

• f_t, i_t, o_t are the forget, input, and output gates, respectively,

• c_t is the new candidate value for the cell state,

• σ is the sigmoid activation function.

LSTM can effectively model the context and capture long-term dependencies in the text. FastText, on the other hand, is a deep learning-based text classification technique that utilizes word embeddings and n-gram representations. The FastText algorithm is based on the bag of words (BoW) model and employs a linear classifier with a softmax function. The objective function for training FastText can be defined as:

$$J\left(\theta \right)=-{\displaystyle \frac{1}{N}\sum _{i=1}^N\sum _{j=1}^C{y}_{ij}\log \left(p_{ij}\right)}$$

where:

• N is the number of training samples,

• C is the number of classes,

• y_ij is a binary indicator of whether class

• j is the correct classification for sample i,

• p_ij is the predicted probability of class j for sample i

It is known for its efficiency and ability to handle out-of-vocabulary words by utilizing subword information. FastText has gained popularity for its performance in various text classification tasks.

In this study for LSTM model, Adam optimizer with learning rate of 0.001 is used whereas learning rate = 0.5 is used in fastText model. LSTM model is trained for five epochs and 10 epochs are used for fastText training.

By employing a combination of traditional supervised machine learning methods and state-of-the-art deep learning techniques, we aim to explore the strengths and weaknesses of different approaches in the cyberbullying detection domain. This comprehensive analysis will enable us to gain insights into the effectiveness of these techniques and their suitability for handling the unique challenges posed by tweets in the context of cyberbullying detection.

Evaluation metrics

The performance evaluation of the models is conducted using standard evaluation metrics, including recall (R), precision (P), and F1 score. These measures are employed to assess the models’ ability to accurately detect cyberbullying in the Urdu tweets dataset. The equations for these metrics are defined as follows:

$$R={\displaystyle \frac{TP}{TP+FN}}$$

$$P={\displaystyle \frac{TP}{TP+FP}}$$

$$F1=2\times {\displaystyle \frac{P\times R}{P+R}}$$where:

• TP is the number of true positives,

• FP is the number of false positives, and

• FN is the number of false negatives.

These metrics provide a comprehensive evaluation, considering both the model’s ability to correctly identify cyberbullying instances (precision) and its ability to capture all relevant instances (recall). The F1-score balances these two measures, offering a consolidated assessment of model performance.

Error analysis

In order to investigate the reasons behind misclassifications, an analysis of the incorrect predictions made by the best performing technique i.e., fastText is conducted. This comprehensive analysis incorporates both quantitative and qualitative methodologies. The quantitative approach involves the use of a confusion matrix, which offers insights into the distribution of misclassified tweets, regardless of their content. In contrast, the qualitative approach delves deeper into the content of the misclassified tweets, enabling a more nuanced examination of the reasons behind the misclassifications. By combining these two approaches, a holistic understanding of the classification performance is achieved, encompassing both statistical patterns and the contextual information present in the misclassified tweets.

Table 7 provides an illustration of the findings, highlighting that the most significant confusion occurs between the “insult” and “none” classes in the context of cyberbullying prediction. This suggests that the fastText model faced challenges in accurately identifying instances associated with bullying behavior.

In the qualitative analysis, two word cloud figures provide insights into the misclassification of tweets. Figure 1 presents a word cloud of bullying (insult) tweets that were classified as None, whereas word of non-cyberbullying tweets classified as bullying (insult) is shown in Fig. 2.

An interesting observation from this analysis is the significant overlap of vocabulary among the misclassified tweets. Notably, words such as ’America’, ‘Bad’, ‘Laikan(But)’, frequently appear in these misclassified tweets. It is evident that these words are commonly utilized across multiple contexts. This analysis underscores the importance of context and the challenges faced in accurately classifying tweets. The presence of shared vocabulary among misclassified tweets highlights the complexity involved in bullying detection tasks.

Comparative analysis with previous research

In evaluating the efficacy of our proposed approach for threatening language detection within Urdu tweets, we conducted a comparative analysis with previous research works that specifically focused on this aspect. Notable studies, such as those conducted by Amjad et al. (2021, 2022), have provided valuable insights into threat detection methodologies within the Urdu language.

Table 8 presents a comparative summary of key metrics, including precision, recall, and F-measure, specifically focusing on threatening language detection. Our proposed approach, with a dataset size of 12,428, achieves a precision, recall, and F-measure of 84.20%. Notably, this outperforms the results reported by Amjad et al. (2021) and demonstrates comparable performance to their 2022 study.

Our approach excels in the challenging task of threatening language detection, demonstrating superior precision, recall, and F-measure compared to previous studies.