Syntax and prejudice: ethically-charged biases of a syntax-based hate speech recognizer unveiled

An Explainable Syntax-based Hate Speech Recognizer

Our Hate Speech Recognizer stems from a recent result on visualizations of activations of syntactic trees for decisions of neural networks (Zanzotto et al., 2020).

We then propose an Unintended-bias Visualizer based on Kermit modeling (KERM-HATE¹) that is a model for hate speech recognizer consisting basically of three components:

1. KERMIT model (Zanzotto et al., 2020).

2. A transformer model.

3. A fully-connected network.

The structure of KERM-HATE (see Fig. 2) makes it a particular model, because it combines the syntax offered by KERMIT with the flexibility of transformers and the possibility to switch a representation space ℝⁿ → ℝ^m of a fully connected network.

The first component, that is, KERMIT, allows the encoding and the visualization of the activations of universal syntactic interpretations in a neural network architecture.

KERMIT component is itself composed of two parts:

1. KERMIT encoder, which converts parse tree T into embedding vectors, and a multi-layer perceptron that exploits these embedding vectors.

2. KERMITviz, its viewer, makes KERMIT the most relevant component inside KERM-HATE.

KERMITviz gives the possibility to extract as output not only the classification target but especially the colored parse tree with the activation value of every single node that composes a generic sentence (see Fig. 1 and Fig. 2). KERMITviz is the real game changer for our purposes as it allows us to visualize how decisions are made according to activations of syntactic structures.

The transformer component consists of BERT (Devlin et al., 2018). KERM-HATE uses the base version of BERT with uncased setting and pre-trained in English language.

The final component is a four-layer fully-connected neural network and boost performances. The component acts on the concatenation of KERMIT output and BERT output. In this network, KERM-HATE changes the input representation space four times: ℝⁿ → ℝ^m → ℝⁿ → ℝ^m. The result is passed to the decoder layer returning the target value.

Prejudice in machine learning models

Since we rely on explainable machine learning models, we can revitalize the operational definition of prejudice of statistical functions (Kamishima et al., 2012) within the context of neural networks:

Prejudice-in-NN A learned model shows prejudice when takes decisions by overusing identity terms.

This definition is in line with the definition of prejudice given in Quasthoff (1989) but differs from the definition of un-intended bias given in Dixon et al. (2018) that focuses on performances of learned models. Moreover, using this definition, we can provide a measure of degree of perceived prejudice since links prejudice to single decisions of learned models.

Possibly unbiased training and validation corpus

Davidson et al. (2017) collected and annotated a possibly unbiased dataset containing about 25,000 tweets, which can be used for our purposes. In fact, the procedure for collecting and annotating the corpus may have reduced the lexical probabilistic bias, although this procedure has been designed for a different purpose.

The procedure for collecting the Davidson et al. dataset stems from an important point: tweets are selected starting from 1,000 terms from HateBase (www.hatebase.org). Therefore, these tweets contain possibly ethically-charged hate terms. Then, at the end of the annotation, tweets of all classes may contain hate terms. This should guarantee that probabilistic lexical bias is reduced to a minimal amount.

The Davidson et al. dataset was manually annotated using CrowdFlower (CF) workers. Workers were asked to label each tweet as one of three categories: Hate speech, Offensive language but not hate speech, or Neither: non-offensive and non-hate. The distinction between Hate speech and Offensive language is the real reason behind this corpus. Each tweet was annotated by at least three annotators. The inter-annotator-agreement score provided by CF-workers was 92%. As another factor to reduce ethically charged bias, annotators paid attention to inter-words and statements often labeled as hate speech which are not: an example is ni**a labeled as Hate speech but used as an interchange by African-Americans (Warner & Hirschberg, 2012). The resulting dataset contains 1,430 tweets labeled as Hate speech, 19,190 tweets as Offensive language and 4,163 tweets as Neither (non-offensive and non-hate).

Then, Davidson et al. dataset can be used for our study as it should help in disentangling the positive or negative use of some given hate words.

Challenging hate speech recognizers on hot topics

This is another important aspect of our study. We aimed to collect fresh corpora generated by identifiable groups of people and which may contain potentially offensive or hate speech utterances. After an initial analysis for selecting the topics (‘Two discussed and divisive topics of 2020’ section), ‘Black Lives Matter corpus’ and ‘United States presidential election corpus’ sections describe how we collected the two novel corpora.

United States presidential election corpus

The second analyzed event of 2020 was the United States presidential election held on November 3, 2020. Our aim was to generate a corpus based on two datasets containing tweets of Americans politically aligned with the two major contemporary political parties in the United States: the Democratic party—with Joe Biden as candidate—and the Republican party—with Donald J. Trump as candidate. In Kaggle there are two datasets about the US 2020 election³: the first dataset—with 775,054 tweets—contains all the tweets having #Biden or #JoeBiden as hashtags, while the second contains 958,580 tweets having #Trump or #DonaldTrump as hashtags. Both datasets were generated from October 15, 2020 to November 8, 2020 and hold for each tweet 11 other fields including geolocation. Using this information, we filtered all tweets written in the United States obtaining only tweets created by possible American voters. Although the two datasets contain the names of presidential candidates this does not imply that they are politically aligned.

Accordingly, we studied the hashtags and slogans used by the major exponents of the two political parties. For each party, we selected the verified Twitter account of: the presidential candidate, the vice president candidate and their political party. Table 2 shows the Twitter profiles analyzed for each party.

Table 2:

Twitter verified profiles selected for each party.

DOI: 10.7717/peerjcs.859/table-2

Then, in each group, we compiled a ranking list of the most used hashtags (the list of hashtags most used by both parties is shown in Appendix A). Finally, we generated our political datasets filtering from the Kaggle dataset holding only the tweets geolocated in the US, all the tweets with at least one of the hashtags present in the lists. In particular, regardless of whether the dataset contains the hashtag #Biden rather than #Trump, a tweet is considered democratic—and therefore included in the appropriate dataset—if it has at least one hashtag present in the democratic list. The same is applied to the Republican dataset. So we obtained a Democratic dataset of 46,898 tweets and a Republican dataset of 35,903 tweets. So, our United States presidential election corpus is composed of 82,801 tweets.

In Table 3 we show some sentences in both the Democratic dataset and Republican dataset that make up the United States presidential election corpus. Unlike the BLM corpus (ref. ‘Black Lives Matter corpus’ section) - where the potentially offensive keywords are more obvious (ref. Table 1), in this case, we included sentences that clearly and concisely explained the political orientation of voters.

Table 3:

United States presidential election corpus example sentences.

#	Sentence
1	He proves that he is the worst president EVER.
2	Good morning Twitter democratic voters.
3	As a supporter you should probably avoid words like ’immoral’.
4	Damn right I support him 100 percent.
5	Biden has always worked to help stuttering kids.
6	Democrats think the Constitution is more important than President.
7	The Evilness of human beings should be measured in TRUMPS.
8	Glad u got out of the house! D**K!
9	Donald Trump 4 more years!

DOI: 10.7717/peerjcs.859/table-3

To test whether the two datasets that make up the corpus actually reflect the political ideologies of American voters, we analyzed the geolocation of tweets and compared them with the National Exit Polls. In all states where the number of geolocated tweets was relevant, we can confirm how the prevalence of tweets placed in a specific dataset actually reflected the outcome of the exit poll. A detailed list of the states that most affected the two datasets—and those that have a larger gap—is shown in Appendix B.

Experimental set-up

For the first set of experiments, which is devoted to understand whether explicit syntactic representation can be useful in building hate speech recognizers (HSRs), we experimented with our model KERM-HATE over three different already annotated datasets: the Davidson et al., the Waseem & Hovy and the Founta et al. dataset. The Davidson et al. dataset is our main annotated dataset and it is used also to train the final model (see ‘Possibly unbiased training and validation corpus’ section). The Waseem & Hovy dataset consists of 16,849 tweets annotated in three classes (Racism, Sexism and Neither (non-racism and non-sexism). Finally, the Founta et al. dataset consists of 91,951 tweets annotated in four classes (Abusive, Hateful, Normal and Spam). For the experiments, these datasets have been randomly split in 80% for training and 20% for testing.

In this first set of experiments, we compared our approach to three HSRs based syntactic-agnostic transformers, that is, BERT_BASE (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019), and to two basic explicit-syntactic-aware models, that is KERMIT BERT and KERMIT_XLNet (Zanzotto et al., 2020). Finally, as suggested in Zanzotto et al. (2020), we explored also two special versions of BERT, that is, BERT_REVERSE and BERT_RANDOM, which constitute the core for testing if the task is syntactically-sensitive. In detail, let S = {w₁, …, w_n} a sentence consisting of n words (|S| = n). In BERT_REVERSE, the input sentence S is given in reverse way S = {w_n, w_n−1, w_n−2, …, w₁} while in BERT_RANDOM the input sentence S is given in random way S = {w_i, w_j, w_k, …, w_z} (with i ≠ j ≠ k ≠ z and i, j, k, z ≤ n). To assess statistical significance, each experiment is repeated 10 times with different seeds for initial weights.

The meta-parameters utilized in training the models are the following and so for KERM-HATE, KERMIT BERT and KERMIT _XLNet:

1. The tree encoder is on a distributed representation space ℝ^d with d = 4000 and has penalizing factor λ = 0.4 (as suggested for tree kernels in Moschitti (2006)).

2. Constituency parse trees have been obtained by using Stanford’s CoreNLP probabilistic context-free grammar parser (Manning et al., 2014).

KERM-HATE’s fully-connected four-layers network change the representation space four times: ℝⁿ → ℝ^m → ℝⁿ → ℝ^m where m = 2, 000 and n = 4, 000, before concluding with the final classification layer. A dropout layer (Srivastava et al., 2014) with 0.1 is added for each layer. Class weight w_i is inversely proportional to its class_i (C_i) cardinality ($w_i=\frac{1}{\left|C_i\right|}$). BERT_BASE, RoBERTa and XLNet, both stand alone or as components of KERM-HATE, was implemented using Huggingface’s transformers library (Wolf et al., 2019). The input text for BERT_BASE has been preprocessed and tokenized as detailed in Devlin et al. (2018). The optimizer used to train all the models is AdamW (Loshchilov & Hutter, 2019) with the learning rate set to 2e⁻⁵. All models used a batch size of 32 and are trained for 4 epochs. Our hardware system consists of: 4 Cores Intel Xeon E3-1230 CPU with 62 Gb of RAM and 1 Nvidia 1070 GPU with 8Gb of onboard memory.

The second set of experiments, which aims to understand if syntax can help in wiping out prejudice, is organized on three tests –the Blind test, the Inside out test and the Prejudice test –carried on our novel collected corpora (described in ‘Black Lives Matter corpus’ and ‘United States presidential election corpus’ sections) with the help of 24 different annotators. The partition of annotators into the three tests does not depend in any way on their cultural background since it is for all different with greater or lesser knowledge of syntax, machine learning and natural language processing. In addition, none of the annotators was born or stayed - during the execution of the test - in the United States of America and nobody has any emotional ties with that land. Finally, all the annotators do not belong to the ethnic groups listed and used in the tests. These restrictions have been used in order to make tests as objective and clear as possible. A methodological note: the design of these three tests is sequential, that is, the result of one test led to the definition of next one. Hence, questions to annotators posed in test_i depend on the results obtained from test_i−1. Limitations of test_i−1 lead to the the selection of sentences presented in test_i. Fleiss’kappa score (Fleiss, 1971) measures the inter-annotator agreement. In all these tests, we used BERT_BASE and KERM-HATE trained on the Davidson et al. dataset.

The Blind test aims to check the accuracy of BERT_BASE and KERM-HATE on a novel, divisive datasets. This test is a classical annotation with respect to the guidelines used in the Davidson et al. dataset. The question posed to annotators for each sentence was: “Based on the guidelines you read, how would you label this sentence?”. The possible answers were the targets given by the Davidson: Hate speech, Offensive language and Neither. Annotators are unaware of the choice generated by the two models. For this reason, the test is called blind. In this test, we randomly selected 34 sentences from the two corpora and each sentence received 10 annotations from the 10 different annotators. The final label given to the sentence is assigned with a majority vote.

The inside-out test aims to evaluate if the annotators agreed or disagreed with the labeling given by KERM-HATE model given the post-hoc explanation of the heat parse tree. This test contains 25 sentences with an overlap of 30% of the sentences in the Blind test. 10 annotators participated in this test. The question posed was: “Based on the label and parse tree activation values, do you agree with the label given by KERM-HATE?” The possible answers were Yes or No. Moreover, if the answer was No, it was asked to indicate what the reasons were. The name of the test comes exactly from the possibility given to the annotators to look inside the model in order to understand its output.

The Prejudice test aims to understand whether classification of KERM-HATE model relies upon prejudice: ethnicity, geographic or gender bias. The set of sentences is the same as Inside out test. Sentences are given along with the classification of KERM-HATE and the heat parse tree The question posed was: “Based on the label and parse tree activation value, do you think this label was obtained through any bias?”. The possible answers were Yes or No. Also in this case—if the answer was Yes—it was asked to specify the reason why the system decision is biased. 4 annotators participated in this test and each annotator answered questions for the 25 sentences.

Results and discussion

Syntactic information is useful to significantly increase performances of Hate Speech Recognizers (HSRs) (see Table 4) and KERM-HATE is the best model. The preliminary test of syntactic-sensitive task seems to suggest that the hate speech phenomena is quite sensitive to syntax. Indeed, in Waseem & Hovy and Founta et al. datasets the performance of BERT_BASE is significantly better than BERT_REVERSE and BERT_RANDOM. However, in Davidson et al. dataset the gap between the three models is smaller but still in BERT_BASE favor in all tests conducted. Nevertheless, in the three annotated datasets, KERMIT and KERMIT_XLNet significantly outperform BERT_BASE and XLNet, respectively. The first pairs of models use explicit syntactic information whereas the second pair does not. Moreover, our KERM-HATE outperforms all the other models, including RoBERTa, which has a very high average accuracy.

Hence, KERM-HATE trained on Davidson et al. (see ‘Possibly unbiased training and validation corpus’ section) is a good candidate for exploring whether or not it holds the prejudice of the corpus where it is trained.

On the novel, divisive corpora (ref. to ‘Two discussed and divisive topics of 2020’ section), trained KERM-HATE and BERT_BASE behave differently (see Fig. 3). Then, these two models definitely look at different features of input sentences. Generally, KERM-HATE seems to be less prone than BERT_BASE to tag sentences as Offensive language. On the other hand, BERT_BASE is oriented to tag sentences as Offensive language in the Black-lives-matter corpus and this predisposition can also be seen in the other two datasets.

The Blind test on the novel corpora confirms that KERM-HATE is better than BERT_BASE (see Table 5). In fact, BERT _BASE matches only 11% of the labels assigned with majority vote by annotators whereas KERM-HATE matches 41% of the labels. However, hate speech recognition in these corpora is rather difficult. In fact, the inter-annotator agreement on the blind test is 0.19 (Slight agreement).

Moreover, decisions of KERM-HATE seems to convince annotators when presented along with explanations. In the Inside-out test, annotators have an average agreement with KERM-HATE of 0.52(±0.08), which is higher with respect to the Blind test (see Table 5). The task confirms to be a very subjective task (Basile et al., 2021) as the inter-annotator agreement is 0.24 (Fair agreement). According to the analysis of this group of annotators, when KERM-HATE fails, the reason seems to be that it focuses on words correlated with ethnicity, continents or nations, proper names and gender⁴. This test suggests that the system is still ethically-biased and, thus, we performed the Prejudice test.

In contrast with what hypothesized, syntactic information does not wipe out prejudice from our HSR. In fact (see Table 5), the average perceived prejudice of KERM-HATE is definitely high, that is, 0.55(±0.04) and annotators have the very high agreement of 0.87 on this fact (Almost perfect agreement).

Sample perturbation analysis

Since KERM-HATE provides heat parse trees as a post-hoc explanation of decisions, it offers the opportunity to carry out perturbation analysis.

The perturbation analysis is a handy method to analyze if comments are really classified with prejudice. The basic idea is to replace one or more keywords in a sentence S_A generating a sentence S_B. The action carried by S_A is the same as S_B but participants or their adjectives are altered. With this operation, the colored syntactic trees derived from KERM-HATE of S_A and S_B should be different and it will be possible to compare them and visualize eventual syntactic biases.

In the Fig. 4 is shown an example: given the sentence “Max is a [W] boy”, KERM-HATE reacts and uses differently parse trees if W is “African” (A)) or “American” (Figure 4B). Firstly, the classification is different: ‘Offensive language’ and ‘Neither’, respectively. Secondly, the active part of the parse tree is different. For the sentence with “African”, the structure [Subject] is [Object: African] seems to be particularly active. Conversely, for the sentence with “American”, this part loses importance that is gained by the second noun phrase. This shows how decisions are correlated with divisive terms. Other examples are given in the additional material (Appendix D).