Zero-shot cross-lingual stance detection via adversarial language adaptation

Cross-topic and cross-target stance detection

Cross-topic stance detection aims to detect stance across topics using knowledge transfer. This approach leverages large stance datasets from source topics to improve performance on smaller target topic datasets. However, most existing work in cross-topic stance detection assumes some labelled data in the target topic is available for training. Zero-shot cross-topic stance detection aims to detect stance in an entirely novel target topic with no labelled training data, instead leveraging source topic data and topic-independent features. Though still relatively understudied, several approaches have been proposed for zero-shot cross-topic stance detection, including training on multi-target datasets, topic-invariant representation learning and unsupervised domain adaptation methods which can also be applied to scenarios involving cross-lingual and cross-domain problems.

Zarrella & Marsh (2016) proposed a system for the SemEval-2016 competition on stance detection in tweets that uses transfer learning from unlabeled datasets and an auxiliary hashtag prediction task to learn sentence embeddings. Their approach, which achieved the top score in the task, uses a recurrent neural network (RNN) initialized with features learned via distant supervision on two large unlabeled datasets, and then fine-tuned on a small dataset of labeled tweets. For the same task, Augenstein, Vlachos & Bontcheva (2016) train a Any-Target stance classifier using a bag-of-words autoencoder that learns feature representations independent of any target. Xu et al. (2018) and Wei & Mao (2019) proposed two different approaches for cross-target stance classification, both of which were able to achieve state-of-the-art results on the SemEval-2016 Task 6 test set. Xu et al. (2018) used a self-attention network to extract target-independent information for model generalization, while Wei & Mao (2019) introduced a topic modeling approach that can leverage shared latent topics between two targets as transferable knowledge to generalize across topics. Building upon these advancements, several other notable contributions have been made in achieving state-of-the-art results in cross-topic stance detection, mainly within the context of the SemEval-2016 Task 6. Wang et al. (2020) employed adversarial domain generalization to learn target-invariant text representations. In addition, Hardalov et al. (2021) combined domain adversarial learning with label-adaptive learning to learn input representation specific to the stance target domain, and handle unseen domains using embedding similarities. A more recent work by Ji et al. (2022) establishes a new state-of-the-art for the task by incorporating a model based on meta-learning, which is also trained on multiple targets considering the similarities between targets. More recently, Khiabani & Zubiaga (2023) introduced a model that leverages features derived from the social network, in addition to the textual content of posts, to improve a model for cross-target stance detection, which they showed could be further used to enhance a language model for stance detection (Jamadi Khiabani & Zubiaga, 2024).

Recent approaches in stance detection for unseen targets, or more formally zero-shot stance detection have shown promising results. One of the first works in this area by Allaway & McKeown (2020), introduced a new dataset VAST for zero shot stance detection (ZSSD) and developed a model that learns generalized topic representations to pre-dict stance labels based on similarities between given topics and the topics present in training set. Allaway, Srikanth & McKeown (2021) also proposed a new TOpic ADversarial Network (TOAD) that conditions document representations on topics and further learns domain-invariant features using adversarial training for zero-shot stance detection. Liang et al. (2022) proposed a novel approach for ZSSD that frames the distinction of target-invariant and target-specific stance features as a pre-text task to better learn transferable stance features. In the case of Twitter, an unsupervised zero-shot stance detection framework Tweet2Stance, proposed by Gambini et al. (2022) uses content analysis of users’ Twitter timelines with an NLI based zero-shot classifier to infer their stance towards a political-social statement.

Cross-lingual stance detection

Current approaches to stance detection are well-studied in English but have received less attention in other languages and cross-lingual settings. A novel approach to improve cross-lingual stance detection introduced by Mohtarami, Glass & Nakov (2019) uses memory networks along with a contrastive language adaptation component to effectively align source and target language data with the same labels and separate the ones with different labels. To facilitate research in multilingual stance detection, Zotova et al. (2020) introduced a new Twitter dataset for the Catalan and Spanish languages annotated with Stance towards the independence of Catalonia. X-stance is another multilingual stance detection dataset introduced by Vamvas & Sennrich (2020) that includes text in German, French and Italian, covering a wide range of topics in Swiss politics. A more recent work by Hardalov et al. (2022) presents a comprehensive study of cross-lingual stance detection, covering 15 datasets in 12 languages that demonstrates the effectiveness of pattern-based training approaches and pre-training using labeled instances from a sentiment model.

Existing research on cross-lingual stance detection has however been limited to few-shot settings, where a small portion of labeled data is available for the target language. In our work, we are interested in going further by enabling zero-shot cross-lingual stance detection where there is no labeled data at all for the target language. With this objective in mind, we propose our novel model MTAB, which leverages adversarial learning through translation-augmented training data for zero-shot cross-lingual stance detection.

Vaccine stance detection

The task of vaccine stance detection is still relatively new, but there has been significant progress in recent years, given the rise of vaccine misinformation on social media. Twitter is a rich source of data for vaccine discourse, and many researchers have used Twitter data to develop mono-lingual and cross-lingual vaccine stance detection models. The method introduced by Bechini et al. (2021) particularly demonstrates the importance of incorporating user related information in the analysis of public opinion on Twitter and combine that with text classification techniques to asses opinions on vaccination topic in Italy. In the month following the announcement of the first COVID-19 vaccine, Cotfas et al. (2021) provided insights into the dynamics of opinions on COVID-19 vaccination based on tweets and compared classical machine learning and deep learning algorithms for stance detection. In a related study conducted on the same period, the authors examined vaccine hesitancy discourse on Twitter using several text analysis techniques such as linear discriminant analysis (LDA), Hashtag and N-gram analysis (Cotfas, Delcea & Gherai, 2021). Addressing this challenge in low-resource languages, Küçük & Arıcı (2022) introduced the first dataset for Turkish tweets that is annotated with stance and sentiment labels towards COVID-19 vaccination. In addition, the first manually annotated Arabic tweet dataset related to the COVID-19 vaccination campaign (ArCovidVac) proposed by Mubarak et al. (2022) consists of stance annotated tweets from different Arab countries and delivers key insights into the discussions and opinions related to the COVID-19 vaccination campaign on social media in the Arab region.

The domain of vaccine hesitancy presents an ideal scenario for our research, as it is an issue of global interest, which is discussed and opinionated across languages in the world, which has in turn led to the creation of labeled datasets in different languages. In our case, we use datasets in four languages, namely English, French, German and Italian, to enable our zero-shot cross-lingual stance detection study.

Training datasets

We experiment in the scenario where a model is only trained on a resource-rich language, English, to then test it on other languages. As such, our training data only contains data in the English language, while test data will be written in other languages. We therefore set up the scenario where the target languages found in the test sets are not seen during training.

To achieve this objective and create our English-only training data, we combined data from three different sources of English tweets on COVID-19 vaccines belonging to different timelines during the period of 2020 and 2021 to create a more diverse and representative sample. We aggregate the following three datasets:

• (1)

  Vaccine Attitude Dataset (VAD) by Zhu et al. (2022): The VAD dataset (Zhu et al., 2022) consists of an extensive collection of 1.9 million English tweets obtained between February 7th and April 3rd, 2021. The authors annotated a random subset of 2,800 tweets from this corpus with stance labels and aspect characterizations. We use this smaller, labeled subset as part of our training data.

• (2)

  COVID-19 Vaccination Tweets dataset by Almadan et al. (2022): The work of Almadan et al. (2022) presented a systematic methodology for annotating tweets with respect to their stance toward COVID-19 vaccination. This approach included the development of a comprehensive codebook for stance annotation and the use of keyword and hashtag based sampling techniques for Twitter data. Leveraging this coding framework, the authors released a stance annotated dataset for research purposes (Almadan et al., 2022), which we also incorporate into our training dataset.

• (3)

  COVID-19 Vaccine Stance Dataset by Cotfas et al. (2021): The authors Cotfas et al. (2021) collected 2,349,659 tweets related to COVID-19 vaccination during the period of November 9th to December 8th, 2020 using a set of predefined keywords via the Twitter API. From this collection, they curated a refined and balanced stance annotated dataset comprising 3,249 tweets, which was made publicly accessible (Cotfas et al., 2021). We incorporate this smaller, labeled subset of their dataset into our training data.

After aggregating the three labeled data sources, our final training data contains 4,493 tweets, of which 2,276 are labeled positive, 1,141 negative, and 1,076 neutral. We randomly sample 90% of the aggregated data for model training and 10% for validation on English data. See Table 1A for more details on the training data, including the final distribution of labels.

Test data

For testing our model on non-English data, we use data pertaining to three languages—French, German and Italian—from the VaccinEU (Giovanni et al., 2022) dataset as introduced by Giovanni et al. (2022). To study the impact of online conversations on COVID-19 vaccines, the researchers used a list of vaccine related keywords and collected a large dataset of over 70 million tweets in three different languages, namely French, German, and Italian, from November 1^st, 2020 to November 15^th, 2021. From this large-scale dataset, the authors labeled small samples for each language, providing four labels—Positive, Negative, Neutral, and Out of Context. However, for our experiments, we only focus on the Positive, Negative, and Neutral classes, as detailed in Table 1B. We can observe that labels are unevenly distributed across different languages, which is itself an additional challenge for our research, but one can realistically expect that these variations across languages will naturally happen, not least because of cultural differences affecting level of support towards vaccines. Given the importance of label distributions in our cross-lingual research, we delve into performance analysis across labels in our analysis of results, in addition to looking at general performance scores aggregated for all labels.

Data pre-processing

Due to Twitter’s terms of service and privacy concerns, these data were made available only using Tweet IDs, and hence, we first used an additional application called Hydrator (https://github.com/DocNow/hydrator) to retrieve the actual tweet content associated with the IDs. Due to the age and controversial nature of the tweets, some of them were lost during retrieval. This however happens generally with Twitter datasets, as research has shown that a percentage of tweets inevitably tends to disappear (Zubiaga, 2018), and hence we conduct our research with the tweets available at the time of our collection, which are the statistics we report in Table 1.

In the first step of data pre-processing, we standardize the stance labels of the three datasets that were combined for model training. This ensures the diverse labeling conventions present in those datasets are harmonized enabling consistent and integrated experimentation.

Once the data were ready, and to prepare the tweets for further analysis and modeling, we carried out a series of textual pre-processing steps, as follows:

• Removal of URLs, emojis, mentions, smileys, and numbers: In the interest of focusing on the textual content of the tweets, we used the “tweet-preprocessor” (https://pypi.org/project/tweet-preprocessor/) library to eliminate URLs, emojis, user mentions, smileys, and numbers from the tweet text.

• Handling retweets: Retweets often contain a standard “RT @” prefix that can affect the consistency of the text by adding noise to it which provides no information about the stance of a tweet. We removed this prefix to standardize the format of retweets while preserving the essence of their content.

• Handling hashtags: To maintain the relevance of hashtags as keywords, we opted to remove only the ‘#’ symbol, retaining the keyword itself. This approach ensured that the hashtags contributed to the context of the tweet without introducing any extraneous characters, while we retain the content of the hashtag whenever this forms part of the sentence, which we did not want to break.

The resulting texts, after the three preprocessing steps above, are then fed to our models for training or testing.

Translation augmentation

Translation augmentation allows a model trained in one language (source language) to gain exposure to diverse linguistic contexts and structures present in other languages. We expanded our English training dataset by incorporating translations of training examples into our target languages. As mentioned in “Training Datasets”, 90% of the total source language data is used for training and the remaining 10% is used as the validation set. For obtaining translations, we used the Opus-MT (https://github.com/Helsinki-NLP/Opus-MT) model from Helsinki-NLP, accessible via the Easy-NMT (https://github.com/UKPLab/EasyNMT) package, which we chose as a competitive and open source solution. By providing translations of the training data into multiple languages, the model can learn to recognize common patterns, sentiment expressions, and stance cues that transcend linguistic boundaries. Different languages may employ distinct grammatical structures, vocabulary, and cultural nuances. By training on a multilingual dataset that includes augmented training data incorporating translations, the model learns to identify semantically equivalent expressions and sentiment cues in different languages.

This approach aligns with prior work that has successfully leveraged translation to enhance zero-shot cross-lingual transfer learning. Translation-based augmentation has been widely used in zero-shot settings to improve model adaptability without requiring human-annotated labels in the target language. Ding et al. (2022) employ translation to align embeddings of English-labeled data with target languages, facilitating more effective cross-lingual transfer using improved multilingual representations. Fazili, Agrawal & Jyothi (2024) use LLMs to generate synthetic English data for multiple NLP tasks and translate them into target languages to create relevant data sets for low-resource languages, thereby boosting zero-shot cross lingual performance. Similarly, Riabi et al. (2021) generate synthetic question-answer pairs in English and translate them into other languages, thereby augmenting the training data available for cross-lingual question answering. These studies demonstrate that machine translation can be a practical and effective strategy for mitigating the scarcity of labeled data in low-resource languages, while still adhering to the principles of zero-shot learning.

Thus, our approach remains within the zero-shot paradigm, as it does not introduce any manually labeled data in the target language. Instead, it leverages labeled data from a high-resource language to enhance model generalization across multiple languages, ensuring a fair and effective adaptation to new linguistic domains.

Multilingual encoder and stance classifier

Multilingual BERT (mBERT), pre-trained on 104 languages, has been shown to perform well for zero-shot cross-lingual transfer, especially when fine-tuned on downstream tasks. Therefore, we employed mBERT to finetune a multilingual encoder that can generate contextual embeddings tailored to our task-specific tweets. Simultaneously, we jointly trained a stance classifier, optimized to categorize stances based on these embeddings.

Adversarial language adaptation

Our approach for language adaptation was inspired by the adversarial domain adaptation (ADA) with distillation technique proposed by Ryu & Lee (2020) for unsupervised domain adaptation. Their method, ADA, aims to enhance the performance of BERT in the face of domain shifts. ADA combines the adversarial discriminative domain adaptation (ADDA) framework (Tzeng et al., 2017) with knowledge distillation.

To adapt our model to the target languages, we utilize unlabeled French, German, and English tweets from the VaccinEU dataset. Our adaptation process comprises three key steps, described as follows:

• (1)

  In Step 1, the multilingual encoder and stance classifier are fine-tuned on labeled stance data.

• (2)

  In Step 2, the target language encoder is adapted through adversarial training and distillation. This involves training the target encoder and discriminator alternately in a two-player game. The discriminator learns to distinguish between English and non-English representations, while the target encoder learns to confound the discriminator. Knowledge distillation serves as a regularization technique, preserving information learned from the English training data.

• (3)

  Finally, in Step 3, the adapted target encoder and stance classifier are utilized to predict stance labels for data in the target languages.

We release the code used to train and evaluate our models on GitHub (https://github.com/amcs18pd05/MTAB-cross-lingual-vaccine-stance-detection-2).

Models

We compare a range of models that we test against our proposed MTAB model. In the absence of existing models for zero-shot cross-lingual stance detection, we use a strong, general-purpose baseline model, multilingual BERT (mBERT). In addition, we also compare with an ablated version of our MTAB model, which we refer to as MTAB-noTL. We describe next the models we compare in our experiments:

• mBERT—Baseline model: This model involves fine-tuning a multilingual instance of the BertForSequenceClassification model from HuggingFace on English training data. It serves as our baseline model for stance detection.

• MTAB: This is our model proposed above in “MTAB: Model Architecture”. It contains the combined English vaccine stance dataset along with translations of the English training set into French, German, and Italian.

• MTAB-noTL: This is an ablated version of our proposed MTAB, which does not make use of translation-augmented data and therefore could be possibly limited for the capabilities of the adversarial learning component. This model in turns tests the contribution of our proposed adversarial learning component on top of the translated data.

For each of the three models above, we test two different architectures, with and without the adversarial learning component, which we indicate in the name of the model by either including or not ‘+ Adversarial learning’ at the end of it. This leaves us with a total of six model variants, i.e., the three models listed above with and without adversarial learning component.

Model hyperparameters

We experiment with various sets of hyperparameters for our models on the validation data, and selected the ones that yielded the best results. Across all our experiments, we maintained a maximum sequence length of 128 and a batch size of 32. However, the learning rate and the number of epochs varied among the three different architectures.

In the case of MTAB, we conducted training for nine epochs using an Adam Optimizer. We experimented with learning rates of $1e^{-5}$, $2e^{-5}$, $3e^{-5}$, $4e^{-5}$, and $5e^{-5}$ and settled at $5e^{-5}$, which resulted in best scores on the validation set. This learning rate was used for the joint training of the language encoder and stance classifier. For the adversarial learning setup in MTAB, again we used an AdamW Optimizer. The learning rate was $1e^{-5}$ for both the Discriminator and the Generator, and each of the target languages went through five training epochs.