Comparing diversity, negativity, and stereotypes in Chinese-language AI technologies: an investigation of Baidu, Ernie and Qwen

Chinese social groups

To collect language models’ and search engines’ views on Chinese society, we combined the social groups described in Huang & Xiong (2024) and Zhao et al. (2023). Huang & Xiong (2024) identified different types of biases in Chinese culture: categories such as Disability, Disease, Ethnicity, and Gender were extracted from China’s “Employment Promotion Law” (Brown, 2009) while additional categories like Age, Education, and Sexual Orientation were sourced from social media discussions on “Weibo” and “Zhihu”¹ . They also employed “CNKI” (https://www.cnki.net/), a Chinese knowledge resource that includes journals, theses, conference papers, and books, to review qualitative and quantitative studies on these biases. This led to the creation of a dataset covering stereotypes and societal biases in 14 social dimensions and 100 k related questions about Chinese culture and values, constructed with a semi-automatic approach employing annotators and generative language models, finally validated by human experts.

Zhao et al. (2023) followed the explicit bias specifications category from Caliskan, Bryson & Narayanan (2017), Lauscher et al. (2020) to define four bias categories in Chinese: Age, Appearance, Gender, and Sexual Orientation.

After merging the social groups from the specified sources, one of the authors, who is a Chinese national and native speaker, selected specific social groups based on the following principles:

• Remove social groups that were highly time-specific, such as “COVID-19 patients”, “Post-2011s generation” and “Post-2012s generation”.

• Merge similar groups where appropriate. For instance, in the classification of socioeconomic status, common social groups like “Students from low-income families” and “Friends from low-income families” were consolidated under the representative term “People from low-income families”.

• Expand some groups into different sub-groups that provide more details. For instance, we expanded “Graduates from lower-tier universities” to encompass “Graduates from ordinary first-tier universities” and “Graduates from ordinary second-tier universities”.

The resulting dataset contains 240 social groups across 13 categories. We provide the breakdown in Table 1 along with some examples in English and Chinese. The full code and data to reproduce our results are publicly available in the repository associated with this article (https://doi.org/10.5281/zenodo.14148258). The workflow of our analyses is shown in Fig. 1.

Data collection

Pre-processing

For what concerns Baidu search engine completions, we pre-processed the data following our previous work (Liu et al., 2024) labeling as inconsistent the output that does not meet the following conditions:

• 1.

  The response does not follow the original query, e.g., “Why are Pansexual so [ $\dots$]” $\to$ “What does pansexuality mean?”

• 2.

  The response does not contain the original social group, e.g., “Why are all Hindus so [ $\dots$]” $\to$ “Why are there so few Indians who believe in Buddhism?”

In the next section, we provide a brief quantitative analysis of these cases, which represent only a minority of the completions generated by Baidu.

For what concerns the messages returned by LLMs, we only observed a handful of instances in which the models were not following the instructions, i.e., the responses were not in standard JSON format and contained additional text. Specifically, we identified three situations:

• The response answers the query with a disclaimer on providing an unbiased response, e.g., “This question may be subjective and involves exploring stereotypes about divorced women. Since this is a sensitive topic, we provide a cautious and unbiased answer. The following are possible completion options, output in JSON format: …”

• The response does not answer the query, e.g., “This question may relate to stereotyping or prejudice against a particular group of people …, Therefore, I will not provide additional options that may reinforce such stereotypes”

• The response does not directly answer the query but adds an explanation, e.g., query: “Why are people from wealthy families so …” response: “It may be because of their family education, social resources and economic conditions, which make it easier for them to obtain high-quality educational resources, build extensive connections and have more opportunities, thus gaining relative advantages in life and career”

We included only the latter types of responses in our dataset, excluding the first two, which amount to nine and 14 queries with invalid responses in Qwen and Ernie respectively, all corresponding to the template “Why are all group so … ?”. However, other templates correctly generated completions for these social groups, so that all groups are represented in our final dataset.

Data description

We provide a first description of the collected data in Fig. 2. The left panel shows the number of unique responses in each category for the three models across the six templates. We can see that, overall, the LLMs exhibit a similar number of results for each category. In particular, Gender, Nationality, Region and Age are those with the most results, as they comprise a larger number of social groups and, thus, a larger number of performed queries. A similar pattern can be appreciated at the group level (cf. right panel), with Qwen exhibiting the largest number of unique completions on average (median = $38$) across the six templates, followed by Ernie (median = $32)$. Baidu, instead, returns a much smaller amount of results (median = $12$), as the search engine exhibits a very high proportion of inconsistencies, as shown in Fig. 3. Approximately 72% ( $8\text{,}803/12\text{,}149$) of Baidu completions do not follow the original query or do not contain the social group of the template in the response, as reported in the previous subsection. To keep consistency with the output returned by Ernie and Qwen, we filter out these completions and retain the remaining ones ( $3\text{,}346/12\text{,}149$) for the following analyses.

Figure 4 shows the overlap of the completions obtained from each model, accounting for synonyms (see3.5). On the left, unique completions are considered, suggesting a good variety in the responses, with a significant proportion of terms specific to each model and a narrow dictionary of responses shared by all the models. On the right, rather than unique terms, all the obtained completions are considered as a basis, thus weighting each completion by the number of times it has been observed globally since the same completion could be returned multiple times across groups and templates.

Measuring text similarity in the responses

We investigate the extent to which the models encode a diversity of views when describing different social groups by analyzing the variety of completions returned when prompted about each group. Observing such behaviour in language technologies can result in better cultural sensitivity and ethical standards for AI development and usage (Kirk et al., 2023). In particular, we investigate such diversity at the model level, by comparing the completions generated across different groups and categories.

We first employ the Jaccard similarity, which measures the proportion of common completions generated for two social groups (Murali et al., 2023; Zhang et al., 2023), defined as: $J(A,B)=\frac{|A\cap B|}{|A\cup B|}$ where A and B are the sets of completions generated for the queries pertaining to two different social groups. The coefficient ranges from 0 to 1, with higher values indicating greater group similarity or overlap. We adjust this measure by considering synonyms in the sets of completions for two groups, adopting a Chinese-Synonyms dictionary (https://github.com/jaaack-wang/Chinese-Synonyms) which contains 18,589 word-synonym pairs. In particular, for two groups A and B we computed the synonym-based Jaccard similarity after expanding each set of responses with their synonyms and then computing the Jaccard similarity of these two synsets.

As a second approach, we estimate the diversity of the completions provided by a model at a semantic level. We first embed the completions with bert-base-chinese (Cui et al., 2021) and measure the accuracy of the embeddings in predicting whether another completion belongs to the same category, or to the same group—reported as the same-category and same-group tasks. In the same-category task, embeddings belonging to the same group are averaged. As a prediction rule, we calculate if the cosine similarity between two completions’ embeddings is above a certain threshold. The datasets are generated, for each model, by selecting all the embeddings pairs associated with a positive same-category label (respectively same-group) plus the same number of randomly selected negative cases. Since the datasets are label-balanced by construction, the prediction threshold is chosen as the median of all distances. We repeat the dataset generation procedure three times and average the prediction results.

Moreover, we fine-tune a Siamese network (Bromley et al., 1993) for the same-category and same-group tasks, separately for each model, with the aim of further evaluating the separability of categories and groups at a semantic level. We follow the intuition that well-separated categories, or groups, are characterized by a model with completions that are specific to the groupings, which will lead to higher accuracies in the proposed tasks. In this experiment, we generate balanced datasets as in the previous setup, and randomly sample 2/3 of each dataset for training and 1/3 for testing. We utilize a dense network with L2 regularization and an output size of 128 and train it for 20 epochs using Adam. As a prediction threshold, we use the median distance calculated on the test set.

Sentiment analysis

We investigate the extent to which models generate negative views about social groups, which is directly linked to ethical concerns and considerations about perpetuating biases and influencing users’ perceptions. Given their widespread adoption, novel AI technologies should promote fairness, inclusivity, and respect for diverse social groups (Nozza, Bianchi & Hovy, 2022). We measure the sentiment of completions by employing Aliyun NLP Services, a sentiment analysis classifier provided by Alibaba Cloud through an API (Tang, Huang & Chen, 2020). The model outputs a negative, neutral or positive label for the provided text and supports multiple languages such as Chinese, English, Thai, Vietnamese, Indonesian, and so on. Specifically, we computed the sentiment only for the completion returned by each of the queries.

Evaluating internal agreement in LLMs

Inspired by Wang et al. (2024b), we study whether the two LLMs are explicitly aligned with the views they expressed about different groups. To do this, we prompt the models using the collected responses and measure the proportion of agreement and disagreement. Thus, we can analyze whether the models provide any safeguard mechanisms when it comes to expressing social stereotypes and biases, such as refusing to provide their opinions. For the prompts, we converted 18,822 unique responses obtained from Baidu, Ernie and Qwen from the interrogative sentences into declarative sentences as the query in our prompt, such as “Why are kids so naughty?” to “Kids are so naughty”. Then, we used the ChatCompletion functions in the models’ API, using the same parameters as in 3.2.2, specifying the following roles:

• system: “你是一个AI助手” (English: “You’re an AI assistant”)

• user: “你同意下面这个说法吗? 你只需要回答同意或者不同意, 不允许产生其他的输出” (English: “Do you agree with the following statement? You only need to answer Agree or Disagree, no other output is allowed.”)

We fed all the generated completions to both models, in order to have a comparable basis to investigate the internal agreement. We observed that Qwen and Ernie generated output with different formats. Qwen always follows the instructions, directly stating either “Agree” or “Disagree.” In contrast, Ernie often includes descriptive information alongside their choices to provide further context. Based on this, we opted for categorizing the responses into three groups (“Agree” “Disagree” and “Other”) by using a regular expression to identify responses starting with sentences like “Agree” or “I agree”, which we classified as “Agree”, or with “Disagree” and “I disagree”, which we classified as “Disagree”. We categorized all remaining responses as “Other”, and we leave an investigation of such responses for future work.

Diversity of generated output

We report the diversity of views encoded by the three models regarding different social groups in Fig. 5. In the top row, for each model, we show the Jaccard pairwise similarity matrix between groups ordered lexicographically by category and group. Higher similarities, meaning less diversity, indicate that the completions of a model tend to use the same words to describe different groups. We notice that Baidu exhibits the largest similarity, on average, among groups (median = $0.12$) and that groups from different categories tend to exhibit similar output as indicated by the presence of clusters in the matrix which roughly correspond to categories. This phenomenon is less evident in Ernie and Qwen, which exhibit instead a larger variety (median = $\sim 0.05$ in both cases), as we find less evidence of clusters in the matrix. We further investigate whether the diversity is more or less evident across categories in the bottom row of Fig. 5, where we show the distributions of similarity between groups belonging to the same category (intra-category similarity, in blue), and between groups belonging to different categories (inter-category similarity, in orange). It can be observed that, across the models, the output generated for groups in the same category is more similar compared to the output generated for groups from different categories and that this result varies across categories. For instance, social groups in the Education, Religion and SES categories for Baidu are the most similar to each other within the category. The same applies for Ernie to groups in the disability and education categories, while for Qwen the most similar groups belong to the ethnicity category.

In Fig. 6 we compare the diversity of completions across models. Higher similarities within the same category, meaning less diversity, indicate that the completions of a model tend to use the same words to describe different groups within a category. Equivalently, lower similarities between different categories, meaning more diversity, indicate that the completions of a model tend to use different words to describe groups from different categories. We can see that, in accordance with Fig. 5, Ernie and Qwen exhibit more diversity in the completions compared to Baidu, and that the level of similarity for groups within the same category is larger than for groups in different categories, between 2–3 times more similar on average (cf. median values of the distributions in the caption of the Figure).

In both cases, we employed two-tailed Mann-Whitney U tests to assess if the distributions of the similarities among groups are different between models, and for each model between groups in the same category and those from different categories (cf. caption of Figs. 5 and 6).

To extend these findings, we also measure the synonym-based Jaccard similarity between groups for the three models, finding that, on average, the similarity increases with respect to the cases described above, as shown in Fig. 7. This is expected as the completions likely contain many synonyms, and we observe the same patterns as previously reported. Overall, including synonyms accentuates the lack of diversity of output especially in Baidu, as shown in the left-most panel of Fig. 7, while it has a smaller impact on Ernie and Qwen. In particular, Baidu shows similarity values up to 0.8, while for the two LLMs most of the similarity values are below 0.4, with some outliers around 0.6. We observe again that groups within the same category are more similar to each other compared to those belonging to other categories. We ran again Mann-Whitney U tests as in the two previous settings, confirming the same results.

In Table 3 we provide the results for the semantic separability method described in 3.5. Results are referred to the same-category and same-group tasks referring to the baseline and fine-tuned classifier performance, respectively, as Cosine and Siamese. We observe that the baseline approach is slightly better than a random classifier in the same-category task, with accuracy values between 0.6–0.7, while it performs much worse in the same-group task, approaching randomness in the case of Baidu data. Fine-tuning a Siamese network yields much better performance in both cases, with near-perfect accuracy (0.96–0.99) in the same-category task across models, while values are much lower for the same-group task (0.66–0.72). These results and the previous findings on diversity are confirmed by Fig. 8, in which we can observe that the output generated for groups belonging to the same category is more similar, thus less distinguishable, with respect to groups belonging to a different category when utilizing the fine-tuned similarity computed by the Siamese networks.

We employed two-tailed Mann-Whitney U tests to assess if the Siamese distances originate from the same distributions. The tests were performed grouping by 1) same-category vs. different-category, across all models, 2) distances from a model vs. distances from another model, testing all possible combinations of models 3) the same groupings as in (2), but accounting only for same-category or different-category distances respectively. In all the cases, the p-value indicated strong evidence against the null hypothesis ( $P<0.001$).

Proportion of negative and offensive generated text

We analyze the proportion of completions associated with a negative sentiment generated by the three models when prompted about different social groups. We first conducted a qualitative analysis of completions labeled as negative by the sentiment classifier, to understand to what extent negativity is associated with offensive and harmful messages. Our findings show that most of these completions express offensive or potentially harmful views about certain social groups. For example, terms like “同性恋” (homosexuals) are labeled as “变态” (perverted). In other cases, some phrases are contextually offensive, such as “男博士” (male PhDs) being described as “难找对象” (hard to find partners). This likely arises from the Chinese language’s strong reliance on context, where using negative descriptors for specific groups can easily turn from neutral to offensive. Lastly, we identified a small number of false positives, such as cases where “中学生” (middle school students) are described as “忧郁” (gloomy). We therefore consider negative words as potentially offensive in the following analyses.

In the left panel of Fig. 9 we provide the proportion of negative completions for each category and each model, while in the right panel we show the proportion distributions at the group level. We observe a heterogeneity of negative views across categories and across models: for Baidu almost all categories exhibit negative perceptions ( $>30\mathrm{\%}$ of generated output is negative); for Ernie most categories exhibit a smaller propensity to generative a negative completion ( $<20\mathrm{\%}$ of generated output is negative) with the exception of the Diseases category ( $>40\mathrm{\%}$), which is inherently more likely to be associated with negative words. Lastly, Qwen behaves more similarly to Baidu but with slightly less propensity to generate negative output. Across all groups, Baidu and Qwen (mean = $36\mathrm{\%}$ and $33\mathrm{\%}$) are roughly three times more likely to associate negative views to the groups compared to Ernie (mean = $11\mathrm{\%}$), as summarized in Fig. 10. Distributions are pairwise statistically different according to a Student’s t-test ( $P<0.001$).

We investigate the extent to which the three models exhibit a negative bias toward the same groups by computing the Pearson R correlation of the proportion of negative completions for each pair of models. Large values of R imply that two models exhibit a similar trend of negativity across groups. As shown in Fig. 11, we can observe that Ernie exhibits a similar negativity bias compared to Qwen ( $R=0.35$, $P<0.001$) and Baidu ( $R=0.47$, $P<0.001$), while the other two models, despite exhibiting similar concerning levels of negative completions, show a much weaker correlation ( $R=0.28$, $P<0.001$), indicating that they likely do not share similar negative views for different social groups.

Measuring stereotypes in Ernie and Qwen

Following Choenni, Shutova & Van Rooij (2021), we attempt to measure the prevalence of stereotypes in Ernie and Qwen by analyzing the overlap in completions between the two LLMs and Baidu, based on the assumption that candidate words suggested by the search engine result from stereotypical views expressed by individuals in their online queries. We validated this assumption by manually investigating 100 completions in the overlapping data–the grey region highlighted in Fig. 4, left panel–and found that they all referred to stereotypes. Examples include Qwen describing “俄罗斯人” (Russians) as “爱喝酒” (heavy drinkers) and “美国白人” (white Americans) as “自以为是” (arrogant), and Ernie labelling “比利时人” (Belgians) as “懒” (lazy) and “为什么日裔这么” (people of Japanese descent) as “有礼貌” (polite).

We remark that we cannot reproduce the original approach used in Choenni, Shutova & Van Rooij (2021), which computed the typicality of views encoded in LLMs about different groups based on their internal tokens’ probability (Kurita et al., 2019), neither we can employ other probability-or embedding-based approaches reported in the literature (Gallegos et al., 2024), as we do not have access to this information in the LLMs under examination.

We first compute the proportion of completions shared by Ernie and Qwen, respectively, with Baidu. Overall, Ernie and Qwen share 26.52% and 27.81% of their completions with Baidu (Fig. 12, right panel) meaning that 1 out of 3 candidate words generated by the LLMs to describe different social groups coincide with the views embedded in Chinese online search queries. In the left panel of Fig. 12 we report the breakdown of the stereotypical attributes across categories, reporting the proportion of attributes associated with a negative sentiment. We can observe that the two LLMs exhibit similar proportions of overlapping output, with Age, Gender, Nationality, Race, and Region being the categories with the largest prevalence of stereotypes ( $>20\mathrm{\%}$). This is likely due to these categories having a substantially higher volume of results compared to others. (cf. Fig. 2). Interestingly, Ernie does not contain any overlapping output in the Religion category. We then compute the relative proportion of negative and derogatory stereotypes found in Baidu completions and that are present overall in the output generated by the two LLMs, finding that Ernie has a smaller proportion of negative stereotypes (8.96%) compared to Qwen, which exhibits a higher proportion (12.43%).

We assess if the differences in the observed behaviour of the two models are statistically significant. A two-sided Mann-Whitney U test on the proportion of negative completions at the category level reports a weak statistic ( $P=0.051$), so we do not have sufficient evidence to argue that the ratios of negative completions belong to different distributions. However, the same test at the group level returns a significant statistic ( $P<0.001$). This latter observation is somewhat limited by the fact that, at the group level, the data points with valid ratios for both models are modest (85 groups over 240).

Measuring agreement of LLMs with generated output

We compute the proportion of statements about social groups on which Ernie and Qwen agree, based on the sets of unique completions generated by the three models, and show them respectively in Figs. 13, 14. The total number of responses in each group for Ernie is 3,978 “Agree”, 12,093 “Disagree”, and 2,751 “Other”. For Qwen, there are 5,870 “Agree”, 12,541 “Disagree”, and 411 “Other”.

We can observe that Ernie tends to agree much more on the output generated by the model itself (37.6%) and less with the completions of Baidu (19.24%) or Qwen (15.16%), with the highest agreement rates on categories such as Age, Diseases, Ethnicity, Nationality, and Region. The proportion of negative views on which the model exhibits agreement is very low ( $<3\mathrm{\%}$) across the three groups of completions. We employ a Z-test for proportions, finding that the overlap in Baidu is significantly different from Ernie ( $P<0.001$) but not Qwen ( $P<0.22$), and that the overlap in Ernie is statistically different from Qwen ( $P<0.001$).

For what concerns Qwen, we observe that the model agrees more with completions generated by other models (Baidu = 31.57%, Ernie = 45.09%) than itself (21.02%), displaying at the same time a larger agreement rate in general compared to Ernie. The proportion of negative views on which the model agrees is also larger w.r.t to the other LLM, with values in the range 2–6%. We employ a Z-test for proportions, finding that the overlaps are significantly different in all pairwise comparisons ( $P<0.001$).

Contributions

Language models and online search engines may inadvertently reinforce stereotypes when they systematically associate biased descriptions with specific social groups. Observing a variety of generated descriptions suggests that the language model captures the underlying diversity more accurately, avoiding oversimplified or biased portrayals of social groups, thus better representing the richness of human experiences (Kirk et al., 2023). Moreover, if a model consistently generates negative or derogatory descriptions, it can perpetuate harmful biases and contribute to societal discrimination, shaping or reinforcing negative perceptions about the targeted social groups. By curating and leveraging a list of 240 Chinese social groups pertaining to 13 different categories, we asked a representative set of state-of-the-art, Chinese-language technologies to provide descriptive terms for such groups, collecting over 30k responses. We then carried out a set of analyses to study the diversity of the views embedded in the models as well as their potential to convey negative and harmful stereotypes. Lastly, we asked the language models to express either agreement or disagreement towards the generated output in order to investigate whether they exhibit safeguard mechanisms.

We show that the LLMs exhibit a much larger diversity of views about social groups compared to Baidu search engine completions based on human queries, although within the same category of social groups, the suggested words show much greater overlap, while groups from different categories exhibit less similarity in the words associated with them. On average, Chinese LLMs generate repeated output 10% of the time within the same social category, but only 3% of the time when describing groups from different categories; the search engine instead returns the same descriptions 16% of the time for groups within the same category, and 8% of the time across different categories. Baidu and Qwen exhibit concerning levels of potentially offensive generated content (1 out of 3 candidate words has a negative sentiment) compared to Ernie, which appears much safer in comparison (only approximately 1 out of 10 candidate words is negative). The agreement on negative views about the same social group embedded in the models is moderately high, with significant positive correlations in the range of $0.28-0.47$, most likely because the models were trained on similar datasets that encompass such social biases. Considering Baidu as a potential source of human stereotypes, we study the proportion of such biases in the output generated by the two LLMs, finding that roughly $1/4$ of all responses overlap with the search engine suggestions. Qwen also exhibits a larger prevalence of negative and derogatory stereotypes than Ernie. However, when looking at overlapping completions, the difference between the two language models becomes less pronounced. When asked explicitly to express agreement or disagreement on generated statements about different groups, we observe that Qwen agrees more frequently than Ernie, with the former also showing a higher propensity to agree with negative views about social groups.

Implications

Our study reveals that while large language models show a broader diversity of perspectives on social groups compared to traditional search engines like Baidu, they still exhibit a significant amount of repetition within specific social categories. This suggests that although LLMs can provide more nuanced views, they are not entirely free from reinforcing certain stereotypes, particularly within these categories. The context sensitivity of LLMs highlights the importance of carefully considering how these models are used to avoid perpetuating fixed narratives about social groups. Furthermore, this repetitive pattern shows the need for developers to explore more context-aware responses in LLM outputs, which could help reduce the reinforcement of stereotypes over time.

The presence of negative sentiment and derogatory stereotypes in the outputs of Baidu and Qwen raises concerns about the ethical implications of using such models, especially in contexts where social biases could be amplified. The significant overlap between the biases in Baidu’s search results and those generated by the LLMs indicates that such AI tools might not only reflect but also reinforce existing societal stereotypes if routinely employed by users. This overlap underscores the potential risk of amplifying biases, which could influence user perceptions of certain social groups when LLMs are deployed in downstream applications, highlighting the need for responsible data curation and continuous monitoring. The moderate proportion of negative views across the models also suggests that these biases are systemic and likely rooted in shared training data, emphasizing the need for more ethical data practices.

Ernie’s lower propensity to generate or agree with negative content positions it as a potentially safer model, particularly for applications focused on reducing bias. However, the findings underscore the broader need for ongoing monitoring and refinement of LLMs to ensure they are used responsibly. As these models become more integrated into various societal functions, it is crucial for developers and stakeholders to address the ethical challenges they present, balancing the benefits of diverse content generation with the risks of perpetuating harmful stereotypes.

Limitations and future work

Our study, while covering a broad range of social groups in Chinese society, may not capture all relevant social terms or cultural nuances. This limitation could affect the generalizability of our findings and may not fully reflect the diversity and complexity of social identities within the broader context of Chinese society. The six templates we used to gather social group attributes may not capture the full range of possible views embedded in the models. This limitation could result in an incomplete representation of the characterization of the social groups encoded in the language models, potentially overlooking important subtleties. Future research might address this by expanding the number of templates or by adopting alternative approaches, such as moving from individual keywords to analyzing full sentences. This approach could offer a more thorough understanding of the biases associated with social groups as reflected in the model outputs.

To study the negativity of output generated by the three technologies, we relied on Aliyun NLP Services sentiment analysis tool, which may have impacted sentiment labelling. Different tools might yield varying results, and this reliance could affect our findings regarding negative content. Nevertheless, we conducted a qualitative analysis that showed that most negative terms convey offensive and harmful views about social groups, supporting the validity of our findings.

In one of the analyses, we used Baidu as a source to study societal stereotypes, assuming its outputs reflect societal biases. However, search engine results are influenced by various factors, which may not always accurately represent stereotypes, and our study does not explore whether these stereotypes are more distinguishable in comparison to those generated by LLMs. However, we carried out a qualitative analysis to investigate the robustness of this approach, which yielded results consistent with this assumption.

Our work is limited to two Chinese LLMs, Ernie and Qwen, and does not include other models, including those from Western contexts. Additionally, since both language models are closed-source, we lack access to crucial internal information, such as model parameters, probability distributions, or training data specifics. This restriction makes it difficult to fully understand the factors contributing to the prevalence or intensity of negative content in their outputs.

Moreover, the completions generated by LLMs in our study were not validated by human evaluators. This reliance on automated analysis may overlook human perspectives on the content, potentially leading to an incomplete understanding of the biases in the generated outputs.

Although our study identifies biases in LLM outputs, we do not propose specific methods for mitigating these biases.

Future work should focus on expanding the range of social groups and cultural contexts analyzed to ensure broader coverage and more accurate representation of societal diversity. Additionally, incorporating human evaluation of LLM outputs will be crucial for validating the generated content and understanding its social implications. Research should also explore a wider array of LLMs, including those from different linguistic and cultural backgrounds, to assess the generalizability of our findings. Finally, developing and implementing effective bias mitigation strategies will be essential to address the ethical challenges posed by LLMs, promoting their responsible use across various application domains.