Fairness modeling for topics with different scales in short texts

Graph-based representation of text

In the graphical representation of text, the graph structure effectively captures the complex relationships within a document (Yao, Mao & Luo, 2019). The nodes and edges in the graph represent multi-level associations between words and documents. Depending on the specific task, the method of graph construction can be adapted. For example, Wu, Xu & Zheng (2025), Cheng et al. (2025) and Hua et al. (2024) explores these complex relationships by modelling documents as heterogeneous graphs consisting of word nodes and document nodes. The edge weights in these graphs are derived from two components: the pointwise mutual information (PMI) between words and the term frequency-inverse document frequency (TFIDF) between words and documents. By applying convolutional operations to the document graph, global word information can be effectively integrated, thereby improving model performance. Song et al. (2025) effectively captures the potential associations between documents and words by introducing edges between original short texts, edges between short texts and their associated long texts, and edges between semantically similar words, based on the document-word dichotomous graph framework.

Typically, word co-occurrence information is obtained by calculating the frequency with which words occur together. However, in short texts, the frequency of word occurrences is often low, making it difficult to obtain reliable co-occurrence statistics. PMI measures the true association between two words by comparing their joint probability with the product of their individual probabilities, thus highlighting significant semantic relationships between word pairs. We want to achieve fair modelling of data at different scales. PMI can be used to transform the text into a graph model, replacing the data dependencies typical of traditional topic models with word correlations. However, PMI is prone to extreme value problems (Bouma, 2009), which can undermine its stability. To address this, the model uses NPMI to compute word associations, thereby increasing the stability of the association measure.

Topic modeling for short texts

Short text topic modelling aims to automatically identify the underlying topics in short texts. While traditional topic modelling methods such as LDA perform well when applied to longer documents, they face significant challenges when applied to short texts. Some studies have used word embedding-based methods (Kaleem et al., 2024; Kinariwala & Deshmukh, 2023; Liu et al., 2022; Rashid et al., 2023; Reuter et al., 2024; Uddin et al., 2024), such as pre-trained models like bidirectional encoder representations from transformers (BERT) (Kenton & Toutanova, 2019), generative pretrained transformer (GPT) (Brown, 2020), which convert words and texts into vector representations. Some other studies introduce incremental models with different priors to capture richer semantic relations; Zhang et al. (2025) introduces an integrated Gaussian and logistic coding network augmentation network in the neural inference network part to improve the performance and interpretability of topic extraction; Koochemeshkian & Bouguila (2024) combines BERTopic derived embeddings with multi-granularity clustered topic modelling (MGCTM) and introduces a Generalised delicacy distribution and a Beta-Liouville distribution as priors to improve the expressiveness of the MGCTM method; Wang et al. (2024) proposes the prompted topic model (SET) based on the Wasserstein Autoencoder to capture the semantic knowledge of words; Qiu et al. (2024) the Prompted Topic Model (PTM) for topic modelling using cue learning, which bypasses the structural constraints of LDA and variational autoencoder (VAE), thus overcoming the shortcomings of traditional topic models. These models map words into continuous spaces, capturing richer semantic information.

Some studies address the issue of short text sparsity by introducing different models for detecting emerging topics in short texts. Shi, Du & Kou (2020) uses the “spike and slab” method to distinguish between emerging and ordinary topics and to track emerging events. Zhu et al. (2022) combines recurrent neural networks (RNNs) with spatio-temporal information to mitigate data sparsity. Asgari-Chenaghlu et al. (2021) uses a combination of transformers and graph mining techniques to improve topic detection in multimodal data. While these methods perform well on large datasets, they are highly dependent on data size when applied to smaller datasets. Limited data may prevent the model from fully capturing semantic relationships or detecting emerging topics, thus affecting detection performance. In this article, we replace the reliance on data size in traditional topic modelling with data correlation, allowing for more robust and fair topic modelling across datasets of different scale sizes.

Graph model construction

In order to account for differences in data scale across topics, reliance on data volume is replaced by data correlation. Specifically, NPMI is introduced to measure the mutual relationship between words. The formula is as follows:

(1) $$\begin{array}{rl}& NPMI=\frac{PMI(x,y)}{-\log p(x,y)}\\ & PMI=\log \frac{p(x,y)}{p(x)p(y)}\end{array}$$where $p(x,y)$ represents the probability of co-occurrence of $x$ and $y$, and $p(x)$ represents the probability of occurrence of $x$.

The value of NPMI ranges from $-1$ to $1$. A value of $1$ indicates that the two words have perfect co-occurrence, i.e., they always appear together (e.g., “bat” and “ball” in a sports corpus, where they are inseparable). A value of $0$ implies statistical independence, i.e., their co-occurrence is no more frequent than random chance. A value of $-1$ represents perfect mutual exclusion, where the words never occur together (e.g., “hot” and “cold” in a corpus where it is logically impossible for them to occur together).

In practical applications (Salle & Villavicencio, 2023; Yao, Mao & Luo, 2019), for ease of analysis, the value of NPMI usually takes the range of $0$ to $1$, with values closer to 1 indicating a stronger correlation between word pairs and values closer to 0 indicating a weaker or negligible correlation. This helps to mitigate the effects of data volume and shifts the focus to the relationships between word pairs. Building on this, we model the collection of documents as a graph.

Specifically, let $G=(V,E)$ be an undirected graph, where V is the set of all words in the document collection: $V=w_1,w_2,\dots ,w_n$. The edges E represent the relationships between words, with the edge weight $O_{i,j}=NPMI(w_i,w_j)$.

Using the NPMI metric, we can measure the mutual information between word pairs. If the data set for certain topics is smaller but more focused, the mutual information between word pairs will be higher, resulting in a higher NPMI value. Conversely, if noise words occur frequently in different topic clusters, the NPMI values between these word pairs will be lower. When several words have strong correlations, there will be several edges between them in the graph, forming a dense subgraph that reflects their close association as a topic. In this case, the document collection graph can be seen as an aggregation of strong semantic structures. Therefore, the task can be advanced by searching for complete subgraphs within the graph. A complete graph is one in which every node is directly connected to every other node. This high connectivity indicates a very close relationship between the nodes, which is consistent with the expectation of topic aggregation.

We use the Bron-Kerbosch algorithm (Bron & Kerbosch, 1973) to identify complete subgraphs, a classical graph algorithm designed to find cliques in an undirected graph. This algorithm efficiently finds all maximal cliques through a recursive process and is well suited to handle undirected graphs of different densities, whether dense or sparse. We input the mutual information constructed text graph as raw data to the algorithm to extract all maximal cliques. These cliques represent textual data sets with strong semantic associations at the semantic level.

Topic modeling

We propose a novel model, MixTM, which addresses the problem of data sparsity by considering a wider range of contextual information. This model is well suited to situations with limited data, as it captures subtle semantic relationships within the text and provides richer topic representations.

We construct documents as a set of mixed word pairs, assuming that the words in each pair belong to the same topic. For example, the short text “company department chip” can be represented as the word pair sets “company department”, “company chip”, “department chip”, “company department chip”. Let $\alpha$ and $\beta$ be the prior parameters for the topic distribution $\theta$ and the topic-word distribution $\phi$ respectively, $Dir$ means Dirichlet distribution, $Multi$ represents the multinomial distribution, the text generation process of the model is as follows:

• 1.

  For each topic z:

• Draw topic-word distribution $\phi \sim Dir(\beta )$

• 2.

  Draw topic distribution $\theta \sim Dir(\alpha )$

• 3.

  For each term in the set of word pairs B

• Draw $z\sim Multi(\theta )$

• Draw $term\sim Multi({\phi }_z)$

• $term=(w_i,w_j)orterm=(w_i,w_j,w_k)$

The graphical representation of the model is shown in Fig. 2. By constructing documents as a mixture of bi-grams and tri-grams, MixTM provides a richer contextual representation than the bi-gram model, significantly improving the model’s ability to distinguish between different topics. In addition, this combination captures more complex relationships between words, making the prediction of the next word more constrained and contextually informed. As a result, the topic modelling process becomes more accurate and robust, particularly in short or sparse texts where limited contextual information could hinder the discovery of meaningful topics.

In the document generation process described above, we need to infer the parameters: $\theta$, $\phi$, and the latent variable k. Given the simplicity and effectiveness of the Collapsed Gibbs Sampling method, we use it to estimate these parameters. The sampling formula for Gibbs sampling is as follows:

(2) $$p(k_b|K_{-b},B,\alpha ,\beta )\propto \frac{p(K,B|\alpha ,\beta )}{p(K_{-b},B_{-b}|\alpha ,\beta )}.$$

The formula represents the conditional probability distribution of the latent variable $k$ of the target word pair $b$, given the topics of all word pairs except the word pair $b$ and the hyperparameters $a$ and $\beta$ of all other word pairs. The joint probability of all word pairs is denoted by $p(K,B|\alpha ,\beta )$. The joint probability of all word pairs except the pair $b$ is represented by $p(K,B|\alpha ,\beta )$. The joint probability of all word pairs is calculated as follows:

(3) $$\begin{array}{rl}p(K,B|\alpha ,\beta )& =p(K|\alpha )p(B|K,\beta )\\ & =\prod _{d=1}^D\frac{\mathrm{\Delta }(n_d^K+\alpha )}{\mathrm{\Delta }(\alpha )}\prod _{k=1}^K\frac{\mathrm{\Delta }(n_k^B+\beta )}{\mathrm{\Delta }(\beta )}\end{array}.$$

In this context, $n_d^k$ represents the frequency of each topic k in document d, and $n_k^B$ represents the number of times a word pair belongs to topic k, $\beta$ denotes the $Beta$ function. The formula represents the joint generation probability of the topic structure and word distribution given the hyperparameters $\alpha$ and $\beta$. Combining the two Formulas (2) (3), the final sampling formula can be derived as:

For the pairs composed with two words ( $w_i$, $w_j$):

(4) $$\begin{array}{rl}p(k_b=k|K_{-b},B,\alpha ,\beta )\propto & \frac{(n_{k,-b}+\alpha )}{({\sum }_{k=1}^K{n}_{k,-b}+K\alpha )}\\ & \times \frac{(n_{k,-b}^{w_i}+\beta )(n_{k,-b}^{w_j}+\beta )}{({\sum }_{w=1}^W{n}_{k,-b}^w+W\beta )({\sum }_{w=1}^W{n}_{k,-b}^w+1+W\beta )}.\end{array}$$

For the pairs composed with three words ( $w_i$, $w_j$, $w_g$):

(5) $$\begin{array}{rl}p(k_b=k|K_{-b},B,\alpha ,\beta )\propto & \frac{(n_{k,-b}+\alpha )}{({\sum }_{k=1}^K{n}_{k,-b}+K\alpha )}\\ & \times \frac{(n_{k,-b}^{w_i}+\beta )(n_{k,-b}^{w_j}+\beta )(n_{k,-b}^{w_g}+\beta )}{({\sum }_{w=1}^W{n}_{k,-b}^w+W\beta )({\sum }_{w=1}^W{n}_{k,-b}^w+1+W\beta )({\sum }_{w=1}^W{n}_{k,-b}^w+2+W\beta )}\end{array}$$where $n_{k,-b}$ denotes the number of word pairs assigned to topic $k$ after excluding word pair $b$, $n_{k,-b}^{w_i}$ represents the number of occurrences of word $w_i$ in topic $k$.

The Gibbs sampling process commences with the random selection of an initial state and the subsequent allocation of initial values to each variable. Subsequently, the new values of the variables are recalculated using the sampling formulas. The process is iterated until a stable state is reached, at which point the topic distribution for each word is obtained. Formulas (6) and (7) are employed to calculate the parameters $\theta$ and $\phi$, while $n_k^w$ signifies the number of times the word is assigned to topic k.

(6) $${\phi }_{k,w}=\frac{n_k^w+\beta }{{\sum }_w{n}_k^w+W\beta }$$

(7) $${\theta }_k=\frac{n_k^w+\alpha }{{\sum }_w|B|+K\alpha }.$$

It is not possible for the model to obtain the topic distribution directly from the learning process. In order to infer topics, it is assumed that the topic distribution of each document is equivalent to the expectation of the topic proportions of word pairs generated from the document.

(8) $$p(k|d)={\sum }_{b}p(k|b)p(b|d)$$and $p(k|b)$ can be calculated by the Bayesian formula:

(9) $$\begin{array}{rl}& p(k|b)=\frac{p(k)p(w_i|k)p(w_j|k)}{{\sum }_{b}p(k)p(w_i|k)p(w_j|k)}or\\ & p(k|b)=\frac{p(k)p(w_i|k)p(w_j|k)p(w_g|k)}{{\sum }_{b}p(k)p(w_i|k)p(w_j|k)p(w_g|k)}\end{array}$$where $p(k)={\theta }_k,p(w_i|k)={\phi }_{k,w_i}$, $p(b|d)$ can be calculated using a simple uniform distribution:

(10) $$p(b|d)=\frac{n_d(b)}{{\sum }_b{n}_d(b)}$$

$n_d(b)$ denotes the frequency of word-pair b in document d.

Dataset

QA-data (Yan et al., 2013): This dataset is collected from the Baidu Zhidao website. THUCNews (http://thuctc.thunlp.org/): This is a news text classification dataset provided by the Tsinghua NLP group. 20NG (http://qwone.com/∼jason/20Newsgroups/): The 20 Newsgroups dataset is a standard dataset that includes 20 different topics, commonly used for topic modeling classification tasks.

In the preprocessing stage, for the Chinese dataset, we use the jieba tool to remove the stop words in the dataset, remove texts with length less than 2. For the English dataset, we use the nltk tool to tokenize the texts, convert characters to lower cases, remove texts with length less than 2 and filter out illegal characters. The dataset statistics are reported in Table 1.

Baselines

(1) BTM (Yan et al., 2013): A topic model specifically designed for short texts, which captures word co-occurrences in word pairs (biterms) to discover latent topics. (2) PYSTM (Niu, Zhang & Li, 2021): A topic model that uses the Pitman-Yor process for self-aggregation. (3) NQTM (Wu et al., 2020): A topic model that combines negative sampling and quantization techniques for short text topic modeling. (4) NSTM (Zhao et al., 2020): A neural topic model that learns document topics by minimizing the optimal transport (OT) distance between word distributions in documents. (5) ETM (Dieng, Ruiz & Blei, 2020): A topic model that combines traditional topic modeling with word embeddings. (6) BERTopic (Grootendorst, 2022): A topic model that generates document embeddings from pre-trained transducer-based language models, clusters these embeddings, and finally generates topic representations using a clas-based TF-IDF procedure. (7) TSCTM (Wu, Luu & Dong, 2022): a topic model that employ a new contrastive learning method with efficient and negative sampling strategies based on topic semantics.

Evaluation metrics

• • Topic coherence: This metric is used to measure the quality of a topic by assessing the semantic coherence of the N words with the highest percentage in the distribution of generated topic words. A higher coherence score is indicative of greater topic consistency. In the experiment, the most widely used metrics Cv (Röder, Both & Hinneburg, 2015) and NPMI (Aletras & Stevenson, 2013) are employed to compute the consistency of the first N words: (11) $$C_V(k)=\frac{1}{N}\sum _{i=1}^N{s}_{\cos }\left({\nu }_{\mathrm{N}\mathrm{P}\mathrm{M}\mathrm{I}}(w_i),{\nu }_{\mathrm{N}\mathrm{P}\mathrm{M}\mathrm{I}}(w_1{;}_N)\right)$$ (12) $${\nu }_{\mathrm{N}\mathrm{P}\mathrm{M}\mathrm{I}}(w_i)={\left\{\mathrm{N}\mathrm{P}\mathrm{M}\mathrm{I}(w_i,w_j)\right\}}_{j=1,2,\dots ,N}$$ (13) $${\nu }_{\mathrm{N}\mathrm{P}\mathrm{M}\mathrm{I}}(w_1{;}_N)={\left\{\sum _{i=1}^{N}NPMI(w_i,w_j)\right\}}_{j=1,2,\dots ,N}$$

• where $s_{cos}$ means the cosine similarity function. For the whole corpus, we use the average Cv $(\frac{1}{k}{\sum }_{k=1}^{K}Cv(k))$ and average NPMI $(\frac{1}{K}{\sum }_{k=1}^{K}NPMI)$ to compute the coherence.

• • Purity and NMI: In the experiment, the topic with the highest probability in the document distribution is designated as the predicted label for the document. The Purity and NMI metrics are utilised to assess the clustering efficacy of the model. A higher Purity and a larger NMI indicate better clustering performance of the model. (14) $$purity(\mathrm{\Omega },C)=\frac{1}{n}\sum _{i=1}^{K}max|w_i\cap c_j|$$ (15) $$NMI(\mathrm{\Omega },C)=\frac{{\sum }_{i,j}\frac{|w_i\cap c_j|}{n}log\frac{|w_i||c_j|}{n|w_i\cap c_j|}}{({\sum }_i\frac{|w_i|}{n}log\frac{|w_i|}{n}+{\sum }_j\frac{|w_j|}{n}log\frac{|w_j|}{n})/2}.$$

Evaluation of the MixTM in topic discovery

Qualitative metrics are used to measure the quality of topics generated by the model. The same number of topics is set for MixTM and the comparison methods. Cv, NPMI, Purity and NMI are calculated and analyzed to objectively assess and compare the semantic quality of different algorithms.

The consistency scores of all models are shown in Tables 2, 3, 4 and Figs. 3, 4, 5. For the two short text datasets, QA-data and THUCNews, the proposed model consistency demonstrates superior performance in comparison to alternative models. In relation to Cv scores, it is evident that the attainment of maximum scores is achieved in scenarios where the number of topics is minimal. However, as the number of keywords increases, a decline in Cv scores becomes observable. This decline signifies that the hybrid word pair approach introduces irrelevant words to a certain extent, thereby causing a disruption to topic consistency. Conversely, the proposed model consistently attains a higher NPMI score. The NPMI metric is a normalized measure that serves to attenuate the impact of extreme keywords on consistency results, thereby reducing the influence of low-frequency keywords and chance associations on these results. The findings suggest that the proposed model is capable of attaining enhanced and more consistent results. For the standard length text set designated 20NG, the proposed model demonstrates a high degree of consistency overall. Among the compared methods, the NQTM and NSTM methods have higher Cv scores but lower NPMI scores. This is probably due to the presence of incidentally co-occurring word pairs in the generated subject keywords, which leads to inflated Cv scores. The BERTopic method demonstrates superior performance on the 20NG dataset, likely attributable to the capacity of the pre-trained knowledge base to accurately identify the jargon and complexity of the 20NG dataset. The extensive text contains sufficient semantic information to facilitate the model’s acquisition of enhanced semantic embedding, thereby ensuring greater consistency in topic information.

From the clustering results (Figs. 6, 7, 8), it has been demonstrated that the proposed approach enhances the efficacy of text clustering in comparison with preceding models. The BTM and the model under consideration employ word pairs for text modelling, with the size of the contextual range of the word being a determining factor. It has been established that the larger the contextual range of a word, the more co-occurring patterns it contains. This has a beneficial effect on the identification of topics and the clustering of documents. Our model extends word pair construction from two to three words. It is evident that the utilisation of a shared word pair construction window in our model results in the acquisition of enhanced word co-occurrence information. This superiority in data processing capacity leads to the model’s superior performance in comparison to BTM in terms of clustering efficacy. However, it is noteworthy that the PYSTM and NQTM models demonstrate suboptimal performance when applied to the 20NG dataset. This may be due to the fact that the plain text dataset contains a lot of repetitive information, and overfitting can result in the model failing to capture the differences between similar document labels, which can lead to all documents being clustered into the same category. The ETM approach may perform weakly on text due to insufficient embedding quality.

Efficiency analysis

The computational cost of the 20NG and QA-data datasets has been measured. The 20NG dataset is composed of normal-length texts, whereas the QA-data set consists of short texts. It is important to note that all models are computed in the same hardware conditions. Each model was executed for 100 iterations, and the mean of each iteration was subsequently obtained. The results have been presented in tabular form in Table 5.

The model that has been demonstrated to demonstrate the highest computational efficiency is the BERTopic approach, in both plain text and short text. This is most likely due to the utilisation of pre-trained BERT in order to generate document embeddings, which consequently renders the subsequent clustering (UMAP+HDBSCAN) process computationally lightweight. The proposed model, due to the fact that it is reconstructing the text into a collection of mixed word pairs, takes approximately longer to train. When the length of each text is increased, the more word pairs are produced. It has been demonstrated that the mean elapsed time per iteration of the proposed model is greater for the 20NG dataset and less for training on QA-data.

Evaluation of small-scale data

In the experiment, 10 short text samples related to the topic of “Dongpo-Pork” have been added to both the QA-data and THUCNews datasets. The short-text data incorporated into the datasets includes characteristics of Dongpo-Pork, its historical origins, and everyday cooking methods. The models were trained with K = 50, 100, and 150 topics, respectively, to identify topics related to small-scale data and to display the top 15 words. The results obtained serve as evaluation metrics for the discovery of topics in small-scale data. Furthermore, the diversity metric was utilised to evaluate the quality of the generated topics.

The findings of the experiment conducted to ascertain the most effective methods of identifying topics from small-scale data are presented in Tables 6 and 7. It has been demonstrated that the quality of the semantic representation of generated topics is contingent upon the prevalence of relevant words and their elevated rankings. This suggests that the ability to identify small-scale topics is enhanced. The results indicated that NSTM and ETM were unable to identify keywords related to the “Dongpo-Pork” topic. In the NQTM and BTM models, related words appeared, but they were primarily noise words for the current topic. While the PYSTM model did identify some relevant topics, the generated topics were mixed. In contrast, the present method successfully identified relevant topic words such as “Dongpo-Pork,” “pork,” “officialdom,” “Chinese,” and “price,” and produced the most relevant words across different topics, with the highest rankings. The findings indicate that the proposed model exhibits superior performance in identifying topics from limited data sets when compared to existing methods.

A comparison of MixTM and MixTM-G reveals that incorporating graph modelling enables the model to identify potential small-scale topics, even with a limited number of topics, and helps it stabilise at an optimal state. This is likely due to the fact that graph modelling effectively captures the relationships between words, thereby rendering the model more flexible in its handling of small-scale topics. Representing words and their interconnections as a graph structure enables the model to detect potential correlations and, even in conditions where data is sparse, extract features of small-scale topics. The graph structure facilitates the model’s capacity to manage noise and uncertainty, thereby ensuring more consistent learning outcomes across varying numbers of topics. This enhanced stability contributes to the model’s reliability in practical applications and facilitates more accurate reflection of latent topics in text. In summary, the integration of graph modelling into MixTM-G enhances its capacity to identify small-scale topics, while concurrently ensuring the model’s stability and efficacy across diverse contexts.