Structural-topic aware deep neural networks for information cascade prediction

Preliminaries

Definition 1. Information Cascades. Suppose we have M messages, denoted by M = {mⁱ}(1 ⩽ i ⩽ M). For each message mⁱ, it is denoted by a cascade $C^i=\left\{\left(u_j^i,v_j^i,t_j^i\right)\right\}$ to record the diffusion process of it, where the tuple $\left(u_j^i,v_j^i,t_j^i\right)$ corresponds to the jth retweet, meaning that user $v_j^i$ retweets message mⁱ from user $u_j^i$ ,and the time elapsed between the original post and the jth retweet is $t_j^i$ . The popularity $R_t^i$ of message mⁱ up to time t is defined as the number of its retweets, i.e., $\left|\left\{\left(u_j^i,v_j^i,t_j^i\right)|t_j^i⩽t\right\}\right|$.

The representation of the topological structure of a cascade within a diffusion network is referred to as a cascade graph. This cascade graph can correspond to one or more pieces of information, necessitating manual extraction. For instance, an information cascade sample may correspond to a topological graph representation. Additionally, datasets may directly provide the topological structure of the cascade graph. This type of topological graph is typically considered a global network, encompassing the topological structural relationships of all information cascade sequences in the entire dataset.

Definition 2. Incremental Popularity Prediction. Given the forwarding path {(u_j, v_j, t_j)|t_j ⩽ T} of cascade C within the period [0, T) (commonly referred to as the observation window), the corresponding cascade graph is denoted as G^T = (V^T, E^T). The task of predicting cascade increment popularity involves a regression problem, aiming to predict the increment in the number of engaged nodes in cascade C after a time interval Δt, denoted as $\Delta s_C=\left|V_C^{T+\Delta t}\right|-\left|V_C^T\right|$. A specific instance of this prediction task is the final popularity prediction, which anticipates the increment $\Delta s_C^{\infty }=\left|V_C^{\infty }\right|-\left|V_C^T\right|$ between the number of observed nodes and the number of final nodes in the cascade.

Extracting structural topic of nodes

In the realm of natural language processing, topic models play a crucial role in elucidating the distributional distinctions embedded in higher-order structural patterns. This study undertakes a theoretical analysis to scrutinize the underlying principles of acquiring substructure topic distributions within graph networks. Subsequently, an adaptive graph neural network is introduced to adeptly leverage such structural information. Please refer to Fig. 2 for an intuitive illustration. In particular, we extract structural patterns through anonymous walks (Micali & Zhu, 2016). Anonymous random walks are sampled for each node to characterize the local structures of a node. Anonymous walks involve obscuring the true identity of nodes during a random walk, retaining only the serial number ID as an identifier to document the transitional rules within the walk. For each node, a set of random walk sequences with a predefined length is sampled. Subsequently, their potential distributions of anonymous walk experiences, along with the overall average experience distribution across the entire graph, are calculated to construct the authentic distribution.

We argue that anonymous walk does not share the identity space of nodes, that is, the id of nodes in each walk will be counted from scratch, so that the id of nodes in multiple walks will be repeated. The sequence generated by a random walk, as depicted in the figure, is transformed into the sequence generated by an anonymous walk. For instance, $\left(0,15,11,15,1,15\right)$ becomes $\left(0,1,2,1,3,1\right)$ in Fig. 2. Long et al. (2020) introduced an algorithm for selecting an anchor structure (‘Anchor’), aiming to filter representative structural features of the network, thereby reducing representation complexity and mitigating noise interference. We refer to their practice in the follow-up processing.

Given a graph G = (V, E), by anonymous walk, two matrices can be learned to depict the structural information of the nodes in graph. One is node-topic matrix N ∈ ℝ^|V|×Q, where each row N_i corresponds to a distribution and $N_i^q$ denotes the probability of node v_i belonging to the q-th structural topic. The second is walk-topic matrix M ∈ ℝ^Q×|A_l|, where each row M_q represents the topic distribution over walks in A_l and $M_q^a$ denotes the probability of walk a ∈ A_l belonging to the q-th structural topic. Here, A_l signifies a set of possible anonymous walks with a length of l, and Q represents the number of desired structural topics.represents the desired number of structural topics. Additionally, we define the set of anonymous walks starting from v_i as B_i. According to the definition of Graph Anchor LDA (Long et al., 2020), we need to get the node-topic matrix N, meaning of getting the distribution matrix of each node topic. We first identify the ‘anchor’ and then proceed with topic modeling. Specifically, we set the walk-walk co-occurrence matrix W ∈ ℝ^|A_l|×|A_l|, with $W_{i,j}={\sum }_{v_q\in V}\mathbb{I}\left(w_i\in B_q,w_j\in B_q\right)$, and utilize non-negative matrix factorization (NMF) to extract anchors (1)${\displaystyle H,Z=argmin\parallel W-HZ{\parallel }_F^2\text{s.t.}H,Z^T\in {\mathbb{R}}^{\left|A_l\right|\times \alpha },H,Z\ge 0.}$

We iteratively update H, Z until convergence, and then identify the anchors using ${\mathbf{A}}_q=argmax\left(Z_q\right),$q =1 , …α, where A is the set of indices for anchors, and Z_q is the q-th row of Z. Based on the selected anchors, we can learn the walk-topic distribution M. In addition, we define node-walk matrix as S ∈ ℝ^|V|×|A_l| with $S_i^a$ denoting the occurrences of a in B_i. We finally get the node-topic distribution N through N = SM^†, where M^† denotes pseudo-inverse. Table 1 summarizes the symbols and their corresponding meanings of our model.

User embedding

Each user in the generated retweet path is denoted as a one-hot vector, h ∈ ℝ^|V|, where |V| is the total number of users. All users share an embedding matrix N ∈ ℝ^|V|×Q, where Q is an adjustable dimension of embedding, N is equal the node-topic matric. This user embedding matrix converts each user into its representation vector: (2)${\displaystyle x=N^{T}h,}$ where x ∈ ℚ^K and N^T is the transpose of N. It is worth noting that the user embedding matrix N is learned from the overall structure of the graph, which can represent the whole information of the graph to some extent. Therefore, the learned user embeddings are optimized for the initial supervised framework.

Topic path coding

The influence among topics (influence transfer) and the significance of topics in topic path structure can be modeled through the GRU in recurrent neural networks (Mikolov et al., 2010). The influence transfer from one prominent subject to another influences the prominence of the latter. The importance of topics in the topic path structure is determined by their frequent occurrence across multiple topic paths. Each theme path has the most GRU output pooled layer, which achieves Hawkes’ self-exciting mechanism by accumulating various effects.

When GRU is used to encode each topic $e_j^i\left(1⩽j⩽Q\right)$ of message mⁱ, where $e_j^i$ is the element of matrix x, the d-th hidden state $h_d=GRU\left(e_j^i,h_{d-1}\right)$ in GRU, where h_d ∈ ℝ^H′ is the output, and the topic represents the vector when $e_j^i\in {\mathbb{R}}^Q$ is input. h_d−1 ∈ ℝ^H′ is the previous hidden state, Q is the dimension of topic embedding, and H′ is the dimension of hidden state.The GRU model is shown below in Fig. 3. Firstly, the reset gate r_d ∈ ℝ^H′ is computed by (3)${\displaystyle r_d=\sigma \left(W^r{e}_j^i+Y^r{h}_{d-1}+b^r\right),}$ where σ is the sigmoid activation function, W^r ∈ ℝ^H′×Q, Y^r ∈ ℝ^H′×H′ and b^r ∈ ℝ^H′ are GRU parameters learned during training.

Secondly, the update gate z_d ∈ ℝ^H′ is computed by (4)${\displaystyle z_d=\sigma \left(W^z{e}_j^i+Y^z{h}_{d-1}+b^z\right),}$ where W^z ∈ ℝ^H′×Q, Y^z ∈ ℝ^H′×H′ and b^z ∈ ℝ^H′.

Then, the actual activation of hidden state h_d is computed by (5)${\displaystyle h_d=z_d\odot h_{d-1}+\left(1-z_d\right)\odot {\tilde{h}}_d,}$ where (6)${\displaystyle {\tilde{h}}_d=tanh\left(W^h{e}_j^i+r_d\odot \left(Y^d{h}_{d-1}\right)+b^d\right).}$ ⊙ represents element-wise product, W^d ∈ ℝ^H′×Q, Y^d ∈ ℝ^H′×H′ and b^d ∈ ℝ^H′.

Modeling the time decay effect

Given the temporal decay of retweet effects, we incorporate the time decay effect through a non-parametric approach. Let us consider the propagation of all messages within a time interval T, and assume the unknown practical delay effect is a continuously changing function within the range [0, T). We partition the time length T into P intersecting intervals {[t₀ = 0, t₁), [t₁, t₂), …, [t_P−1, t_P = T)} to estimate this time delay function and derive the corresponding discrete time delay variable λ_p. Assuming $t_j^i$ represents the time elapsed from the original post to the jth retweet of message mⁱ, and $g\left(T-t_j^i\right)$ denotes the corresponding time interval for the time decay effect of jth retweet, then the mapping function g from continuous time to discontinuous time is defined as: (7)${\displaystyle g\left(T-t_j^i\right)=p,if{t}_{p-1}⩽T-t_j^i<t_p.}$

Based on the time decay effect, the cascade topic $c_e^i$ of mⁱ message can be denoted as: (8)${\displaystyle c^i=\sum _{j=1}^{G_T^i}{\lambda }_{g\left(T-t_j^i\right)}{h}_j^i}$ where cⁱ ∈ ℝ^H′ represents the final representation of cascade C, which is assembled by the sum pooling mechanism, and for each retweet path $e_j^i$, we use the last hidden states as the representation of the entire diffusion path, denoted as $h_j^i$.

Popularity prediction

Finally, the output from the sum pooling layer (cⁱ) is conveyed as input to the fully connected layer, known as the multi-layer perceptron (MLP). The MLP constitutes a feedforward artificial neural network model. We employ the representation vector from the sum pooling layer, as introduced in our model, directly as the input for the MLP, with the resulting output serving as the predicted cascade growth scale. This can validate the effectiveness of our sum pooling layer representation vector after topic learning, path encoding and time delay effect processing through the implementation of a straightforward model. The cascade representation vector derived from the pooling layer (cⁱ) can be directly fed into the MLP model, culminating in the retrieval of information cascade popularity prediction outcomes: (9)${\displaystyle {\hat{y}}_t^i=MLP\left(c^i\right).}$ The optimization objective function is defined as: (10)${\displaystyle Ob{j}_{\min }=\frac{1}{m}\sum _{i=1}^m{\left(log{\hat{y}}_t^i-log{y}_t^i\right)}^2,}$ where ${\hat{y}}_t^i$ is the predicted incremental popularity of cascade C, $y_t^i$ is the real incremental popularity and m is the total number of messages.

Data sets

We evaluate the performance of proposed model on two scenarios of information diffusion popularity prediction and compare with state-of-the-art methods to verify the effectiveness and generality of our model. The first scenario is to predict the future size of re-tweet cascades on Sina Weibo, and the second one is to forecast the citation count of papers from American Physical Society (APS). The weibo data is available at Github (https://github.com/CaoQi92/DeepHawkes) (Cao et al., 2017). The APS data comes from public datasets (https://journals.aps.org/datasets/inquiry). All requests will be sent to APS staff for review, and then researchers will receive a copy of this request via email. The dataset we used is available at Zenodo (https://zenodo.org/badge/latestdoi/670429356).

In the first scenario, the dataset is sourced from Sina Weibo, a social media platform based on user relationships and one of the most popular microblogging platforms in China. To ensure data fitting, we reorganize the Weibo dataset provided in Cao et al. (2017), which captures all original tweets generated on June 1st, 2016, and tracks all retweets of each message within the subsequent 24 h. In Fig. 4A, the distribution of cascade popularity, representing the number of retweets for each message, follows a power-law distribution. Notably, in contrast to the experimental setup in DeepHawkes (Cao et al., 2017), the observation time window in this study is limited to only 1 h. After data arrangement and selection, we obtain 42,183 cascade graphs, comprising 1,390,020 nodes, each corresponding to a unique Weibo user. Subsequently, a social network graph with information on all node connections is constructed based on the selected graphs. Next, the dataset’s 29,529 cascade graphs are allocated to the training set, 6,328 to the validation set, and 6,327 to the test set.

The second dataset is derived from the American Physical Society (APS), one of the most prestigious professional physics societies globally, with a history spanning 122 years since its establishment. This dataset encompasses all papers published by the 11 APS journals from 1893 to 2009, along with the citations among these papers. In this context, the number of days elapsed since the publication of the cited paper is recorded. The dataset comprises 24,338 cascade graphs, containing 122,975 nodes. A cascade consists of all the citations to a paper, and the number of citations reflects the cascade’s popularity. Figure 4B illustrates the distribution of cascade popularity. Subsequently, we partition the dataset into training, validation, and test sets, comprising 17,037, 3,651, and 3,650 cascades, respectively.

In summary, the Weibo dataset comprises 42,183 information cascades, while the APS dataset contains 24,338 information cascades. In both scenarios, nodes represent users or papers, and each sample in the dataset constitutes an information cascade graph documenting the diffusion of a target message or paper. Additionally, we allocate 70% of the cascades to the training set, the middle 15% to the validation set, and the remaining 15% to the test set.

Evaluation metric

In order to evaluate the accuracy of predictions, following the practice of related models, we use the mean square log-transformed error (MSLE) and the median square log-transformed error (mSLE). Denote M the total number of messages, and SLE_i the square log-transformed error for a given message m_i, MSLE is employed to measure the error between predicted values and actual values, and it is defined as: (11)${\displaystyle MSLE=\frac{1}{M}\sum _{i=1}^{M}SL{E}_i}$ The mSLE is capable of effectively mitigating the impact of outliers, which is defined as: (12)${\displaystyle mSLE=median\left(MSL{E}_i\right)}$ where $SL{E}_i={\left(log{\hat{y}}_i-log{y}_i\right)}^2$, ${\hat{y}}_i$ is the predicted increment of the popularity for message m_i and y_i is the actual increment of the popularity.

Baseline methods

In this section, we have chosen six state-of-the-art methods for comparative analysis with our proposed approach, namely Feature-linear/deep, DeepCas, DeepHawkes, CasCN, and CasTCN, respectively. It is worth noting that the models under consideration employ comparable or identical datasets as inputs for a fair comparison. Subsequently, we provide a detailed description of these baseline models.

Parameter setting

For the baselines above and our proposed structural-topic aware deep neural networks model (STDNN), in order to obtain the best results for each data validation set, we adjust the hyper-parameters. For instance, the L2-coefficient is chosen from $\left\{10^{-8},10^{-7},\dots ,0.01,0.1\right\}$ in feature-linear. For LIS, the parameter 231 setting consist with Wang et al. (2015). For LIS, the parameter setting consist with Wang et al. (2015). For the MLP, the dimensions of each layer are $\left\{512,256,128,64,32\right\}$. Sigmoid functions are employed as activation functions in the hidden layers, while the output layer utilizes [mention the specific activation function]. The learning rate is set at 0.01. For all the deep learning-based approaches, we follow the setting of DeepCas, where the embedding size of users is set to 50, and for DeepCas, DeepHawkes, and CasCN, the GRU in the hidden layer is configured with 32 units. The hidden dimensions of the two-layer fully connected layers in all MLP-based predictors are set to 32 and 16, respectively. The CasTCN, all parameters involved are consistent with those in the original paper. In deep neural networks, we set the time interval of non-parametric time decay effect to 10 min for Sina Weibo and 3 months for APS.

Overall performance

The prediction performance of our proposed STDNN and the state-of-the-art baselines on both the Weibo Dataset and the APS Dataset is demonstrated in Table 2. Intuitively, our proposed model demonstrates superior performance compared to all baselines in predicting information cascades for the two scenarios, as assessed by the MSLE and mSLE evaluation metrics. Furthermore, our model significantly outperforms feature-based and diffusion model-based approaches, e.g., Features-deep, LIS. It also outstrips the state-of-the-art deep learning approaches, e.g., DeepCas, CasCN and CasTCN. Now, in the following, we compare the differences and effectiveness of the MSLE-based metric among our proposed model and these baselines and analyze the reasons in detail.

It is observed that features-deep does not outperform features-linear on both the Weibo and APS datasets. This indicates that, when a set of appropriate features is provided, the linear method is not necessarily inferior to the deep learning approach. However, the error of features-linear results is significantly higher than the error of our proposed model predictions. This emphasizes our previous claim that feature-based approaches heavily rely on manually crafted features, rendering them challenging to generalize across diverse scenarios.

For the diffusion model-based methods, LIS did not achieve satisfactory results in predicting information cascades; it performed the poorest among all methods. The main reason is that diffusion model-based methods like LIS typically model the propagation process of information cascades but do not effectively predict the future popularity of information cascades. Therefore, LIS performed the worst on both datasets.

For the deep-learning approaches, STDNN has also performed better than state-of-the-art baseline methods. DeepCas relies solely on random walk strategies, failing to capture crucial information about the network structure, leading to subpar performance in cascade prediction tasks. On the other hand, DeepHawkes, encoding the dynamics of cascades using Hawkes processes, combines the advantages of generative processes and deep learning, resulting in better performance compared to DeepCas. However, DeepHawkes does not extract structural information from cascade networks, leading to inferior performance compared to the latest models like CasCN and CasTCN.

Our proposed STDNN outperforms all peers on all datasets and surpasses the state-of-the-art methods in information cascade prediction. Regarding the reasons, firstly, our model rigorously incorporates the topological structure of individual nodes within the network, facilitating a more precise and comprehensive modeling of the entire cascade network when contrasted with approaches reliant on subgraph modeling. Secondly, the utilization of Hawkes processes to encode the propagation of the entire cascade enhances the interpretability of the model. Our approach that not only combines the advantages of deep learning and generative methods but also takes into account the network topology of each cascade, results in highly satisfactory performance.

Ablation experiments

To better analyze the impact of different factors on cascade prediction in the STDNN model, we design several variants of the STDNN:

• STDNN-Linear: In STDNN-Linear, we refrain from optimizing the user embedding process. In this configuration, our model effectively degenerates to be consistent with the DeepHawkes model. We directly utilize the representation vectors learned from the data as user embeddings.

• STDNN-Node: In STDNN-Node, we do not take the original node’s thematic features as user embeddings. Instead, we employ the subgraph of node distributions as input, specifically using the node-walk matrix as user embeddings.

• STDNN-Path: In STDNN-Path, we focus solely on the influence of each forwarding event in the Hawkes process, rather than encoding the impact of the entire structural topic path through the GRU structure.

The specific performance of these variant models on the two datasets is summarized in Table 3 and illustrated in Fig. 5. Firstly, STDNN-Linear demonstrates a significant advantage over traditional feature-based methods and diffusion-based methods, as our model aligns with DeepHawkes in this configuration. Secondly, for STDNN-Node, using the walk-topic matrix as user embeddings yields better performance than STDNN-Linear, indicating the crucial role of network topology in the prediction process. However, its performance is inferior to our overall STDNN model, implying that considering the network topology of individual nodes yields better predictive performance than focusing solely on subgraph structures. Finally, in STDNN-Path, considering only the influence of single forwarding events significantly reduces the prediction accuracy, which indicates that future popularity is not only influenced by the current forwarding user but also by the entire forwarding path. This further emphasizes the necessity of the proposed approach.

Parameter sensitivity

We primarily conduct sensitivity analyses on two parameters in our model, specifically, the length of walks l and the number of topics Q. Following related work, we employ the MSLE as a metric to assess sensitivity to parameters on two datasets. For the length of walks l, as depicted in the Fig. 6, there is a noticeable “V”-shaped variation pattern in MSLE with the increase of l on both datasets. It’s crucial to avoid setting values that are excessively large or too small, as they can adversely impact the model’s performance. However, the fluctuation range is minimal, with a slight difference between the maximum and minimum values of MSLE, namely 2.557 vs 2.530 and 1.263 vs 1.250, respectively. Concerning the number of topics Q, as shown in Fig. 7, with the number of topics Q increases, the performance on the two datasets exhibits a slight fluctuation in MSLE, specifically ranging from 2.53 to 2.55 and from 1.25 to 1.27, respectively. These experimental results indicate that our model is not sensitive to the parameters of the length of walks l and the number of topics Q, making it relatively easy to implement in practice.