Enhanced industrial text classification via hyper variational graph-guided global context integration

Node attention.

Given a specific node v_i, the HyperGraph layer will learn the representation of all edges that connected to the node, whereas not all edges connected to this node have the same weight. Thus the weights are assigned through the attention mechanism, which satisfies: (4)${\displaystyle f_j^l=\sigma \left(\sum _{v_k\in e_j}{\alpha }_{jk}{W}_1{h}_k^{l-1}\right)}$

where σ is a nonlinear activation function, usually refers to the sigmoid function, W₁ is a trainable matrix, α_jk refers to the coefficient of edge e_j at the node v_k, which is calculated as follows: (5)${\displaystyle {\alpha }_{jk}=\frac{exp\left(a_1^T{u}_k\right)}{\sum _{v_p\in e_j}exp\left(a_1^T{u}_p\right)}{u}_k=\text{LeakyReLU}\left(W_1{h}_k^{l-1}\right)}$

Edge attention.

For all edge representations $\left\{h_k^{l-1}\left|\right.\forall v_k\in e_j\right\}$, the attention mechanism is also used to learn the node representation of the next layer. It is formulated in the following function: (6)${\displaystyle h_i^l=\sigma \left(\sum _{e_j\in {\varepsilon }_i}{\beta }_{ij}{W}_2{f}_j^l\right)}$

where $h_i^l$ denotes the output representations of the node v_i, W₂ is a trainable matrix, β_jk expresses the attention weight coefficients, which satisfies: (7)${\displaystyle {\beta }_{ij}=\frac{exp\left({\mathbf{a}}_2^{\mathrm{T}}{\mathbf{v}}_j\right)}{\sum _{e_p\in {\varepsilon }_i}exp\left({\mathbf{a}}_2^{\mathrm{T}}{\mathbf{v}}_p\right)}{\mathbf{v}}_j=\text{LeakyReLU}\left(\left[{\mathbf{W}}_2{\mathbf{f}}_j^l\left|\right|{\mathbf{W}}_1{\mathbf{h}}_i^{l-1}\right]\right).}$

The HyperGraph layer incorporates a two-way attention mechanism for nodes and edges, enabling it to capture high-level interactions between words while highlighting the weight distribution at different levels of granularity. In sentences, where word semantics can be ambiguous, it is crucial to appropriately maintain or eliminate different semantic information. Resolving this ambiguity requires the establishment of a relatively stable feature distribution.

To address this challenge, we propose the reconstruction of the text information entropy matrix (TIEM). Given an encoded graph, we solve the problem by sampling the feature distribution to avoid semantic ambiguity during the graph reconstruction process. Initially, we obtain an encoded feature distribution denoted as Z, serving as an intermediate variable. The relationship between the encoded feature distribution h and the matrix A(TIEM) can be expressed as follows: (8)${\displaystyle q\left(Z\left|H,A\right.\right)=\prod _{j=1}^{n}q\left(z_j\left|H,A\right.\right)q\left(z_j\left|H,A\right.\right)=N\left(z_j\left|{\mu }_j,{\sigma }_j\right.\right)}$

where $\mu ={\text{AGGR}}_{\mu }^l\left(H,A\right),\sigma ={\text{AGGR}}_{\mu }^l\left(H,A\right)$ represent the mean and variance of the distribution, and Z follows Gaussian distribution. Next sampling from its distribution. Since the direct sampling process does not contain gradient information, it is useless for the whole network. Thus, the result is resampled: Z = μ + ɛσ, where $\varepsilon \sim N\left(0,1\right)$. The decoder reconstructs the graph by calculating the probability of an edge between any two nodes in the graph: (9)${\displaystyle p\left(A\left|Z\right.\right)=\prod _{j=1}^n\prod _{k=1}^{n}p\left(A_{i,j}\left|z_i,z_j\right.\right)p\left({\hat{A}}_{i,j}\left|z_i,z_j\right.\right)=\sigma \left(z_i^T{z}_j\right)}$

where $\hat{A}$ is the reconstructed TIEM, and the final feature presentation is obtained: (10)${\displaystyle \left\{\begin{array}{c}{f}_1={\text{conv}}_1\left(\text{relu}\left(\hat{A}HW\right)\right)\hfill \\ f_2={\text{conv}}_2\left(\text{relu}\left(\hat{A}HW\right)\right)\hfill \\ \hfill & \cdots \hfill \\ f_n={\text{conv}}_n\left(\text{relu}\left(\hat{A}HW\right)\right)\hfill \\ h_f=\left[f_1,f_2,\dots ,f_n\right]\hfill \end{array}\right.}$

The variable n represents the number of filters for the convolution layer. These filters play a crucial role in enhancing fragment fusion syntactic information. Moreover, the hypergraph layer is particularly important, because it enables the mapping of text features from a word-based perspective. This ultimately results in the production of a reconstructed feature vector, which can then be subjected to further backpropagation processing. The article shows how to successfully reconstruct TIEM, so we called this layer as hyper variational graph (HVG).

Layer number effects.

In the previous section, we provided a detailed explanation of the multi-layer HVG structure. The number of HVG layers, denoted by N, is a crucial hyperparameter in the multi-layer reasoning structure model, as it directly affects the model’s reasoning ability. To thoroughly investigate this matter, we conducted an ablation experiment to compare and analyze the model. In particular, we created five variations of the model, corresponding to N = 1, 2, 3, 4, and 5, respectively. As shown in Fig. 4, when N was set to 1 or 2, there was a slight increase in the model’s prediction accuracy. On the other hand, setting N to 3 resulted in the highest accuracy and F1 score. However, when N was set to 4 or 5, there was a significant decrease in the results. Based on our extensive experimental data, we can conclude that N = 3 is the optimal hyperparameter for the HVG layers, thus confirming the feasibility and effectiveness of our proposed algorithm. As a result, the multi-layer hypergraph variational reasoning module HVG can accurately represent text features, thereby enhancing its capabilities in text classification.

Window size effects.

This section aims to investigate the impact of different window sizes on matrix construction through a series of experiments. Specifically, several model experiments were conducted using window size values ranging from W = 1 to W = 6. As shown in Fig. 5, the experimental results convincingly demonstrate that the size of the window has a significant impact on the performance of the model. Moreover, for longer text classification datasets, such as the CLUEEmotions2020, the window effect is more prominent, especially when the window size is set to 4, 5, or 6. Among these, N = 4 shows the best results. Conversely, for shorter texts, the effect is less apparent. Although increasing the window size leads to some marginal improvement, the optimal situation still occurs when N = 4.

Noteworthily, the sliding window determines the range of key information covered in the sentence. Accordingly, better prediction results can be obtained by selecting an appropriate sentence length and window size, thereby verifying the effect of the window effect on the performance gain of the model.