Multi-granularity adaptive extractive document summarization with heterogeneous graph neural networks

Extractive document summarization

The significance and interrelationships among nodes of varying granularities are ascertained through mechanisms of adaptive width and depth. The majority of related studies concentrate on utilizing the encoder-decoder framework, often employing recurrent neural networks for this purpose (Cheng & Lapata, 2016; Nallapati, Zhai & Zhou, 2017; Zhou et al., 2018). Additionally, the adoption of pre-trained language models has gained traction for enabling contextual word representations in summarization tasks (Zhou et al., 2018; Liu & Lapata, 2019b; Zhong et al., 2019b).

Graph structures serve as another intuitive approach for extractive summarization, offering an enhanced capacity to exploit statistical or linguistic correlations between sentences. Initial efforts in this domain were centered on document graphs assembled based on content similarity among sentences, as exemplified by algorithms like LexRank (Erkan & Radev, 2004) and TextRank (Mihalcea & Tarau, 2004). More recent studies have ventured into the dissection of documents into nodes at diversified levels of granularity, employing heterogeneous graph neural networks for this objective.

Heterogeneous graph for NLP

Graph neural networks and their associated learning methodologies (Gilmer et al., 2017; Veličković et al., 2017) were initially conceived for homogeneous graphs, wherein all nodes share a uniform type. However, real-world applications often involve graphs with diverse types of nodes, or heterogeneous graphs. Recent studies have initiated exploratory efforts to model these complex structures (Shi et al., 2016). For instance, researchers have introduced heterogeneous graph neural networks to encode documents, entities, and candidates simultaneously for tasks like multihop reading comprehension (Tu et al., 2019). Others have focused on semi-supervised short text classification, employing a topic-entity heterogeneous neural graph (Linmei et al., 2019). In the realm of automatic document summarization, various approaches have been proposed, such as designing a heterogeneous graph comprising topic, word, and sentence nodes and employing a Markov chain model for iterative node updates (Wei, 2012). Algorithms like HeteroRank extend existing models (Wang, Chang & Huang, 2019) like TextRank by incorporating additional elements like keywords and sentences. Other models like HDSG Wang et al. (2020) employ words, sentences, and even paragraphs to form graph structures. The HEROS (Jia et al., 2020) model applies graph-based techniques to long-text fields, taking into account input article discourse. These methodologies primarily showcase two categories of variations: those pertaining to graph construction and those modifying the prior values of graph nodes. As for graph construction, the principal alterations relate to vertex granularity and edge weights. Some approaches operate under the assumption that significant sentences are composed of important words, employing a hybrid of graph models and sorting algorithms to rank both words and sentences (Fang et al., 2017). Other methods introduce more complex heterogeneous graphs that include not just words and sentences as nodes, but also integrate topic information. Yet another set of studies, such as those employing ExpandRank, utilize information from adjacent documents to enhance the sentence graph of a target document (Yang et al., 2018).The current approach maximizes the utility of both local document information and external data for evaluating the significance of sentences (Wan & Xiao, 2010). The method also incorporates topic information to bolster summarization efficacy. A distinguishing feature of this methodology is its incorporation of adaptive depth and breadth, facilitating the handling of long-distance dependencies across multiple layers. Additionally, the model adjusts the level of connectivity for each graph neural network (GNN) layer according to the local neighborhood size of individual nodes. Moreover, topic probability distribution values are employed as edge features between topic nodes and sentence nodes, thereby enabling the latter to aggregate more pertinent topics. Separate research has introduced innovative graph models for text summarization (Ferreira et al., 2013), constructing graphs based on a four-dimensional framework: similarity, semantic similarity, co-citations, and textual information. With regard to modifications in prior values, diverse strategies exist for assessing the preliminary importance of nodes within the graph. For example, a biased TextRank algorithm has been proposed to extract focused content by modifying the random restart probability based on node relevance to the targeted task (Kazemi, Pérez-Rosas & Mihalcea, 2020). Such a modification skews the algorithm toward selecting nodes of higher relevance. This particular methodology begins by scoring sentences using a pre-trained model, employing these scores to establish the preliminary importance of the sentences within the graph.

In recent years, the advent of distributed representations in natural language processing (Mikolov et al., 2013; Devlin et al., 2018) has led to a proliferation of deep learning-based techniques for automatic summarization (An et al., 2021; Liu, 2019). These methods have made strides in enhancing the efficacy of automatic document summarization, yet the task of managing long-distance dependencies continues to pose a significant challenge, and the training procedures are often resource-intensive. Notwithstanding, textual semantic information can be efficiently mapped as nodes within a graph-based framework. This configuration augments the model’s information flow and elevates the coherence of the generated summaries. The architecture also enables the efficient encoding of information across multiple levels of granularity. As a result, several methodologies have integrated multi-granular information with graph models to amplify text summarization effectiveness. In the present study, a sentence-topic-word structure has been established, proving instrumental in addressing cross-sentence dependencies for both single-document and multi-document summarization tasks (Chen, 2023; Mao et al., 2021; Shafiq et al., 2023). Moreover, an approach employing adaptive breadth and depth has been implemented to update nodes of varied granularity, thereby facilitating a more effective aggregation of node-specific information.

Modeling heterogeneous graphs

A heterogeneous graph G is defined as $G=\left\{V,E\right\}$, where V represents nodes of various classes and E represents edges between nodes. Detailed definition $V=V_w\cup V_s\cup V_t$, $E=E_{ws}\cup E_{wt}$, where $V_w=\left\{w_1,...,w_m\right\}$ denotes m unique words of the document, $V_s=\left\{s_1,...,s_n\right\}$ corresponds to the n sentences in the document and $V_t=\left\{t_1,...,t_n\right\}$ denotes j topics in the article. E is a real-value edge weight matrix and $E_{ws}^{ij}\ne 0$ $\left(i\in \left\{1,...,m\right\},j\in \left\{1,...,n\right\}\right)$ indicates the j-th sentence contains the i-th word, $E_{st}^{uv}\ne 0$ $\left(u\in \left\{1,...,n\right\},v\in \left\{1,...,g\right\}\right)$ indicates the u-th sentence contains the v-th topics.

Graph initializers

In the graph initialization phase, vector representations for each node category are generated through the utilization of GloVe embeddings. GloVe (Pennington, Socher & Manning, 2014) serves as a word embedding technique designed for the conversion of a vocabulary into a vector space, capitalizing on co-occurrence data from extensive text corpora. This results in high-quality vector representations that enrich the model’s comprehension of textual semantics. When compared with Transformer-based language models such as BERT (Devlin et al., 2018) and GPT (Radford & Narasimhan, 2018), GloVe embeddings require lower computational costs and offer quicker query speeds, thereby exerting minimal impact on the training pace of the model. Following this, features corresponding to sentences, topics, and words are extracted from the document. The architecture of the Transformer (Vaswani et al., 2017) has been shown to excel in the extraction and analysis of document features, employing self-attention mechanisms that permit the model to concentrate on various positions within the input sequence in a concurrent manner. This allows for effective capturing of long-range dependencies. Nevertheless, the Transformer model necessitates the retention of a complete self-attention matrix, which can lead to memory constraints when lengthy sequences are processed. Moreover, the performance of the Transformer is often contingent upon expansive pretraining datasets, demanding considerable volumes of textual data for its initial training. Given these considerations surrounding training time and computational resource needs, this study opts for CNNs (LeCun et al., 1998) furnished with diverse kernel sizes for the extraction of local features. Additionally, bidirectional long short-term memory (BiLSTM) (Graves & Graves, 2012) networks are employed to extract more global features during the feature extraction phase.

Specifically, let $X_w\in R^{m\ast d_s}$, $X_s\in R^{n\ast d_s}$ and $X_t\in R^{j\ast d_t}$ represent the input feature matrices corresponding to word nodes, sentence nodes, and topic nodes, respectively. Here, $d_w$ signifies the dimension of the word vector, $d_s$ denotes the dimension of the sentence vector, and $d_t$ indicates the dimension of the topic vector. For the extraction of topic and sentence features, CNNs equipped with various kernel sizes are utilized to obtain local n-gram features, represented as $L_j$. Subsequently, a BiLSTM is deployed to capture global features, denoted as $G_j$. The resultant sentence node features and topic node features are formulated as $X_{sj}=\left[L_j;G_j\right]$ and $X_{ti}=\left[L_i;G_i\right]$, through the concatenation of CNN local features and BiLSTM global features.

CNNs furnished with different kernel sizes have the capability to extract features at multiple scales, enabling the model to capture the document’s content in a comprehensive manner. Moreover, BiLSTM maps each word in the document to its corresponding representation within the sentence, thereby incorporating the contextual nuances of the vocabulary. This contributes to an enhanced understanding of the contextual relationships among words within the document.

As for topic nodes, which serve as more complex semantic units, probability distribution values are obtained through the latent Dirichlet allocation (LDA) (Blei, Ng & Jordan, 2003) topic model and are incorporated into the edge weights between the topic and sentence nodes. This process ensures that edge features encapsulate crucial relationships between the topic and sentence nodes. Likewise, TF-IDF values are infused into the edge weights connecting word nodes and sentence nodes. Term frequency (TF) refers to the occurrence frequency of a word within a document, while inverse document frequency (IDF) represents the inverse function of a word’s outdegree.

Graph neural network layer

Adaptive layer

Since there are nodes of different thickness and fine granularity defined in the graph structure, nodes of different granularity need different propagation widths and depths to capture the complex relationship between propagated information. Therefore, it is necessary to learn the importance and correlation between nodes of different granularity by means of node adaptive width and depth. For adaptive width, the feature vectors of neighbor nodes are weighted and summed by improved GATv2 to aggregate 1-hop features. For the adaptive depth, the forgetting and saving between different hops can be controlled by fusing an LSTM module. In traditional GAT, information between different hop can be implicitly fused through multi-layer GAT. However, it is difficult for this fusion method to ensure that only features of feature regions are fused and features of other regions are discarded, while LSTM can automatically forget some links to ensure feature fusion. The adaptive width is completed by GATv2, and the adaptive depth layer is designed as follows:

$$I_i=\sigma \left(W_i^{{\left(t\right)}^T}{u}_i\right).$$

$$f_i=\sigma \left(W_f^{{\left(t\right)}^T}{u}_i\right).$$

$$o_i=\sigma \left(W_o^{{\left(t\right)}^T}{u}_i\right).$$

$$\tilde{C}=\tan \mathrm{h}\left(W_i^{{\left(t\right)}^T}{u}_i\right).$$

$$C_i^{\left(t+1\right)}=f_i\odot C_i^t+I_i\odot \tilde{C}.$$

$$h_i^{\left(t+1\right)}=o_i\odot \tan \mathrm{h}\left(C_i^{\left(t+1\right)}\right).$$

The gating unit $I_i$ with sigmoid output is used to extract new useful signals from $\tilde{C}$, and the two are added to the memory after dot multiplication $I_i\odot \tilde{C}$. The gating unit $f_i$ is used to filter useless signals in the old memory, and a new neighborhood is given. Therefore, the memory of each node can be filtered as $C_i^{\left(t+1\right)}$.

Finally, the embedding value $h_i^{\left(t+1\right)}$ of node i in $(t+1)-t$ layer can be output by using the gating unit $o_i$ and the latest memory $C_i^{\left(t+1\right)}$. The overall adaptive width and depth layers are shown in Fig. 2.

Iterative updating

The topic, word, and sentence nodes in the diagram structure need to pass messages to each other for updates, so we design message propagation as Fig. 3. Specifically, after the graph attention layer has processed the data, the data is then entered into a position-wise feed-forward (FFN) layer consisting of two linear transformations, like Transformer (Vaswani et al., 2017).

After initializing the nodes, we use neighbor nodes to update the feature representation of sentence nodes with improved GATv2 and FNN layer and adaptive depth layer. The details are as follows:

$$U_{s\leftarrow w}^1=GAT\left(H_s^0,H_w^0,H_w^0\right).$$

$${\hat{H}}_s^1=FFN\left(Depth\left(U_{s\leftarrow w}^1\right)+H_s^0\right).$$

$$U_{s\leftarrow w}^1=GAT\left({\hat{H}}_s^1,H_t^0,H_t^0\right).$$

$$H_s^1=FFN\left(Depth\left(U_{s\leftarrow t}^1\right)+{\hat{H}}_s^1\right).$$where $H_w^1=H_w^0=W_s$, $H_s^0=X_s$ and $H_{s\leftarrow w}^0\in R^{m\ast d_h}$. $depth()$ is the adaptive depth layer, $GAT\left(H_s^0,H_w^0,H_w^0\right)$ indicates that $H_s^0$ is used as the attention query, and $H_w^0$ is used as the key and value.

Then, the updated sentence nodes are brought into the calculation together with the unupdated word nodes or topic nodes to obtain the updated representations of the word nodes and topic nodes, and the updated word nodes and topic nodes are used to further iteratively update the sentence nodes. Denote the process of t iterations as follows:

$$U_{t\leftarrow s}^{t+1}=GAT\left(H_t^t,H_t^t,H_s^t\right).$$

$$H_t^{t+1}=FFN\left(Depth\left(U_{t\leftarrow s}^{t+1}\right)+H_t^t\right).$$

$$U_{w\leftarrow s}^{t+1}=GAT\left(H_w^t,H_s^t,H_s^t\right).$$

$$H_w^{t+1}=FFN\left(Depth\left(U_{w\leftarrow s}^{t+1}\right)+H_w^t\right).$$

$$U_{s\leftarrow t}^{t+1}=GAT\left(H_s^t,H_t^{t+1},H_t^{t+1}\right).$$

$${\hat{H}}_s^{t+1}=FFN\left(Depth\left(U_{s\leftarrow w}^{t+1}\right)+H_s^t\right).$$

$$U_{s\leftarrow w}^{t+1}=GAT\left({\hat{H}}_s^{t+1},H_w^{t+1},H_w^{t+1}\right).$$

$$H_s^{t+1}=FFN\left(Depth\left(U_{s\leftarrow w}^{t+1}\right)+{\hat{H}}_s^{t+1}\right).$$

As shown in Fig. 3, document-level information in sentences is aggregated into topic nodes and word nodes in order to fuse information effectively. For example, if a topic node has a high probability in many sentences, it is likely to be a key topic for this document. In the sentence nodes, the sentence focus often selects the nodes with more words and key topics as the summary.

Sentence selection

In the final stage, sentence nodes are classified to identify those that contain words of greater importance within the heterogeneous graph. The model undergoes training, targeting minimization of cross-entropy loss. Trigram Blocking is employed for the decoding phase, wherein each sentence is ranked based on its associated score, and sentences featuring overlapping letter combinations with preceding ones are discarded.

Multi-document summary

In multi-document summarization, the interrelationships between document-level nodes play an indispensable role for the model’s ability to discern core topics and prioritize content across multiple documents. Yet, many extant models neglect this hierarchical organization, opting instead for a flat, sequential arrangement of multiple documents (Liu et al., 2018). Alternative approaches attempt to model these document-level relationships through attention-based, fully-connected graphs, or by leveraging similarity and statement relationships (Liu & Lapata, 2019a).

To establish document-level relationships analogously to sentence-level counterparts, the model can be effortlessly adapted for multi-document summarization by incorporating super-nodes (as shown in Fig. 4) for each document. Specifically, the heterogeneous graph is augmented with four distinct types of nodes: $V_w\cup V_s\cup V_d\cup V_t$, $V_d=\left\{d_1...d_l\right\}$, $l$ represents the number of documents.

As illustrated in Fig. 4, word nodes serve as the intermediary between sentence and document nodes, with relationships being established based on content similarity. Topic nodes directly link with sentence nodes according to the document’s importance.

A document node, which is essentially a specialized form of a sentence node but with extended content, associates with its corresponding word node, utilizing the TF-IDF value as the edge weight. The updating procedure for document nodes parallels that of sentence nodes, with the key distinction being in initialization; the document node is initialized via mean-pooling of the sentence node features. During the sentence node selection process in multi-document mode, the representation of each sentence node is concatenated with the document representation, leading to the final score post multi-document aggregation.

Datasets

Experimental settings

The vocabulary is restricted to 50,000, with token initialization using 300-dimensional GloVe embeddings. A total of 6,214 word nodes are created, while filtering out stop words and punctuation. The input document is truncated to 50 sentences of maximum length. To mitigate the impact of noisy common words, $10\mathrm{\%}$ of the vocabulary with the lowest TF-IDF values across the entire dataset is eliminated. Sentence nodes, phrase nodes, and entity nodes are initialized as $d_s=d_p=d_v=128$, and the edge feature $e_{ij}$ is set to $d_e=50$. Each GATv2 layer consists of eight heads, the hidden size is $d_h=64$, and the internal hidden layer size of the FFN layer is 512.

During training, a batch size of 32 is employed, and the Adam optimizer (Fabbri et al., 2019) is utilized with a learning rate of $5e^{-4}$. The iteration number $t=1$ is selected based on performance on the validation set. Regarding the length of abstracts, the CNN/DailyMail dataset incorporates the first three sentences, while the Multi-News dataset includes the first nine sentences, based on the average length of manually generated abstracts.

Models for comparison

Single-document summarization

The single-document model is assessed using the CNN/DailyMail dataset, reporting overlap metrics of unigram, bigram, and longest common subsequence in relation to R-1, R-2, and R-L scores. Given the constraints of computational resources, a pre-trained contextual encoder was not incorporated into the mode (Devlin et al., 2018). Consequently, comparisons are made solely among models that do not employ BERT.

Multi-document summarization

Initially, as a baseline metric, the top k sentences from each source document in the dataset are concatenated. Subsequently, additional models employ the output generated from the code and model made public by Fabbri et al. (2019).

Analysis of feature extraction methods

The methodology for extracting features from primary topic nodes and sentence nodes is an essential aspect of the model’s training. Beyond the utilization of BiLSTM, incorporation of BiGRU has also been explored. BiGRU (Cho et al., 2014), similar to LSTM, employs gating mechanisms but usually requires fewer parameters and incurs lower computational overhead. Although BiLSTM generally excels in managing long-range dependencies owing to its specialized gated mechanisms, BiGRU also demonstrates competence in handling such dependencies, albeit possibly necessitating additional time steps.

Initial vector representations for sentences and topics, denoted by $X_s\in R^{n\ast d_s}$ and $X_t\in R^{j\ast d_t}$, are acquired via GloVe word embeddings. Following this, BiGRU is applied to glean global features, $G_j$, while Convolutional Neural Networks (CNN) with diverse kernel sizes are utilized to capture local n-gram features, $L_j$. The amalgamation of CNN-derived local features and BiGRU-obtained global features results in the sentence node features $X_{sj}=\left[L_j;G_j\right]$ and topic node features $X_{ti}=\left[L_i;G_i\right]$.

Figure 5 presents the ROUGE score trajectories for both BiLSTM and BiGRU. It is evident that BiLSTM’s enhanced capability in managing long-distance dependencies renders a more favorable impact on the ROUGE-L score. Balancing the trade-offs between training speed and model quality, it is concluded that BiLSTM serves as a more appropriate choice for lightweight models.

Limitations

The model assumes a basic premise that multiple source files share at least one theme. This assumption may limit the applicability of the model in document sets with non-existent or significantly different thematic structures. Future research directions include exploring the complex relationships of entities and events, the sparsity of information, and the density of data in various input fields (such as dialogue summarization, long document summarization) for this model. Future avenues also include the potential for pre-training the model, paralleling other large-scale pre-trained neural summarizers, which would necessitate an additional encoding layer, thereby increasing the model’s complexity. For the purpose of training a BART-based architecture, a GPU endowed with a minimum of 32 GB memory capacity is a requirement. Subsequent endeavors may explore the feasibility of model distillation into more compact configurations without compromising performance efficacy.