Graph-based deep fusion for architectural text representation

BERT

BERT, a pre-trained NLP model introduced by Google in 2018, utilizes the Transformer architecture, known for its self-attention mechanisms. To predict the sentiment polarities for the given aspects or targets, Li et al. (2020a) use gating mechanisms and context-aware aspect embedding to enhance and control the BERT representation of aspect-based sentiment analysis. Yang et al. (2020) leveraged BERT to encode captions and videos, enabling them to answer questions related to the visual content of an image or video. Moon, Chi & Im (2022) further employ a BERT-based utterance classification model to identify risk information in each utterance within a architecture specification. Kim et al. (2022) employed BERT to extract pertinent information from architecture codes, aiming to develop an information retrieval chatbot in alignment with building codes. Gao et al. (2019) harnessed BERT’s context-awareness for conducting sentiment analysis on complex sentences within aspect-based sentiment analysis.

RoBERTa

RoBERTa is a Transformer-based pre-trained model that has a series of enhancements over BERT, including eliminating the next sentence prediction task (Liu et al., 2019). Liu et al. (2022) used RoBERTa to extract entities and relationships from open-source intelligence for constructing a knowledge graph in a specific domain. They validated their approach using different text similarity algorithms and classification methods. Ahammad et al. (2024) introduces the RoBERTa-GCN model, which combines RoBERTa with a GNN, to effectively address the issue of fake news detection in the Bengali language. Kong et al. (2024) adopts RoBERTa and Global Vectors for Word Representation (Glove) for global vector representation, solving the challenge of automatically detecting cyberbullying in tweets. Li et al. (2020b) introduced a RoBERTa-based word embedding model using the power grid corpus, applying it to the task of entity recognition. In addition, Dai et al. (2021) employed fine-tuning with RoBERTa to conduct sentiment analysis on English sentences characterized by intricate grammatical structures.

Graph neural networks

GNNs are connectionist models designed to capture dependencies and relationships among graph nodes by facilitating message passing through the edges connecting these nodes (Scarselli et al., 2008). Yao, Mao & Luo (2019) proposed using text graph convolutional networks for text classification, which builds a unified text graph based on word co-occurrence and document-word relations. Research has also explored the integration of BERT’s semantic understanding capabilities with the graph structure of GCN to tackle diverse tasks. BERTGCN integrates traditional BERT and graph convolutional network (GCN) models. In the BERTGCN model, BERT serves as an embedding layer to represent documents at a foundational level and integrates them into GCN as the node representation. This approach combines the deep semantic feature extraction capabilities of BERT with the ability of GCN to capture relationships in structured data, such as graph structures (Xue, 2023; Ding et al., 2022; Yang & Liu, 2023; Xu et al., 2023). She, Chen & Chen (2022) employed a joint model that combines BERT and GCN for text classification tasks. Similarly, Xiao et al. (2021) employed a joint model for emotion classification, leveraging BERT’s semantic understanding capabilities and GCN’s strengths in capturing relationships between nodes. Kong et al. (2024) propose a GNN approach for building function classification that integrates multi-source data and mines contextual information between buildings. Wang, Sacks & Yeung (2022) introduces a novel approach to semantic enrichment of building information modeling (BIM) models using graph representation and GNNs. Wang, Sacks & Yeung (2022) use the room classification task to develop, test and illustrate a novel approach to semantic enrichment of BIM models—representation of models as graphs and application of graph neural networks (GNNs). These studies not only expand the application of graph neural networks in the field of natural language processing but also provide new ideas and methods for solving complex text analysis problems.

Semantic fusion

Our principal objective is to conduct an effective methodology for architectural text representation, with a specific focus on enhancing semantic richness. The texts are denoted as $T_a=\left\{a_1,a_2,\dots ,a_m\right\},a_i\in A$, $m$ refers to the number of architectural texts. Our methodology involves concurrently inputting each text into BERT and RoBERTa to obtain two sets of token embeddings. The text embedding, denoted as $u_{bert}$, is derived by computing the mean pooling of token embeddings obtained from BERT. Similarly, the text embedding, denoted as $v_{roberta}$, is obtained by applying mean pooling to the token embeddings acquired from RoBERTa. Subsequently, the integration of these two disparate semantic representations yield the composite embedding denoted as $e_{composite}$.

Graph-based semantic enhancement

A large heterogeneous text graph is constructed, which includes keyword nodes and document nodes. The extraction of keyword nodes utilizes the term frequency-inverse document frequency (TF-IDF) method, which calculates the term frequency (TF) of each term in the document and the inverse document frequency (IDF) across the entire corpus, and then multiplies these two values to obtain the TF-IDF score. The terms with the highest scores are selected as keywords. The weight of the edges is obtained by calculating the PMI value between pairs of words, which is calculated using the formula $PMI(i,j)=\log \frac{p(i,j)}{p(i)p(j)}$, where $p(i,j)$ is the probability of words $i$ and $j$ co-occurring, and $p(i)$ and $p(j)$ are the probabilities of words $i$ and $j$ occurring, respectively.

Keyword nodes can be represented as a set $W=\{w_1,w_2,...,w_{n_1}\}$, where $w_i$ is a term in the vocabulary, and $n_1$ is the size of the vocabulary. Document nodes can be represented as a set $D=\{d_1,d_2,...,d_{n_2}\}$, where $d_j$ is a document in the dataset, and $n_2$ is the size of the dataset. The number of nodes in the text graph, denoted as |V |, $|V|=n_1+n_2$. This graph can be represented by an adjacency matrix $A\in R^{|V|\times |V|}$, where $A_{ij}$ represents the weight of the edge between nodes $i$ and $j$.

We input this heterogeneous graph into GAT. GAT can capture and learn the complex relationships between nodes in the graph. A key feature of GAT is its ability to learn the weights of the edges in the graph. This attention mechanism is based on the features of the neighbor nodes $h_j$ and the current node $h_i$. Nodes aggregate the features of neighbor nodes according to these weights $a_{ij}$, generating their new features $h_i^{\prime }$. In this way, GAT can effectively capture and learn the complex structure of the text graph, and finally output a text embedding that is richer in semantics.

Semantic fusion module

In the semantic fusion module, we propose a novel approach for constructing robust embeddings for architecture texts by leveraging the strengths of both BERT and RoBERTa models. The text matrix of architecture text $T_a$ is composed by architecture text:

(1) $$T_a=\left\{a_1,a_2,\dots ,a_m\right\},a_i\in A$$where $a_i$ refers to the i-th Architectural text, $m$ refers to the number of Architectural texts.

Utilizing BERT to generate embeddings for a given text $a_i$ involves the initial step of feeding the $a_i$ through the BERT model, which produces contextualized word embeddings. Following this, the mean-pooling operation is applied to these embeddings across the entire sequence, yielding a singular vector representation $u_e^i$ that encapsulates the aggregated semantic information of the input text. we refer to this function as $f_1$. This resultant embedding $u_e^i$ serves as a condensed representation of the original text $a_i$.

The process of obtaining text embeddings through BERT can be formulated by the following equation:

(2) $$\begin{array}{rl}{E}_{bert}& =F_1(T_a)\\ & =\{f_1(a_1),f_1(a_2),\dots ,f_1(a_m)\}\\ & =\{u_e^1,u_e^2,\dots ,u_e^m\},a_i\in T_a\end{array}$$where $u_e^i$ is the embedding of Architectural text $a_i$, $m$ refers to the number of Architectural texts.

Leveraging RoBERTa for text embedding involves initially processing the input text $a_i$ through the RoBERTa model, generating contextualized word embeddings. Unlike BERT, RoBERTa adopts a masked language model objective during pretraining, facilitating a deeper understanding of contextual relationships. The embeddings are then subjected to the subsequent pooling operation, such as mean-pooling across the entire sequence. We refer to this function as $f_2$. Similarly, this operation also results in a singular vector representation $v_e^i$, capturing the aggregated semantic information of the input text. This resultant embedding $v_e^i$ serves as a contextually rich representation of the Architectural text. The process of obtaining text embeddings through BERT can be formulated by the following equation:

(3) $$\begin{array}{rl}{E}_{roberta}& =F_2(T_a)\\ & =\{f_2(a_1),f_2(a_2),\dots ,f_2(a_m)\}\\ & =\{v_e^1,v_e^2,\dots ,v_e^m\},a_i\in T_a\end{array}$$where $v_e^i$ is the embedding of architectural text $a_i$, $m$ refers to the number of architectural texts.

Subsequently, the embeddings from both models are concatenated to form a composite textual embedding, providing a more comprehensive representation of semantic information. This can be expressed mathematically as follows:

(4) $$e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}^i=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(u_e^i,v_e^i)$$

(5) $$E_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}^i=\{e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}^1,e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}^2,\dots ,e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}^m\}$$where $e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}^i$ is the fusion embedding of Architectural text $a_i$, $m$ refers to the number of architectural texts.

This hybrid embedding approach leverages the differences between BERT and RoBERTa in text representation processing. By combining BERT’s next sentence prediction (NSP) task with RoBERTa’s dynamic masking mechanism, and concatenating the resulting vectors, the model enhances its ability to extract semantic features from architectural text.

Graph-based semantic enhancement module

Text graph

The goal of this module is to construct a large, heterogeneous text graph that integrates both word nodes and document nodes. This approach aims to explicitly model global word co-occurrences, providing a more comprehensive representation of text relationships. Additionally, it enhances the flexibility of adjusting graph convolution operations. An example illustrating the method used in this article for graph construction is presented in Fig. 2.

To ensure consistency in the keywords extracted by both models when merging the keyword tables generated by BERT and RoBERTa, we opted not to use these models for keyword extraction. Instead, we employed the TF-IDF method. First, preprocessing operations such as word segmentation are performed on the text to create a basic vocabulary list. Subsequently, we compute the TF for each term in the document, as well as the IDF across the entire corpus. By multiplying each term’s TF and IDF values, we can calculate its TF-IDF score. This score reflects the term’s importance in a specific document and its uniqueness in the entire corpus.

In order to filter out the most representative keywords, we designed a ranking mechanism to rank terms in the vocabulary based on their TF-IDF scores. Finally, the top five terms with the highest TF-IDF scores are selected as the keywords of the document. This approach prioritizes terms that appear frequently in documents but are relatively rare in the entire corpus, thus providing a richer and more comprehensive representation of semantic information.

The number of nodes in the text graph, denoted as |V |, is equal to the sum of the number of documents in the Architectural text corpus and the size of the vocabulary of unique words in the Architectural text. Our methodology involves concurrently feeding each text into both BERT and RoBERTa models, yielding two sets of token embeddings. These text embeddings, denoted as $e_{\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{t}}$ and $e_{\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{a}}$, are derived by computing the mean pooling of the token embeddings obtained from each model. Subsequently, we concatenate $e_{\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{t}}$ and $e_{\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{a}}$ to form a combined embedding denoted as $e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}$. Every word and document is represented as $e_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}}$ and $e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}$, which serve as the input for GAT.

We construct edges between nodes based on the occurrence of words in documents (document-word edges) and the co-occurrence of words across the entire corpus (word-word edges). The weights of edges between document nodes and word nodes are determined by TF-IDF of words in the documents, where term frequency represents the number of occurrences of a word in a document, and inverse document frequency is the logarithmically scaled inverse of the number of documents containing that word. We found that using TF-IDF weights performs better than using term frequency alone.

In order to fully exploit the global word co-occurrence information, we adopt a fixed-size sliding window to traverse all documents in the corpus and collect co-occurrence statistics. When calculating the weight between two word nodes, we use pointwise mutual information (PMI), a popular measure of word association. In our preliminary experiments, we observed that employing PMI yielded superior results compared to using word co-occurrence counts.

Formally, the weight of the edge between nodes $i$ and $j$ is defined as:

(6) $$A_{ij}=\{\begin{array}{cc}PMI(i,j)& \mathrm{i}\mathrm{f}i,j\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{s}\mathrm{a}\mathrm{n}\mathrm{d}i\ne j\\ TF-ID{F}_{ij}& \mathrm{i}\mathrm{f}i\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{d}\mathrm{o}\mathrm{c}\mathrm{u}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{d}j\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\\ 1& \mathrm{i}\mathrm{f}i=j\\ 0& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}$$

TF-IDF The PMI value of a word pair i, j is computed as:

(7) $$\begin{array}{c}PMI(i,j)=\log \frac{p(i,j)}{p(i)p(j)}\hfill \\ p(i,j)=\frac{\mathrm{\#}W(i,j)}{\mathrm{\#}W}\hfill \\ p(i)=\frac{\mathrm{\#}W(i)}{\mathrm{\#}W}\hfill \end{array}$$where $\mathrm{\#}W(i)$ represents the number of sliding windows in the corpus that contain word i, and W(i,j) represents the number of sliding windows in which both word i and j co-occur. Positive PMI values indicate higher semantic relevance between words in the corpus, while negative PMI values indicate low or no semantic relevance. Therefore, we only add edges between word pairs that have positive PMI values.

In the graph, nodes D4, D23, and D101 correspond to the labels assigned to architectural texts. other words represent the keywords extracted from the text. The black lines indicate the connections between the articles and the keywords, whereas the orange line represents the connections between the keywords and the articles. The complete configuration constitutes a graph structure, which serves as the input for the GAT model.

GAT

In the architectural text (Pal, Selvakumar & Sankarasubbu, 2020), for any two nodes $i,j$, the $e_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}}$ and $e_{\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{e}}$ are used as the initial node feature representations, respectively, and the two are spliced together, and then the inner product is computed with a weight vector $\alpha$ to derive the attention coefficient ${\alpha }_{ij}$ used to measure the importance of the relationship between node $i$ and node $j$. The formula is calculated as follow:

(8) $${\alpha }_{ij}=\frac{\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({\alpha }^T\left[w{h}_i||w{h}_j\right]\right)\right)}{{\sum }_{k\in N_i}\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({\alpha }^T\left[w{h}_i||w{h}_k\right]\right)\right)}$$

In the above equation, N is the total number of nodes, || is the splicing operation, denoted as transpose, $w$ is the weight matrix, and $k$ is one of the N nodes. LeakyReLU is an activation function that allows a small, non-zero gradient for negative inputs, preventing the “dying ReLU” problem by introducing a slight slope for negative values. The feature vector of node $i$ after the attention mechanism can be expressed as:

(9) $$h_i^{\prime }=W_a\left(\sum _{j\in N_i}{\alpha }_{ij}W{h}_j\right)$$

$W_a$ is the corresponding weight matrix, $h_i$ is the feature vector of node $\mathrm{i}$ and $h_j$ is the feature vector of node $\mathrm{j}$. In order to stabilize the learning process of self-attention, we incorporate the use of multi-head attention. In our model, K attention mechanisms (K = 8 in the study) are employed to perform the transformation described in the equation above. The resulting features from these K attention mechanisms are then concatenated. For the final layer of the network, we apply multi-head attention using an averaging approach. The outputs of the K attention mechanisms are processed through an aggregation function, and the output of the final attention layer as $h_i^{\ast }$:

(10) $$h_i^{\ast }=\frac{1}{K}\sum _{k=1}^K{W}_a\left(\sum _{j\in N_i}{\alpha }_{ij}^k{W}^k{h}_j\right).$$

In Fig. 3, node $h_i$ is connected to its neighboring nodes, represented by node $h_1,h_2,h_3,h_4,h_5$. The attention coefficients between node $i$ and its neighboring nodes are denoted by ${\mathit{\alpha }}_{\mathit{i}1},{\mathit{\alpha }}_{\mathit{i}2},{\mathit{\alpha }}_{\mathit{i}3},{\mathit{\alpha }}_{\mathit{i}4},{\mathit{\alpha }}_{\mathit{i}5}$. The graph in the illustration only shows a single-head attention mechanism.

Finally, We can get the embeddings of the architectural text, ultimately contributing to improved downstream tasks and applications.

Time and space complexity analysis

For the combined BERT and RoBERTa model, the time complexity is primarily determined by the self-attention mechanism and the feedforward neural network layers. The time complexity can be expressed as:

$$T_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}+\mathrm{R}\mathrm{o}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\mathrm{a}}=O\left(L_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}\cdot (n^2\cdot d+n\cdot d^2)+L_{\mathrm{R}\mathrm{o}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\mathrm{a}}\cdot (n^2\cdot d+n\cdot d^2)\right)$$where $L_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}$ and $L_{\mathrm{R}\mathrm{o}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\mathrm{a}}$ denote the number of layers in BERT and RoBERTa, $n$ is the input sequence length, and $d$ is the embedding dimension. The self-attention mechanism contributes a complexity of $O(n^2\cdot d)$, while the feedforward neural network adds a complexity of $O(n\cdot d^2)$.

Similarly, the space complexity for the combined BERT and RoBERTa model, which includes the storage of model parameters and intermediate activations, is given by:

$$S_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}+\mathrm{R}\mathrm{o}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\mathrm{a}}=O\left(L_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}\cdot (d^2+n\cdot d)+L_{\mathrm{R}\mathrm{o}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\mathrm{a}}\cdot (d^2+n\cdot d)\right)$$where $L_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}$ and $L_{\mathrm{R}\mathrm{o}\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}\mathrm{a}}$ represent the number of layers, $d$ is the embedding dimension, and $n$ is the sequence length.

For the GAT (Graph Attention Network), the time complexity is largely influenced by the number of nodes and edges in the input graph and the attention mechanism used during graph convolution. The time complexity for a single layer of GAT is:

$$T_{\mathrm{G}\mathrm{A}\mathrm{T}}=O\left(L_{\mathrm{G}\mathrm{A}\mathrm{T}}\cdot (|V|\cdot d^2+|E|\cdot d)\right)$$where $L_{\mathrm{G}\mathrm{A}\mathrm{T}}$ represents the number of GAT layers, |V| is the number of nodes, |E| is the number of edges in the graph, and $d$ is the feature dimension of each node. The $O(|V|\cdot d^2)$ term arises from the dense transformation of node features, while the $O(|E|\cdot d)$ term stems from the attention mechanism, which computes attention coefficients for each edge.

The space complexity for GAT involves storing node features, edge features (attention coefficients), and intermediate activations. It is expressed as:

$$S_{\mathrm{G}\mathrm{A}\mathrm{T}}=O\left(L_{\mathrm{G}\mathrm{A}\mathrm{T}}\cdot (|V|\cdot d+|E|)\right)$$where |V| and |E| represent the number of nodes and edges, respectively, and $d$ is the dimension of the node embeddings. The term $|V|\cdot d$ accounts for storing node embeddings, and |E| corresponds to the space needed to store edge-wise attention coefficients.

Datasets

Due to the lack of professional dataset at the architecture domain, we collected text data from professional websites at the architecture field through crawlers and other techniques. Meanwhile, we invited experts in the domain of a total of 20,013 architecture to guide and label the dataset, which contains 13 categories and data. These categories cover a variety of topics, including architecture materials, science, and architecture appreciation. To measure the model’s ability to generalize over generic dataset, we used generic news dataset to evaluate its performance. The generic news dataset contains 18,569 news articles from 2005 to 2011. It covers 14 different categories including various topics such as sports, finance, and stocks. The architecture dataset and news dataset are shown in Table 1.

Training settings

Before training the model, the two dataset are randomly divided into a training set and a test set in the ratio of 8:2. In order to ensure that the model can converge stably and achieve optimal performance, we adopt the Adam optimizer and set the batch size to 5 and the learning rate to 0.00005. We also implement algorithms including dropout and regularization to avoid over-fitting, which generates a slight improvement in model performance and thus will not be discussed in detail. On both dataset, we train our models for at most 60 epochs to obtain the best performances. In the experiments, the graph is constructed using documents and keywords as nodes. Keyword nodes are linked to document nodes based on their occurrence within the documents, and keyword nodes are interconnected based on the semantic similarity of their embeddings. This structure captures both document-level and term-level relationships within the text.

Baseline

To evaluate the effectiveness of graph-based deep fusion method, this article designed comparative experiments including Word2Vec, XLNet, SentenceTransformer, Bert, BERT+GCN and TextConvoNet. In evaluating the representational capabilities of each model, the performance of each embedding method was chosen to be judged with a text categorisation task and accuracy, precision, recall and F1-score were used as evaluation metrics. The baseline models are described in detail below.

• Word2Vec (Church, 2017): Word2Vec utilizes a shallow neural network to train word embeddings.

• XLNet (Yang et al., 2019): XLNet combines the self-attention mechanism of the Transformer with the benefits of an autoregressive language model. It learns bidirectional context relationships of words through a permutation language modeling approach.

• SentenceTransformer (Reimers, 2019): SentenceTransformer is a BERT-based framework specifically designed to generate sentence-level embeddings. It modifies the target function and the structure of BERT to produce embeddings that are more suitable for comparing sentences.

• BERT (Devlin et al., 2018): BERT is a deep learning model that uses the Transformer for pre-training language representations. It learns word embeddings by considering the context on both sides of the sentence simultaneously.

• BERT+GCN (Lin et al., 2021): The BERT+GCN method combines GCN and BERT to improve performance to further improve performance through graph structures. It applies BERT to process textual data and produce node embeddings, which are then fed into the GCN. The document embedding is iteratively updated based on the graph structure using GCN, and its output is considered the final embedding of the document node.

• TextConvoNet (Soni, Chouhan & Rathore, 2023): TextConvoNet is a CNN-based architecture for binary and multi-class text classification. It captures n-gram features both within and across sentences, using a novel input matrix representation and applying two-dimensional multi-scale convolution for feature extraction.

Results and analysis

To validate the effectiveness of our proposed method, we first evaluate the model’s representational capabilities on the architecture dataset to demonstrate its performance on architecture domain. Subsequently, we tested the model on the news dataset to verify its generality and generalization ability. In the experiments, we selected all the baselines mentioned in “Baseline” for comparison along with the ablation study of the models. Also we used evaluation metrics to evaluate the model performance. Table 2 lists the overall results on both datasets with the evaluation metrics mentioned, and Table 3 lists the results of the ablation study on both datasets.

Table 2 shows the accuracy of various models tested on the architecture domain dataset and the news dataset. In the architecture domain dataset, The graph-based deep fusion method performs the best with 63.22%, 60.91%, 57.78%, and 58.45% in terms of accuracy, precision, recall, and F1-scores of its model, which improves compared to BERT in these four metrics by improving the improvement of 7.51%, 7.52%, 6.87%, and 7.54%, respectively. To further validate the generalisation of the model, it can be seen that on the news dataset, the graph-based deep fusion method achieves a significant performance, with an accuracy of 95.39%, which is significantly higher than that of other models for classification. This highlights the strong applicability of this method. This approach takes advantage of the complex text encoding capability of BERT and RoBERTa, and the complex relationship modelling capability of GAT, which enhances the model’s ability to capture textual contextual relationships, improves the text modelling capability of BERT and RoBERTa, and ultimately improves the accuracy of text classification. Of course, compared to the news dataset, the metrics of all models decreased on the building dataset, reflecting the problems associated with this highly specialised dataset. Nonetheless, this proposed approach still performs well on all four evaluation metrics. The poor performance of these baselines in the architectural domain highlights the limitations of word embedding models and pre-trained models in capturing the latent features specific to architectural text. Our approach effectively exploits the differences between BERT and RoBERTa by combining them to capture textual features that are ignored by individual models. The attention mechanism introduced by the GAT graph neural network improves the quality of the keyword representations, which in turn improves the robustness and performance of the models.

In addition, we conducted ablation study on the models. The results in Table 3 indicate that the absence of the BERT or RoBERTa model results in a decline in metrics. For instance, in the precision metric, the removal of BERT and RoBERTa models led to decreases of 2.58% and 2.94%, respectively. This is because the combination of BERT and RoBERTa, leveraging their complementary features, allows for a more comprehensive understanding of architectural text. In addition, we removed the GAT module and replaced it with GCN, observing a significant decrease in the mode’s performance across various parameters. The drop in performance indicators highlights the critical role of GNN in generating the final document representation, with GAT demonstrating the most substantial impact on improving the model. The attention mechanism in GAT enables the model to learn accurate representations of keywords from context, leading to a better understanding of relationships between keywords. Moreover, comparative experiments using the graph-based deep fusion method highlight the contribution of the joint representation to the model’s ability to represent architectural text, enhancing its robustness and performance. RoBERTa compensates for any shortcomings that BERT may exhibit on specific architectural data. The ablation study emphasize the importance of each component in the proposed method structure, as the removal of either BERT or RoBERTa results in a decline in evaluation metrics across all aspects.

Analysis of reduced dimension

In addition, this study includes the reduced dimension by t-SNE for the different embedding methods, followed by a comparative analysis through the visualisation of scatter plots. The primary criterion for evaluation is the degree of clustering within the same categories in the scatter plots. The results of the performance of each model in the architecture dataset after reduced dimension are shown in Fig. 4, and the results of the performance of each model in the news dataset after reduced dimension are shown in Fig. 5.

As can be seen from Fig. 4, the sentence representations in Fig. 4D are clearly optimal compared to Figs. 4A–4C, where the samples of different architecture document are apparently separated from each other. These models, having not been explicitly trained on architecture domain data or having limited training samples in this domain, exhibit poor text representation capabilities. In contrast, we propose a text representation method that combines RoBERTa and BERT. This approach enhances the model’s comprehension of architectural texts and effectively extracts keywords to represent articles. These keywords are subsequently processed with the assistance of GAT. Leveraging the attention mechanism of the nodes in the graph neural network, the model captures relationships between keywords, resulting in an improved representation of the text.

As shown in Fig. 5, we compare the four different model feature representation capabilities in a generic news dataset by t-SNE visualisation. In Figs. 5A–5C, samples from different embedding models are poorly separated, while in Fig. 5D, samples from different news categories are densely clustered together. It shows that the results of feature representation of the model constructed by the method proposed in this article are significantly better than those in Figs. 5A–5C. It also shows that the feature representation capability of the model can effectively learn architectural texts information features with good representability.