RSRNeT: a novel multi-modal network framework for named entity recognition and relation extraction

Multi-modal features representation

Multi-modal learning came into existence in the 1970s and were categorized into four eras: the behavioral era, computational era, interaction era and deep learning era (Baltrusaitis, Ahuja & Morency, 2017). Multi-modal learning has entered the deep learning era. Multi-modal deep learning leads to a wide range of applications from audio-visual speech recognition, multimedia content indexing and retrieval, understanding human multi-modal behaviors, emotion recognition, multi-modal affect recognition, image and video captioning, visual question answering, multimedia retrieval, to health analysis (Jabeen et al., 2023).

In the realm of deep multi-modal learning, five principal technical challenges have surfaced: multi-modal representation, multi-modal translation, multi-modal alignment, multi-modal fusion and multi-modal co-learning (Baltrusaitis, Ahuja & Morency, 2017). In this article, we only focus on deep multi-modal representation and fusion. Deep multi-modal representation involves the process of representing information from various media in tensor or vector form. The primary challenge is how to effectively represent and summarize multi-modal data in a manner that leverages complementarity and redundancy (Jabeen et al., 2023). The method of multi-modal features representation first learn features from each mode, and then integrates them into a single representation to maximize the retention of common and unique features. Visual features are typically extracted through CNNs or attention mechanisms, while text information is obtained through text embedding, RNNs, and BERT variants. Subsequently, neural networks models, such as deep Boltzmann machines(DBM) (Liu et al., 2020a), auto-encoder (Khattar et al., 2019), generative adversarial network (Zhang et al., 2021b), and attention mechanisms (Dai et al., 2021), are employed to fuse different modal features. The outcome is a unified feature representation tailored for specific applications.

Multi-modal features fusion and pre-trained language model

Multi-modal feature fusion aims to integrate information from different modes into a unified feature representation. Methods for multi-modal feature fusion can be broadly categorized into three types: simple operations-based, attention-based, and tensor-based approaches. Here, we specifically focus on attention-based approaches. Attention-based fusion methods can be further classified based on the specific objects of the fusing operator. This classification includes image attention, co-attention of images and text, and attention facilitated by transformers. Image attention methods fuse text features and image features obtained through attention mechanisms. For instance, Zhu et al. (2016); Shih, Singh & Hoiem (2016) and Huijuan & Saenko (2016) explore the incorporation of attention on images using LSTM, RNN, and GRUs models for visual question answering. Co-attention employs symmetric attention structures to generate both attended image feature vectors and attended language vectors (Lu et al., 2016). For example, Gao et al. (2021) introduces an approach that accelerates the convergence of detection transfomer by incorporating spatially modulated co-attention mechanisms. Regarding attention in transformers, recent studies have emphasized intermediate methods that facilitate fusion to occur on pre-trained language models.

In the context of attention in transformers, recent studies have primarily focused on intermediate methods that enable fusion to occur on pre-trained language models. This fusion process can be categorized into single-stream and dual-stream methods based on their interaction mode (Su et al., 2019). In single-stream structures, text and images are combined into a sequence and fed into pre-trained language models, such as BERT, to learn the embedding representation of the context. Representative works in single-stream structures include VL-BERT (Su et al., 2019), VisualBERT (Li et al., 2019), Unicoder-VL (Li et al., 2020a) and UNITER (Chen et al., 2020). On the other hand, the dual-stream method encodes different modalities separately, obtaining distinct single-modal features. These single-modal features are then fused at a later stage to create a multi-modal representation. Representative works in dual-stream structures include ViLBERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019). Single-stream methods boast a simpler structure, making them suitable for resource-constrained environments. However, they may encounter challenges when dealing with diverse information and complex tasks. In contrast, dual-stream methods offer improved capability in handling diverse information but come at the cost of increased complexity and resource requirements. In a certain sense, our approach belongs to the dual-stream structure. Our method initially extracts and represents the visual features using a multi-scale extraction and an adaptive module, and then fuses the text and visual features in a variant of BERT. Our approach has led to achieving SOTA performance on three complex datasets.

Multi-modal information fusion is usually implemented using pre-trained language models. In this context, we introduce some pre-trained models that are relevant to our study. Pre-trained language models have revolutionized natural language processing by learning from large amounts of unlabeled data, acquiring general knowledge that significantly enhances performance in downstream tasks. The advantage of utilizing pre-trained models lies in avoiding the need to train a new model from scratch (Qiu et al., 2020). RoBERTa (Liu et al., 2019) improves upon BERT by implementing several changes. These changes involve adjustments to the masking strategy, eliminating next sentence prediction, optimizing hyperparameters and training on a larger corpus. They lead to better representation learning and higher performance. On the other hand, XLM-RoBERTa (Conneau et al., 2019) leverages a large-scale unlabeled dataset from over 100 languages for pre-training. This approach enables the model to acquire a broader range of language knowledge, leading to improved performance on various multilingual tasks. ALBERT (Lan et al., 2019) focuses on lightweight model design, with the aim of speeding up training and improving performance.

Uni-modal NER and RE methods

A knowledge graph is a semantic graph composed of nodes and edges, representing real-world objects and their relationships (Zhong et al., 2023). The general process of constructing a knowledge graph includes data acquisition, knowledge extraction, knowledge fusion, and knowledge evolution. NER and RE are two essential technologies for acquiring knowledge in KG construction. NER focuses on identifying entities with specific meanings in the text, such as people, places, things, proper nouns, etc. RE involves extracting factual triples from data and establishing connections between different entities in the text. For the NER and RE tasks, approaches are mainly divided into three categories, i.e., rule-based methods, statistic-based methods and deep learning methods. Rule-based NER methods rely on matching patterns using predefined rules and dictionaries to recognize entities or extract relationships. They often require expert knowledge and manual design of rules.

Previously, statistic-based methods, such as the hidden Markov model (HMM) (Zhou & Su, 2002), and the conditional random field (CRF) (Finkel, Grenager & Manning, 2005), were used for NER and RE tasks. These statistic-based approaches are based on probability graph model. They treat NER as a sequential classification tagging task, where entities are tagged according to the BIES (beginning, intermediate, ending, single) scheme along with their respective types (Zhong et al., 2023). For the RE task, these approaches are popular designs for allowing contextual information to flow through free text and identify relationships between entities. Researchers employ probabilistic-graph (Mulwad, Finin & Joshi, 2013), CRF (Chen & Cafarella, 2013) and Markov logic network (Zhu et al., 2009) to identity relationships.

Recently, deep learning has gained popularity in NER and RE tasks. For the NER task, these models aggregate contextual embeddings according to the input and context encoders, then output word type tags through tag decoders such as a CRF structure or a softmax structure (Li et al., 2020b). Collobert et al. (2011) were the first to apply CNNs to NER, enabling the extraction of contextual information from text. Following CNN, IDCNN (Strubell et al., 2017) and other variants were proposed for the NER task. Representative uni-modal models of NER include CNN-BiLSTM-CRF (Ma & Hovy, 2016), HBiLSTM-CRF (Lample et al., 2016) and BERT-CRF. Additionally, PCNN (Zeng et al., 2015) and MTB (Soares et al., 2019) models are RE-oriented pretraining models based on BERT (Devlin et al., 2018). The introduction of Transformer-based models, such as BERT, has revolutionized NER and RE tasks in the context of knowledge graphs. For RE tasks, BiLSTM models (Zhou et al., 2016) with inter-word attention can capture long-distance dependencies between entities for relation extraction. Moreover, GNN-based models, such as EPGNN (Zhao et al., 2019) and AGGCN (Guo, Zhang & Lu, 2019), have shown promising results in relation extraction.

Multi-modal NER and RE methods

With the increasing prevalence of multi-modal data on social media platforms, MNER and MRE have become important areas of research. MNER aims to improve entity identification by incorporating images as auxiliary inputs alongside text (Li et al., 2022). However, the primary challenge in the tasks of MNER and MRE is how to effectively leverage visual information to aid entity recognition and relation extraction while minimizing interference from irrelevant images. Several approaches have been proposed to tackle this challenge. Zhang et al. (2018) introduced a co-attention layer that calculates fusion weights between text and image feature vectors. Moon, Neves & Carvalho (2018) proposed a multi-modal attention mechanism to selectively extract word-level, character-level, and region-level information from Twitter text, addressing grammatical inconsistencies. Yu et al. (2020) pioneered the use of BERT as the text encoder and designed a multi-modal interaction module based on Transformers to model text-image relations, while also addressing bias introduced by irrelevant image features. Zhang et al. (2021a) created graphical connections between textual words and visual objects using a visual grounding toolkit. Yang et al. (2019) proposed a graph fusion method for graph encoding. Xu et al. (2022) introduced a matching and alignment framework to enhance the consistency of representations across different modalities in MNER. Recently, MAF (Xu et al., 2022), MRC-MNER (Jia et al., 2022) and MNER-QG (Jia et al., 2023) use BERT to extract text features and adopt ResNet to extract visual features, respectively. They then perform cross-modal feature matching and fusion before conducting downstream tasks. Specially, Chen et al. (2022) proposed HVPNeT, which performed at a SOTA performance in MNER and MRE. However, the inappropriate design of the backbone network results in insufficient feature extraction from text and images and inadequate multi-modal feature fusion. Thus, HVPNet has room for performance improvement.

Multi-scale visual feature extraction

The sentence-related images encompass both global images and local target images. A global image refers to the entire image, while a local image represents the region of the target object within a global image. The features extracted from the global image serve to enhance each word representation, while the local image provides the targeted object region as a clue, which is more likely to help us identify some words as the correct entity type (Zhang et al., 2021a). By combining both global and local features of respective images, we design a multi-scale visual feature extraction module to capture a more fully visual information. The construction of multi-scale visual feature extraction module is depicted in blue dashed rectangle in Fig. 1. In this module, we use ResNeSt (Zhang et al., 2022) as the backbone network to encode global and local images.

ResNeSt, a powerful convolutional neural network model designed for image classification tasks, is depicted in Fig. 2. The structure of ResNeSt incorporates four key components, including split-attention, squeeze-and-excitation (SE) blocks, bottleneck blocks, and feature reuse. Split-attention enhances the model’s feature perception by learning the correlations between local features. SE blocks dynamically adjust the importance of features across channels, thereby improving feature representation. Bottleneck blocks reduce computational complexity and parameter count. Feature reuse enables the direct utilization of feature information from preceding blocks. In summary, ResNeSt provides advanced feature representation and perception capabilities, capturing image details and key information effectively. There are two key operations, including feature-map group and split attention in cardinal groups.

Feature-map group. The features are divided into several groups, and the number of feature-map groups is determined by a cardinality hyperparameter K. We refer to the resulting feature-map groups cardinal groups. A radix hyperparameter R indicates the number of splits within a cardinal group. Thus, the total number of feature groups is G = KR. A series of transformations $\left\{{\mathcal{F}}_1,{\mathcal{F}}_2,\dots ,{\mathcal{F}}_G\right\}$ is applied to each group, and the representation of each group is $U_i={\mathcal{F}}_i\left(X\right)$, for $i\in \left\{1,2,\dots ,G\right\}$ .

Split attention in cardinal groups. The k-th cardinal group is denoted as ${\hat{U}}^k={\sum }_{j=R\left(k-1\right)+1}^{RK}{U}_j$ , where ${\hat{U}}^k\in {\mathbb{R}}^{H\times W\times \frac{C}{K}}$ for k ∈ 1, 2, …, K. Here, H, W and C are height, width and channels of the block output, respectively. Global contextual information with embedded channel-wise statistics can be gathered through global average pooling across spatial dimensions $s^k\in {\mathbb{R}}^{\frac{C}{K}}$. The c-th component is calculated by (1)${\displaystyle s_c^k=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{\hat{U}}_c^k\left(i,j\right).}$

The weighted fusion of the cardinal group representation $V^k\in {\mathbb{R}}^{H\times W\times \frac{C}{K}}$ is aggregated using channel-wise soft attention. In this process, each feature-map channel is obtained through a weighted combination over splits. The c-th channel is calculated as: (2)${\displaystyle V_c^k=\sum _{i=1}^R{a}_i^k\left(c\right)U_{R\left(k-1\right)+i},}$

where $a_i^k\left(c\right)$ denotes a (soft) assignment weight given by: (3)${\displaystyle a_i^k\left(c\right)=\left\{\begin{array}{c}\hfill \frac{exp\left({\mathcal{G}}_i^c\left(s^k\right)\right)}{\sum _{j=0}^{R}exp\left({\mathcal{G}}_j^c\left(s^k\right)\right)}ifR>1,\hfill \\ \hfill \frac{1}{1+exp\left(-{\mathcal{G}}_i^c\left(s^k\right)\right)}ifR=1,\hfill \end{array}\right.}$

Here, mapping ${\mathcal{G}}_i^c$ determines the weight of each split for the c-th channel based on the global context representation s^k.

We adopt the visual toolkit introduced by Zhang et al. (2021a) to extract the most salient top m local images from each global image. Subsequently, both the global and local images are resized to 224 × 224 pixels and used as inputs to the ResNeSt model. We denoted the global image as $\mathcal{I}$ and the visual objects as $\mathcal{O}=\left\{o_1,o_2,\dots ,o_m\right\}$. Using ResNeSt, we encode the input image and generate a series of image features at different granularities, denoted as $\left\{F_1,F_2,F_3,\dots ,F_c\right\}$. Here, c = 4 represents the number of ResNeSt blocks. After the multi-scale visual feature extraction module, we obtained visual features F₁, F₂, F₃ and F₄ with dimensions size 256 × 56 × 56, 512 × 28 × 28, 1024 × 14 × 14 and 2048 × 7 × 7, respectively. This process enables the extraction of multi-scale image features for subsequent tasks.

Adaptive module

In this module, we refer to the work of HVPNeT. The adaptive module is to generate vectors, which represent the importance of visual features from the module of multi-scale visual feature extraction. These feature vectors serve as inputs to the multi-modal feature fusion module, which are implemented by a variant framework BERT and will be discussed in the next subsection.

The structure of adaptive module is shown in Fig. 3. It consists of two steps, i.e., project processing and adaptive processing. The projection processing takes multi-scale visual features $\left\{F_1,F_2,\dots ,F_c\right\}$ as input, and uses project function $M_{\theta }\left(\cdot \right)$ to transform F_i to features with the same dimension size. The project function $M_{\theta }\left(\cdot \right)$ is expressed by (4)${\displaystyle V_c={Conv}_{1\times 1}\left(F_c\right),}$

and (5)${\displaystyle V_i={Conv}_{1\times 1}\left(Pool\left(F_i\right)\right),i=1,2,\dots ,c-1.}$

where $Pool\left(\cdot \right)$ indicates the pooling operation to aggregate the features to the same spatial size. The Conv_1×1 convolutional layer is used to make the visual features of different granularities match Transformer’s embedding size. The Transformer is a layer in multi-modal fusing module, and will be discussed in next subsection. After project processing, {F₁, F₂, …, F_i} are transform to {V₁, V₂, …, V_c} features. If c = 4, {V₁, V₂, V₃, V₄} are with 256 × 2 × 2, 512 × 2 × 2, 1024 × 2 × 2 and 2048 × 2 × 2 dimension size, respectively.

Next, we discuss the adaptive processing. There are four steps to generate visual vectors. Firstly, we used a learnable neural network to obtain a gate signal, which is expressed by (6)${\displaystyle {\alpha }^{\left(l\right)}=f\left(W_l\left(\frac{1}{c}\sum _{i=1}^{c}P\left(V_i\right)\right)\right),}$ where $P\left(\cdot \right)$ is a global average pooling layer, W_l is the lth MLP layer, and $f\left(\cdot \right)$ is an activation function. It should be noted that our proposed method adopts SeLU activation function, whereas the original method uses Leaky_{_}ReLU. Thus, we obtained gate signals α = {α^l|l ∈ {1, 2, …, L}}. Here, l indicates the lth Transformer layer, and L is the number of Transform layers in the variant of BERT. Secondly, we calculated the probability vector for the lth layer of Transformer by (7)${\displaystyle g^{\left(l\right)}=Softmax\left({\alpha }^{\left(l\right)}\right).}$ Thirdly, based on $g^{\left(l\right)}$, we derived the final aggregated hierarchical visual features $V_{gated}^{\left(l\right)}$, which correspond to the first layer in the lth Transformer layer. The $V_{gated}^{\left(l\right)}$ is expressed as follows (8)${\displaystyle V_{gated}^{\left(l\right)}=g^{\left(l\right)}{V}^{\left(l\right)}.}$ Since visual information includes feature from global images and local target images, the final visual feature corresponding to the l-th Transformer layer was obtained by (9)${\displaystyle {\stackrel{\sim }{V}}_{gated}^{\left(l\right)}=\left[V_{gated}^{\left(l,I\right)};V_{gated}^{\left(l,o_1\right)};\dots ;V_{gated}^{\left(l,o_m\right)}\right],}$ where $V_{gated}^{\left(l,I\right)}$ is the visual feature of global images, and $V_{gated}^{\left(l,o_m\right)}$ are the visual feature of local target images. The ${\stackrel{\sim }{V}}_{gated}^{\left(l\right)}$ is used to enhance the layer-level representations of the textual modality.

Multi-modal feature fusion

In this subsection, we address the challenge selecting or improving the BERT model to enhance the adequacy of multi-modal feature fusion for downstream tasks. Specifically, we employed RoBERTa to fuse multi-modal features for MRE tasks, and we utilized XML-RoBERTa to fuse multi-modal features for MNER tasks.

RoBERTa is a pre-trained language model based on the self-attention mechanism, which is improved and optimized based on the BERT model. In comparison to BERT, RoBERTa exhibits two key differences. Firstly, RoBERTa eliminates the next sentence prediction task present in BERT, retaining only the masked language modeling task. This modification allows it to concentrate more on learning contextual information and enhancing language representation capabilities. Secondly, RoBERTa employs the byte-pair encoding (BPE) tokenization algorithm, whereas BERT uses the wordpiece tokenization algorithm. The BPE tokenization algorithm splits the text into subwords, offering better handling of unknown vocabulary and multilingual scenarios. Similarly, XML-RoBERTa also is a pre-trained language model specifically designed for handling multilingual text. It leverages a large-scale unlabeled dataset from over 100 languages for pre-training, enabling the model to acquire a broader range of language knowledge.

The multi-modal feature fusion processes are similar with the RoBERTa and XML-RoBERTa models. Here, we take the RoBERTa model as an example to present the steps of multi-modal feature fusion. As shown in Fig. 1, the multi-modal feature fusing module has L = 12 Transformer layers, each represented in Fig. 4. For the l-th Transformer layer, the l-th visual gated feature ${\stackrel{\sim }{V}}_{gated}^{\left(l\right)}$ and the output of (l − 1)-th Transformer, denoted by H^(l−1), are used as input. It is should be noted that the first Transformer layer takes textual sequence X as H⁽⁰⁾. We integrate the multi-scale visual feature information as a prefix of textual information at each self-attention layer in the RoBERTa model. Specifically, for l-th Transformer layer, given an input sequence $X=\left\{x_1,x_2,\dots ,x_n\right\}$, the contextual representation H^l−1 ∈ ℝ^n×d is first projected into the query/key/value vector, respectively: (10)${\displaystyle Q^l=H^{l-1}{W}_l^Q,K^l=H^{l-1}{W}_l^K,V^l=H^{l-1}{W}_l^V,}$

where $W_l^Q$, $W_l^K$, and $W_l^V$ are learnable projection matrices. These matrices have the same dimensions as H^l−1. For the aggregated hierarchical visual features ${\stackrel{\sim }{V}}_{gated}^{\left(l\right)}$, in the self-attention module of l-th layer, we used a set of linear transformations $W_l^{\varnothing }\in {\mathbb{R}}^{d\times 2\times d}$ to project them into the same embedding space of textual representation. Next, we defined the visual prompt ${\varnothing }_k^l,{\varnothing }_v^l\in {\mathbb{R}}^{hw\left(m+1\right)\times d}$ operation as: (11)${\displaystyle \left\{{\varnothing }_k^l,{\varnothing }_v^l\right\}={\stackrel{\sim }{V}}_{gated}^{\left(l\right)}{W}_l^{\varnothing }.}$

Here, hw(m + 1) denotes the length of the visual sequence, and m denotes the number of visual objects detected by the target detection algorithm. Formally, the attention based on visual prefixes was calculated as follows: (12)${\displaystyle {Prefix\text{\_}Attention}^l=softmax\left(\frac{Q^l{\left[{\varnothing }_k^l;K^l\right]}^T}{\sqrt{d}}\right)\left[{\varnothing }_v^l;V^l\right],}$

where $\left[{\varnothing }_k^l;K^l\right]$ denotes the concatenation of ${\varnothing }_k^l$ and K^l, $\left[{\varnothing }_v^l;V^l\right]$ denotes the concatenation of ${\varnothing }_v^l$ and V^l.

Classifier

Through the variant of BERT, we obtained the final representation of multi-modal feature, $H^L=U\left(X,{\stackrel{\sim }{V}}_{gated}^{\left(l\right)}\right)$. Here, U(⋅) denotes the visual prefix-based attention operation. Finally, we used different classifier layers for the NER and RE tasks, respectively.

Named entity recognition. We used the CRF decoder for the NER task, which is similar to the method proposed by Moon, Neves & Carvalho (2018). We provide the final hidden vector of BERT H^L to the CRF model. For a sequence of labels $\mathbf{y}=\left\{y_1,\dots ,y_n\right\}$ , the probability of y and the objective of NER are defined as follows: (13)${\displaystyle p\left(y|H^L\right)=\frac{\prod _{i=1}^n{S}_i\left(y_{i-1},y_i,H^L\right)}{\sum _{y^{\prime }\in Y}\prod _{i=1}^n{S}_i\left(y_{i-1}^{{}^{\prime }},y_i^{{}^{\prime }},H^L\right)},}$ (14)${\displaystyle {\mathcal{L}}_{ner}=-\sum _{i=1}^{M}log\left(p\left(y^{\left(i\right)}|U\left(X^{\left(i\right)},{\stackrel{\sim }{V}}_{gated}\right)\right)\right).}$

where Y denotes the predefined set of tags with BIO (B-begin, I-inside, O-outside) tagging schema, and S(⋅) denotes the potential functions (Lample et al., 2016).

Relation extraction Assume an RE dataset represent as ${\mathcal{D}}_{re}={\left\{\left(X^{\left(i\right)},r^{\left(i\right)}\right)\right\}}_{i=1}^M$, the objective of RE is to predict the relation $r\in \mathcal{Y}$ between subject entities and object entities. A [CLS] head is employed to compute the probability distribution over the class set $\mathcal{Y}$ using the softmax function $p\left(r|X\right)=\text{Softmax}\left({\left[\mathbf{W}H\right]}_{\left[CLS\right]}\mathbf{L}\right)$. Refer to (Chen et al., 2022), the parameters of L and W are fine-tuned by minimizing the cross-entropy loss function ${\mathcal{L}}_{re}$ over $p\left(r|X\right)$ on the entire $\mathcal{X}$. (15)${\displaystyle {\mathcal{L}}_{re}=-\sum _{i=1}^{M}log\left(p\left(r^{\left(i\right)}|U\left(X^{\left(i\right)},{\stackrel{\sim }{V}}_{gated}\right)\right)\right).}$

Datasets

We utilized Twitter-2015 (Zhang et al., 2018) and Twitter-2017 (Lu et al., 2018) datasets for the task of MNER, and MNRE (Zheng et al., 2021b) datesets for the task of MRE. The details of Twitter-2015 and Twitter-2017 can be found in Table 1. And Table 2 provides the details of the MNRE dataset.

Experimental setup

The proposed approach was implemented using PyTorch 1.8.1. The training and testing of the network models were conducted on a computer server, which equipped with an Intel Xeon Gold 4260 CPU, 4 T V100 GPUs, 96 GB of DDR4 memory and CentOS 7.9 operating system. For optimization, the AdamW (Loshchilov & Hutter, 2017) optimizer was employed. The learning rate was gradually increased to a maximum value during the first 10% of the gradient update using linear warm-up, and then it was linearly decayed for the remainder of the training. The weight decay for all non-biased parameters was set to 0.01. Additionally, the number of image objects used was set to 3.

For the MNER task, we set the batch size to 16, and conducted experiments with different learning rate intervals [1e−5, 3e−5]. The training process was carried out over 30 epochs, with validation performed after the 16th epoch. Regarding the MRE task, we employed a batch size of 16 with a fixed learning rate 1e−5. The model was trained for 15 epochs, and validated after the 8 th epoch. In both tasks, we selected the model that exhibited the best performance on the validation set and the test set.

Baseline models

To demonstrate the superiority of our approach, we conducted a comprehensive comparison between our model and several baseline models . This comparison involved evaluating our model against three groups of models: text-based models, the previous SOTA models of MNER and MRE, and other features fusion models.

• Text-based models: We first consider a representative set of text-based models, including CNN-BiLSTM-CRF (Ma & Hovy, 2016), HBiLSTM-CRF (Lample et al., 2016) and BERT-CRF for NER. PCNN (Zeng et al., 2015) and MTB (Soares et al., 2019) models are for RE task. MTB is an RE-oriented pretraining model based on BERT.

• Previous SOTA models: We also considered another set of SOTA multi-modal methods previously used for MNER and MRE tasks. For the MNER task, we examined the latest SOTA models, including AdapCoAtt-BERT-CRF (Zhang et al., 2018), OCSGA (Wu et al., 2020), UMT (Yu et al., 2020), and UMGF (Zhang et al., 2021a). UMGF proposes a unified multi-modal graph fusion approach. For the MRE task, we explored the latest SOTA models, such as BERT+SG (Soares et al., 2019), and MEGA (Zheng et al., 2021a). BERT+SG connects the textual representation of BERT with visual features generated by the scene graph (SG) tool (Tang et al., 2020), while MEGA proposes a dual graph for multi-modal alignment to capture correlations between entities and objects for better performance. Additionally, we considered two other SOTA models, VisualBERT (Li et al., 2019) and HVPNet (Chen et al., 2022). Unlike the aforementioned SOTA approaches which primarily rely on co-attention, VisualBERT is a single-stream structure, serving as a strong baseline for comparison.

• Other features fusion models: Further, we conducted comparisons with recent methods such as MAF (Xu et al., 2022), MRC-MNER (Jia et al., 2022) and MNER-QG (Jia et al., 2023). The three approaches utilize BERT for extracting text features and ResNet for extracting visual features. Subsequently, they employ cross-modal feature matching and fusion before proceeding to downstream tasks. In comparison to MAF, MRC-MNER and MNER-QG methods incorporate queries from machine reading comprehension to retrieve relevant visual information in linguistic contexts. It is worth noting that, among these models, MNER-QG currently represents the SOTA model for MNER, while HVPNeT holds the position of the latest SOTA model for MRE.

Experimental results

In this subsection, we conduct a comprehensive comparison of our proposed method with various baseline approaches. Firstly, we present the performance comparison between RSRNeT and baseline models. Secondly, we present the results of our ablation study. Thirdly, we compare the training time of our approach with that of other methods. Fourthly, we present the results obtained under the low-resource scenario. Finally, we provide the comparison results of case study.

Low-resource scenario

To evaluate the performance of our method in a low-resource setting, we conducted experiments by randomly sampling 5% to 50% of the data from the original training set, thereby creating low-resource training sets. The results of these experiments are presented in Fig. 5, offering a performance comparison between our method and several baseline methods in the low-resource scenario. The analysis of these results yields the following observations: firstly, in the low-resource scenario, our method, HVPNet, UMT, and MEGA consistently outperformed BERT-CRF. This demonstrates that incorporating multi-scale visual features remains beneficial for NER and RE tasks even with limited resources. Secondly, from the Fig. 5, RSRNeT exhibits superior performance compared to other baselines, further confirming the effectiveness and data efficiency of our proposed method.

Case study

To validate the effectiveness and robustness of our method, we conducted case studies, as shown in Table 5. We present results from two groups of experiments. For each group, we selected different image-text pairs with varying relevance from the MNRE dataset (Zheng et al., 2021b), as shown in the first two rows in a group. The subsequent rows indicate the entity types of the selected text and the relationship labels between entities. The last rows display the predicted entity types and relationships between entities for these test samples. We conduct this experiment using various methods, including BERT (Devlin et al., 2018), VisualBERT (Li et al., 2019), MEGA (Zheng et al., 2021a), HVPNeT (Chen et al., 2022) and our proposed RSRNet.

Table 5:

Case analysis.

Relevant image-text pair	Weak relevant image-text pair	Irrelevant image-text pair
The winter scenery of JinshanlingGreat WallHebei Province of China	Baby female Dachshund Beagle mix in Redondo BeachCA	Bill Cosby’sHollywoodWalk of Fame vandalized
Gold Relations: loc/loc/contain	/loc/loc/contain	per/misc/present_in
BERT:          loc/loc/contain ✓	per/per/present_in     ✗	misc/org/present_in ✗
VisualBERT:  loc/per/contain ✗	loc/loc/contain          ✓	per/loc /present_in   ✗
MEGA:           loc/loc/contain ✓	misc/per/member_of ✗	loc/loc/contain        ✗
HVPNeT:       loc/loc/contain ✓	per/org/member_of   ✗	loc/loc/present_in    ✗
RSRNeT:       loc/loc/contain ✓	loc/loc/contain         ✓	per/misc/present_in ✓
Best Day Trips from Zagreb Tour Croatia	Commercial in US MichiganBelleville MondProp	Come and see Billo at Clerkenwell Design Week
Gold Relations:/loc/loc/contain	/loc/loc/contain	/per/misc/present_in
BERT:         loc/per/member_of ✗	loc/per/present_in ✗	per/org/member_of ✗
VisualBERT:  loc/loc/contain ✓	per/per/peer          ✗	org/misc /contain   ✗
MEGA:          loc/loc/contain ✓	loc/per/present_in ✗	loc/per/contain       ✗
HVPNeT:       loc/loc/contain ✓	loc/loc/contain     ✓	per/misc/present_in ✓
RSRNeT:       loc/loc/contain ✓	loc/loc/contain     ✓	per/misc/present_in ✓

DOI: 10.7717/peerjcs.1856/table-5

From the Table 5, it can be observed that in scenarios with high semantic correlation between image and text, most methods successfully identified the entity types and relationships, which between entities in image-text pairs. In scenarios with weak semantic correlation between image and text, only our method RSRNeT and a few others can successfully capture more features and accurately predict entity types and relationships. Specially, in scenarios with irrelevant image-text pair, our method RSRNeT continues to predict all cases correctly, whereas the other four methods almost failed. These experimental results clearly demonstrate that our method performs well in multiple case studies, indicating that our model effectively alleviates the negative impact of irrelevant image pairs on entity and relationship recognition. This showcases the robustness and effectiveness of our approach.