Local-global multi-scale attention network for medical image segmentation

Traditional medical image segmentation methods

Traditional medical image segmentation methods include edge-based methods (Ding & Goshtasby, 2001), threshold-based methods (Dong, 2014), region-based methods (Lalaoui & Mohamadi, 2013; Freixenet et al., 2002), feature-based methods (Likas, Vlassis & Verbeek, 2003), and active contour-based algorithms (Park, Schoepflin & Kim, 2001). Edge-based methods identify image boundaries by detecting changes in image gradients, but they are susceptible to noise and complex backgrounds. Threshold-based methods perform image segmentation by utilizing gray-level histograms to separate the foreground from the background, but they perform poorly on images with complex gray-level distributions. Region-based methods segment images by merging or splitting similar pixels, yet they are highly sensitive to noise. Feature-based methods perform segmentation using texture, color, and other features, but they heavily rely on handcrafted feature design and the quality of training data. Active contour-based methods are capable of handling complex shapes, but they typically involve high computational costs. Overall, these methods are relatively inefficient and lack stability when handling complex data. In recent years, deep learning-based approaches have become a major research focus (Ottesen et al., 2023; Umirzakova et al., 2024).

Deep learning-based medical image segmentation methods

With the rapid development of deep learning technology, an increasing number of researchers have begun to adopt deep learning methods to tackle medical image segmentation tasks. This section will introduce medical image segmentation methods based on convolutional neural networks and those based on attention mechanisms, respectively.

Issues that need to be addressed

Despite significant advancements in existing medical image segmentation methods, several issues still remain:

• 1.

  Current deep learning based medical image segmentation methods still face the challenge of insufficient extraction of both local and global information. This limitation prevents the networks from fully capturing the detailed features of the images, ultimately affecting the accuracy of the segmentation results.

• 2.

  Many existing methods struggle with inaccuracies in selecting core image features, failing to effectively identify and emphasize the most critical regions of the image. As a result, this undermines the model’s learning capacity and reduces the precision of the segmentation outcomes.

To address these challenges, we propose a novel medical image segmentation method-local-global multi-scale attention network (LGMANet). Compared to traditional medical image segmentation methods, LGMANet effectively overcomes issues such as reliance on handcrafted feature extraction, susceptibility to noise, and reduced accuracy when handling images with complex backgrounds. In contrast to existing deep learning based medical image segmentation methods (Umirzakova et al., 2024), LGMANet integrates the local-global information processing blocks (LGIPBs) and the efficient multi-scale reconstruction attention (EMRA) modules, enabling the deep mining of both local and global information from images and more precise selection of core features. This effectively addresses the challenges of insufficient local-global information extraction and inaccurate core feature selection in existing methods.

Overall structure

To address the main challenges in current medical image segmentation tasks, such as the insufficient extraction of both local and global information from images and the inaccurate selection of core image features, we designed the LGMANet. The specific structure of LGMANet is illustrated in Fig. 1. Inspired by the U-Net model (Ronneberger, Fischer & Brox, 2015), LGMANet adopts an effective skip connection structure and a five-layer encoder-decoder framework as its basic architecture. In the encoder, the first three stages of LGMANet each consist of two innovative LGIPBs, replacing the standard convolutional layers, thereby deeply mining and extracting both local and global information during the image downsampling process. The subsequent two stages employ shifted MLP modules to better capture complex feature relationships, further enhancing the overall expressive power and generalization capabilities. In the decoder, the first stage employs the shifted MLP module, the second stage integrates the shifted MLP with the EMRA module, and the remaining three stages combine the EMRA module with convolutional layers. This decoder structure accurately extracts core features, effectively suppresses irrelevant information, and has strong capabilities for learning multi-scale image information.

We first briefly introduce the shifted MLP, which is a key component of our network architecture. Given the remarkable performance of shifted MLP (Liu et al., 2021) in medical image segmentation and related tasks, this study adopts its architecture to enhance the network’s ability to model both local and global feature dependencies. Shifted MLP offers several significant advantages, including its dynamic shifting mechanism, which captures spatial dependencies across neighboring regions, the integration of depthwise convolution to further enhance local feature extraction, and its efficient modeling of global features through linear transformations. This combination achieves a balance between computational efficiency and representational power. The shifted MLP module consists of several key components. First, a linear layer projects the input features into a higher-dimensional feature space. Next, a dynamic shifting operation, combined with depthwise convolution, captures both local and long-range feature dependencies. Finally, another linear layer maps the features back to their original dimensionality. The module also incorporates activation functions (e.g., GELU) and dropout mechanisms to enhance non-linear expressiveness and mitigate overfitting. This module was first proposed by Liu et al. (2021), with further implementation details available in their original work.

Local-global information processing block

To address the challenge of insufficient extraction of both local and global information in medical images, this article introduces a LGIPB, whose detailed structure is shown in Fig. 2. It is important to note that when LGIPB is positioned at the beginning of the network encoder, its input is a pathological image; otherwise, the input is a feature map. Additionally, the output of LGIPB is always a feature map. Specifically, the input feature map (or pathological image) is first processed by a $3\times 3$ convolutional layer to preliminarily extract image information. Subsequently, the processed feature map undergoes three $3\times 3$ convolutional layers with dense connectivity structures, each equipped with batch normalization and Prelu activation functions, to effectively fuse information from different levels as well as local and global features. Finally, the output feature map is processed by another $3\times 3$ convolutional layer with batch normalization and Prelu activation functions to adjust the number of channels back to that of the input channels. Parametric Rectified Linear Unit (PReLU) is a nonlinear activation function whose slope in the negative axis is determined by a parameter that is automatically learned during training through backpropagation. This parameter is initialized to a small positive value (this model adopts the widely used default value of 0.25) and is continuously adjusted during training. The purpose of this parameter is to prevent the “dying neuron” problem and to make the model’s response in the negative region more flexible, thereby enhancing the model’s representation capability and generalization performance.

This module effectively fuses feature information from various levels by progressively increasing the number of channels and concatenating features, thereby enhancing the model’s ability to represent complex features and capture both local and global information. Additionally, the dense connectivity structure within the module promotes feature reuse and information flow, alleviates the vanishing gradient problem, and improves the model’s training efficiency and overall performance. This design not only enhances the model’s performance in medical image segmentation tasks but also strengthens its ability to capture multiscale features, enabling the model to better handle medical images with complex structures and diverse characteristics.

To facilitate readers’ understanding, the pseudocode for the LGIPB is provided in Algorithm 1. In the pseudocode, $cat(\cdot )$ represents the feature concatenation operation, which concatenates multiple feature maps along the channel dimension; $Con{v}_{3\times 3}(\cdot )$ denotes a $3\times 3$ convolution operation; $BatchNorm(\cdot )$ indicates batch normalization, which standardizes the feature maps to accelerate model convergence and enhance stability; and $Prelu(\cdot )$ refers to a parametric rectified linear activation function that introduces nonlinearity.

For a more rigorous mathematical perspective on the detailed structure of the LGIPB and to more clearly illustrate its internal mechanisms, we also present a set of equations describing the LGIPB. Let the input feature map be $x_{in}\in {\mathbb{R}}^{B\times C_{\mathrm{i}\mathrm{n}}\times H\times W}$ and the output feature map be $x_{out}\in {\mathbb{R}}^{B\times C_{\mathrm{o}\mathrm{u}\mathrm{t}}\times H\times W}$. Here, B represents the batch size, $C_{\mathrm{i}\mathrm{n}}$ represents the number of input channels, $C_{\mathrm{o}\mathrm{u}\mathrm{t}}$ represents the number of output channels, H represents the image height, and W represents the image width. The main operations are described by Eqs. (1)–(5):

(1) $$x_1=Con{v}_{3\times 3}(x_{in}),$$

(2) $$x_2=Prelu(BatchNorm(Con{v}_{3\times 3}(x_1))),$$

(3) $$x_3=Prelu(BatchNorm(Con{v}_{3\times 3}(cat(x_1,x_2)))),$$

(4) $$x_4=Prelu(BatchNorm(Con{v}_{3\times 3}(cat(x_1,x_2,x_3)))),$$

(5) $$x_{out}=Prelu(BatchNorm(Con{v}_{3\times 3}(x_4))).$$

Here, the meanings of $cat(\cdot )$, $Con{v}_{3\times 3}(\cdot )$, $BatchNorm(\cdot )$, and $Prelu(\cdot )$ remain the same as in Algorithm 1. Specifically, in Eq. (1), a $3\times 3$ convolution maps the input $x_{in}$ from $C_{\mathrm{i}\mathrm{n}}$ channels to 12 channels, producing $x_1$. Next, in Eq. (2), a second $3\times 3$ convolution followed by batch normalization and Prelu activation yields $x_2$ (12 channels). In Eq. (3), $x_1$ and $x_2$ are concatenated and fed into another $3\times 3$ convolution, batch normalization, and Prelu, resulting in $x_3$ (24 channels). Then, in Eq. (4), $x_1,x_2,x_3$ are concatenated and passed through a $3\times 3$ convolution, batch normalization, and Prelu, generating $x_4$ (48 channels). Finally, in Eq. (5), a $3\times 3$ convolution maps 48 channels to $C_{\mathrm{o}\mathrm{u}\mathrm{t}}$, followed by batch normalization and Prelu, yielding the output $x_{out}$.

Efficient multi-scale reconstruction attention

To address the issue of insufficient precision in selecting core features of medical images, we have designed an EMRA module. This module is inspired by ECANet (Wang et al., 2020), and its detailed structure is shown in Fig. 3.

Specifically, in the EMRA module, the input feature map is first processed using an attention mechanism. The attention mechanism starts with global average pooling to capture the global statistical features of each channel. Next, a one-dimensional convolution with an adaptive kernel size, followed by a sigmoid activation function, computes the channel attention weights. These weights represent the importance of each channel: higher weights correspond to features that contribute more to the final task, while lower weights indicate redundant or irrelevant features. Finally, the attention weights are element-wise multiplied with the original input, suppressing redundant or irrelevant features and amplifying the critical ones. Next, the weighted feature map is fed into multi-scale convolutional layers with varying kernel sizes (1 $\times$ 1, 3 $\times$ 3, 5 $\times$ 5, and 7 $\times$ 7) to capture features across different receptive fields. These features are then concatenated along the channel dimension to enhance their representational capacity. Finally, a 1 $\times$ 1 convolution compresses the concatenated feature map back to the target number of channels, producing the final output feature map. Through this process, the EMRA module effectively extracts multi-scale information from the image, highlighting key features while suppressing redundant or noisy information, thereby improving the model’s adaptability to complex image patterns. By integrating the attention mechanism with multi-scale feature fusion, the network achieves more accurate selection of important features in medical images, significantly enhancing its robustness and representational power.

To better illustrate its functionality, the pseudocode of the EMRA module is provided in Algorithm 2. In this pseudocode, $AvgPool(\cdot )$ denotes the global average pooling operation, which generates channel descriptors, while $Reshape(\cdot ,[B,1,C])$ and $Reshape(\cdot ,[B,C,1,1])$ represent tensor reshaping operations to adjust the feature map to the required shapes for subsequent processing. $Con{v}_{1D}(\cdot )$ indicates the 1D convolution operation used to model interactions between channels, $\odot$ denotes element-wise multiplication, and $Sigmoid(\cdot )$ represents the Sigmoid activation function for normalizing attention weights. The $Expand(\cdot )$ operation extends the attention weights to match the size of the input feature map for element-wise weighting. Additionally, $Con{v}_{1\times 1}(\cdot )$, $Con{v}_{3\times 3}(\cdot )$, $Con{v}_{5\times 5}(\cdot )$, and $Con{v}_{7\times 7}(\cdot )$ correspond to convolution operations with kernel sizes of $1\times 1$, $3\times 3$, $5\times 5$, and $7\times 7$, respectively, to extract features at different receptive fields. Finally, $cat(\cdot )$ denotes the feature concatenation operation, which integrates multi-scale features along the channel dimension into a unified representation.

For a more rigorous mathematical perspective on the detailed structure of the EMRA module and to clearly illustrate its internal mechanisms, we provide a detailed mathematical description. Let $c$ denote the number of channels in the input tensor, and let $\gamma$ and $b$ be tunable parameters. In the implementation, the 1D convolution kernel size $k$ is first determined by Eq. (6).

(6) $$t=\lfloor \frac{|{\log }_2(c)+b|}{\gamma }\rfloor ⟹k=\{\begin{array}{cc}t,\hfill & ift\mathrm{m}\mathrm{o}\mathrm{d}2=1,\\ t+1,& ift\mathrm{m}\mathrm{o}\mathrm{d}2=0,\end{array}$$where ${\log }_2(\cdot )$ denotes the base-2 logarithm, and $\lfloor \cdot \rfloor$ is the floor operation. Ensuring $k$ is odd helps maintain a symmetric channel interaction for the 1D convolution. In the EMRA module, let the input feature map be $x_{in}\in {\mathbb{R}}^{B\times C_{in}\times H\times W}$ and the output feature map be $x_{out}\in {\mathbb{R}}^{B\times C_{out}\times H\times W}$. Here, B represents the batch size, $C_{\mathrm{i}\mathrm{n}}$ represents the number of input channels, $C_{\mathrm{o}\mathrm{u}\mathrm{t}}$ represents the number of output channels, H represents the image height, and W represents the image width. The main operations are described by Eqs. (7)–(9):

(7) $$\begin{array}{rl}{x}_{weighted}=& x_{in}\odot Expand(Reshape(Sigmoid(Conv1D_k(Reshape(AvgPool\\ & (x_{in}),[B,1,C]))),[B,C,1,1])),\end{array}$$

(8) $$\begin{array}{rl}{f}_1& =Con{v}_{1\times 1}(x_{weighted}),f_3=Con{v}_{3\times 3}(x_{weighted}),f_5=Con{v}_{5\times 5}(x_{weighted}),\\ f_7& =Con{v}_{7\times 7}(x_{weighted})\end{array}$$

(9) $$x_{out}=Con{v}_{1\times 1}(cat(f_1,f_3,f_5,f_7)).$$where $Conv1D_k$ denotes a 1D convolution with kernel size $k$ (from Eq. (6)), the meanings of $AvgPool(\cdot )$, $Reshape(\cdot ,[B,1,C])$, $Reshape(\cdot ,[B,C,1,1])$, $\odot$, $Sigmoid(\cdot )$, $Expand(\cdot )$, $Con{v}_{1\times 1}(\cdot )$, $Con{v}_{3\times 3}(\cdot )$, $Con{v}_{5\times 5}(\cdot )$, $Con{v}_{7\times 7}(\cdot )$ and $cat(\cdot )$ remain the same as in Algorithm 2. Specifically, in Eq. (7), the attention mechanism begins by applying global average pooling to $x_{in}$, reshaping the result, processing it with a 1D convolution of kernel size $k$, and applying a Sigmoid activation. The generated attention weights are then used to reweight the input feature map, yielding the weighted feature map $x_{\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\mathrm{e}\mathrm{d}}\in {\mathbb{R}}^{B\times C_{in}\times H\times W}$. At this stage, the channel dimension remains $C_{in}$.

Subsequently, in Eq. (8), $x_{\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\mathrm{e}\mathrm{d}}$ is fed into four parallel convolutions with kernel sizes $1\times 1$, $3\times 3$, $5\times 5$, and $7\times 7$, producing multi-scale feature maps $f_1,f_3,f_5,f_7\in {\mathbb{R}}^{B\times C_{in}\times H\times W}$. Here, B represents the batch size, $C_{\mathrm{i}\mathrm{n}}$ represents the number of input channels, H represents the image height, and W represents the image width. Each $f_i$ preserves the channel count $C_{in}$. Finally, in Eq. (9), these feature maps are concatenated along the channel dimension, resulting in a feature map $f_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}}\in {\mathbb{R}}^{B\times 4C_{in}\times H\times W}$, and passed through a $1\times 1$ convolution to reduce the channels to $C_{out}$, producing the output $x_{out}\in {\mathbb{R}}^{B\times C_{out}\times H\times W}$.

Dice loss

In segmentation tasks, the imbalance between background and foreground pixels is a common challenge. To address this issue, the Dice coefficient loss (DL) is widely used, as it effectively alleviates the impact of pixel class imbalance and significantly improves segmentation performance. Therefore, this study also employs this loss function.

The DL modifies the segmentation evaluation metric, the Dice similarity coefficient (DSC), to optimize the consistency between predicted samples and ground truth annotations. It has demonstrated superior performance in segmentation tasks. The formula is shown in Eq. (10):

(10) $$DL(p,g)=1-\frac{2{\sum }_{i=1}^N{p}_i{g}_i+\epsilon }{{\sum }_{i=1}^N{p}_i^2+{\sum }_{i=1}^N{g}_i^2+\epsilon }$$where $\epsilon$, ranging from $[0,1]$, is a tunable parameter designed to prevent division by zero errors and improve gradient propagation for negative samples.

It is worth noting that, in this study, we employ a single loss function only at the final output stage, rather than assigning and weighting separate loss functions at each upsampling stage, as is typical in U-Net architectures. The main reasons for this choice are as follows: on the one hand, the introduction of the LGIPBs and the EMRA modules in our model (compared to a standard U-Net design) increases network complexity to some extent. If we also assigned and weighted multiple loss functions at each upsampling stage, the model structure would become excessively complicated, slowing down training and potentially leading to instability. In contrast, using a single loss function at the final output stage allows the model to focus on the quality of the final segmentation result, avoiding interference from intermediate feature map optimization. This approach reduces the competition among gradients at different resolutions and stabilizes the training process. On the other hand, our experimental results show that integrating the LGIPBs and the EMRA modules significantly enhances the network’s learning ability, making it feasible to achieve favorable performance with just a single loss function at the final output stage.

Experimental datasets

This study evaluates the efficacy of LGMANet using four widely used, publicly available biomedical imaging datasets: GlaS (Sirinukunwattana et al., 2017), ISIC2018 (Codella et al., 2019; Tschandl, Rosendahl & Kittler, 2018) BUSI (Al-Dhabyani et al., 2020), and CVC-ClinicDB (Bernal et al., 2015). These datasets include both the original images and their corresponding mask images, and they are commonly employed in medical image segmentation tasks. It should be noted that the annotations in these datasets focus primarily on lesion areas, without separate labeling of normal tissues, because the main goal of medical image segmentation is to assist clinical diagnosis and treatment by accurately identifying the location and shape of lesion regions, while normal tissues are usually regarded as background.

For the ISIC2018 dataset (Codella et al., 2019; Tschandl, Rosendahl & Kittler, 2018), the images are resized to $512\times 512$ for both training and testing. For the remaining three datasets, the images are resized to $384\times 384$. The training-testing ratio follows standard practices in medical image segmentation, with 80% of the images allocated for training and the remaining 20% for model evaluation. Prior to being input into the model, the training data undergoe random rotations and random horizontal or vertical flips, along with normalization, to enhance data diversity and prevent instability due to varying input feature scales. More detailed information about the datasets can be found in Table 1.

Implementation details

All experiments in this article were conducted on an NVIDIA RTX 3060 GPU with 12GB of VRAM. The total number of training epochs was set to 400, with a batch size of 8. During training, the model employed the Adam optimizer and utilized a cosine annealing strategy for learning rate decay. The initial learning rate was set to 0.0001.

Experimental results

This subsection presents the numerical results of various methods evaluated on the four datasets used in this study. The evaluation metrics include Dice coefficient, intersection over union (IoU), recall, and precision, and the image outputs generated by each method are also compared. In addition, statistical significance tests were conducted between our method and several selected methods. To ensure fairness and accuracy, we reproduced these methods on the datasets used in this study based on open-source code, keeping the experimental parameters consistent with those used in the original articles.

Qualitative results

The qualitative results of LGMANet and other methods are shown in Figs. 4–7. For the ISIC2018 dataset (Codella et al., 2019; Tschandl, Rosendahl & Kittler, 2018), five images were selected (each row represents a different image) for display, comparing them with current state-of-the-art methods. For the other three datasets, three images were selected (each row represents a different image) and compared with existing methods. In all displayed images, the first column on the left shows the original medical image input to the model, the second column presents the corresponding mask label provided in the dataset, and from the third column onward, each column shows the segmentation results of a different method. It can be observed that LGMANet achieves superior segmentation results compared to current state-of-the-art methods across most of the four datasets.

In addition to presenting conventional qualitative results, this article also demonstrates the effectiveness of LGMANet on specialized data. We showcase the segmentation results of various models on data containing noise interference, unclear boundaries, and overlapping regions, selected from four datasets, as shown in Figs. 8–11. Specifically, three images from each dataset were selected for display (each row represents a different image). In all the displayed images, the first column on the left shows the original medical image input to the model, the second column presents the corresponding mask label provided in the dataset, and from the third column onward, each column shows the segmentation results of a different method. Figure 8 shows images from the ISIC2018 dataset with noise interference (including hair and patches). Figures 9 and 10 display images with unclear boundaries from the CVC-ClinicDB and BUSI datasets, respectively. Figure 11 presents images from the GlaS dataset, featuring overlapping regions and unclear boundaries.

Ablation study

Qualitative ablation study

In addition to the quantitative results, we also provide qualitative results to visualize the outcomes of the ablation study, thereby offering a clearer validation of the effectiveness of the proposed modules. As shown in Fig. 12, ablating any of the proposed innovative modules leads to a significant decline in segmentation performance across images from all four datasets. This clearly demonstrates the effectiveness of our proposed modules.