Research on building extraction from remote sensing imagery using efficient lightweight residual network

Architecture

This study aimed to construct a novel deep learning network for building extraction, to achieve a better tradeoff between accuracy and efficiency. The proposed efficient lightweight building extraction network, referred to as ELRNet, uses an encoder–decoder structure with 23 layers. A three-band image is fed into two consecutive downsampling blocks to enlarge the feature extraction receptive field and reduce the computational complexity, as shown in Fig. 1. Subsequently, the extracted feature maps are passed through five consecutive LFEMs for further feature extraction. After the feature maps go through a downsampling block and eight LFEMs, feature maps containing rich information are output, which finalises the encoder. To fuse the features between different feature maps and match the resolution of the input image, the extracted feature maps are decoded by passing them through three deconvolution blocks and four LFEMs. Finally, a building extraction image containing pixel-level classification is obtained.

The proposed architecture is detailed in Table 1. The entire framework is that of an encoder–decoder architecture. Taking the input of a three-channel 512 × 512 image as an example, after 16 layers of encoder phase, 128-channel 64 × 64 images are obtained as output feature maps. Following decoder upsampling and fusion stages, these feature maps are finally transformed into a two-channel (i.e., two classes) 512 × 512 image of the same size as the input image. The encoder phase consists of 1–16 layers, and contains three downsampling blocks and 13 LFEMs. Layers 17–23 comprise the decoder, which includes upsampling blocks and LFEMs to further fuse the features between different layers.

In the encoder phase, the feature extraction of VHR remote sensing images of the original size requires the calculation of a large number of convolution operations, which incurs a high computational load. The downsampling block can be leveraged in the first few layers of a deep learning network to reduce the number of calculation units. Moreover, considering that downsampling processes reduce the resolution of the feature map, which results in the loss of important details, this study focuses on minimising the use of downsampling. Hence, only the downsampling blocks at layers 1, 2, and 8 are considered, which can alleviate the computational load and ensure the effectiveness of information extraction. In the remaining layers of the encoder, the proposed LFEM is added to perform key feature information extraction with fewer parameters.

The decoder further fuses the features between different layers to match the resolution of the input image. Deconvolution layers with stride two were used for upsampling at layers 17, 20, and 23. Deconvolution does not require sharing of the pooling indexes of the encoder, thus reducing memory and computation requirements. In addition, to enhance feature recovery, LFEM was applied to layers 18–19 and 21–22. The encoder–decoder architecture renders the proposed network lightweight. The downsampling block and proposed LFEM are detailed below.

Downsampling block

The downsampled images have larger receptive fields and more contextual information. For classification tasks, not only the colour and texture of buildings, but also the scenes in which they appear are important. In addition, downsampling can narrow the resolution of the image to reduce computational complexity.

Therefore, a novel downsampling block with two branches was used, as shown in Fig. 2. The branch of max pooling with stride two can compress redundant spatial information in the image into a more effective representation, thereby improving the feature extraction results. In addition, the loss of important feature information caused by premature downsampling was compensated by using ENet (Paszke et al., 2016), whereby a convolution was performed with stride two in parallel with max pooling. Finally, the convolution and pooling results were concatenated to obtain richer feature outputs. Accordingly, it can be deduced that this downsampling block can extract effective feature information and plays an essential role in improving the accuracy and efficiency of the final extraction results.

Lightweight feature extraction module (LFEM)

An efficient feature extraction module is crucial for improving network performance. This study focused on designing a lightweight feature extraction module for more effective feature extraction with lower computational complexity. The bottleneck and non-bottleneck structures proposed in Ji, Wei & Lu (2019) are classic feature extraction modules that are widely used in image classification tasks. The two structures are shown in Figs. 3A and 3B. In the third layer of the network, taking the feature map with input and output sizes of 128 and 64 channels as an example, the number of parameters in the bottleneck structure is 53,248, and that of the non-bottleneck structure is 147,456, which is greater by almost a factor of 2.77. The two structures exhibit almost the same precision. However, the bottleneck structure requires fewer computing resources (Romera et al., 2017; Lin et al., 2019). Therefore, the structure of the bottleneck is consistent with the design of the lightweight module and was used as a benchmark for the proposed feature extraction module.

The following design process considers the problem of insufficient feature extraction by the bottleneck structures. Therefore, first a bottleneck model with standard convolution (BM-SC) was designed, as shown in Fig. 4A. In the convolution branch, the design of the original bottleneck structure consists of 1 × 1, 3 × 3, and 1 × 1 three-layer convolution operations. The 1 × 1 layers are responsible for reducing and subsequently restoring the dimensions of the feature map. In the residual-mapping branch, a 3 × 3 standard convolution block is added for further feature extraction. BM-SC can compensate for the lack of feature information extraction in lightweight networks. However, the addition of a convolution block increases the computational complexity of BM-SC. To improve the information extraction ability without increasing the computational load, a bottleneck model with depthwise-factorised convolution (DFC), referred to as BM-DFC, was designed to replace the 3 × 3 standard convolution in the convolution and residual mapping branches of BM-SC, as shown in Fig. 4B. DFC is a combination of factorised convolution (Alvarez & Petersson, 2016) and depthwise convolution (Chollet, 2017). In factorised convolution, a standard 2D convolution with 3 × 3 kernels can be converted into two 1D convolutions with kernels of 3 × 1 and 1 × 3. Factorised convolution shrinks the model by reducing the number of redundant parameters and acts as a regulariser in the entire network, enhancing generalisation ability. The key to depthwise convolution is changing the convolution method by setting groups in the convolution process. The operation of DW is illustrated in Fig. 5A. A convolution kernel in DW extracts the feature information of a channel from the input feature map. The number of convolution kernels is similar to the number of channels in the input feature map. In Fig. 5B, compared with the standard convolution operation, DW can clearly reduce the computational complexity. Therefore, DFC is essential for designing a lightweight feature extraction module. Compared with BM-SC, the BM-DFC significantly reduces the number of parameters.

To further improve the accuracy of the final building extraction results, after improving efficiency, ECA (Wang et al., 2020) was used to capture the information of cross-channel interactions with a negligible amount of computation. The architecture of ECA is shown in Fig. 6. The efficient channel attention module first performs a global average pooling operation on the input feature map, which is further subjected to a 1D convolution operation with a convolution kernel size of k to perform information cross-channel interactions. Note that k can be adaptively selected in this process based on the number of channels. The formula for the adaptive selection of k is expressed as follows: (1)${\displaystyle t=int\left(abs\left(\left(log\left(\mathrm{c},2\right)+\mathrm{b}\right)/\gamma \right)\right)}$ (2)${\displaystyle k=\mathrm{f}\left(\mathrm{C}\right)={\left|\frac{{log}_2\left(\mathrm{C}\right)}{\gamma }+\frac{\mathrm{b}}{\gamma }\right|}_{\text{odd}}}$

where |t|_odd represents the odd number closest to t. Note that γ and b are set to two and one, respectively, as described in Wang et al. (2020).

Equations (1) and (2) indicate that high-dimensional channels are characterised by long-distance interactions, whereas low-dimensional channels are characterised by short-distance interactions. In the following operation, a sigmoid activation function is implemented to obtain the weight of each channel. Finally, the weights are multiplied by the corresponding elements of the original input feature map to obtain the final output feature map. ECA can obtain effective channel information and reduce the impact of noise, which improves feature extraction performance.

Considering the advantages of the ECA module, after the DFC in the convolution branch based on BM-DFC, an ECA model was added. This is because ECA can further refine the feature map, which allows the generation of more effective feature maps for dimension recovery. This new module is referred to as the LFEM lightweight feature extraction module and is the final core module of the proposed network, as shown in Fig. 7. LFEM effectively improves feature extraction capacity with a small number of parameters.

Flowchart

The process followed in this study is clarified in the flowchart shown in Fig. 8. In the training stage, the model learns the building features in the remote sensing images based on the training and validation sets, prior to generating the weight model. In the prediction stage, the weight model is applied to the test set for automatic detection of buildings, and relevant evaluation metrics are calculated to measure the quality of the model. When the test results do not attain the expected values, the model is retrained on the datasets until the best prediction results are generated, to obtain the optimal weight model.

Datasets

The effectiveness of the proposed method was demonstrated on the WHU public building extraction dataset (Ji, Wei & Lu, 2018), comprising data from two cities (New Zealand and Christchurch), with a total area greater than 450 km². The spatial resolution of the images is 0.3 m. The dataset contains 8,188 optical remote sensing images with a crop size of 512 × 512 pixels and their corresponding label images. The entire dataset was divided into training, validation, and test sets with 4,736, 1,036, and 2,416 images, respectively. All the experiments were conducted using the original partition of this dataset.

Evaluation metrics

To measure the accuracy of the proposed ELRNet for the building extraction tasks, three general metrics were used: IoU, F1-score, and overall accuracy (OA). The IoU calculates the ratio of intersection and union between the predicted and actual segmentation areas. The OA represents the ratio of correctly classified pixels to all pixels. The F1-score is a composite metric of precision and recall. The higher the values of the above three metrics, the better is the model performance. (3)${\displaystyle IoU=\frac{TP}{TP+FP+FN}}$ (4)${\displaystyle F1-score=\frac{2\times TP}{2\times TP+FN+FP}}$ (5)${\displaystyle OA=\frac{TP+TN}{TP+TN+FP+FN}}$

where TP, FP, FN, and TN are the number of pixels corresponding to true positive, false positive, false negative, and true negative, respectively.

To measure the efficiency of the proposed ELRNet for building extraction, the floating-point number (FLOPs) and number of parameters (Params) were used as evaluation metrics in the network training process. In addition, the time required to complete the predictive inference for each network (TestTime) was evaluated.

Experimental setup

All experiments were implemented using the Pytorch1.11.0 deep learning framework, training and testing the model on workstations with GPU RTX3090. The workstation configuration was based on an Intel(R) Xeon(R) Gold 6230R CPU @ 2.10 GHz, 252 GB memory, 64-bit Ubuntu 20.04 operating system, and 11.3 CUDA.

As optimizer, Adam (Kingma & Ba, 2014) was used with weight decay rate and momentum set to 0.0002 and 0.9, respectively. Validation was performed every 10 epochs and the best-weighted model in the validation set was used as the final model for performance evaluation. The initial parameter details for training and testing are shown in Table 2.

Ablation studies

A series of ablation experiments was designed to demonstrate the effectiveness of the ELRNet architecture on the WHU dataset and the importance of each main component of the proposed method. The experiments replaced the feature extraction module in the network with the five different modules described in the Lightweight feature extraction module (LFEM) section: non-bottleneck, bottleneck, BM-SC, BM-DFC, and LFEM.

The results of the ablation experiments displayed in Table 3 indicate that the backbone of ELRNet can effectively extract buildings, regardless of the feature extraction module used. ELRNet-BM-SC with almost four times more floating-point calculations compared with ELRNet-bt, increases the accuracy of the extraction results. Compared with ELRNet-bt, the parameter and computational complexities of ELRNet-BM-DFC are reduced by 0.43 and 0.07, respectively, with IoU and F1-scores that are slightly better than those of ELRNet-bt. Furthermore, compared with ELRNet-BM-DFC, the IoU in ELRNet-LFEM increases by 0.6% without a significant increase in computational complexity and number of parameters. This result shows that ECA can improve accuracy without affecting efficiency. Therefore, ELRNet-LFEM achieves a better balance between accuracy and efficiency in building extraction. Examples of visual comparisons of the ablation experiments on the WHU test set are shown in Fig. 9. The proposed ELRNet is insensitive to the shape and density of buildings, and the extraction results are more complete and more consistent with the original label image.

In summary, the ablation experiments proved that the designed architecture and feature extraction module are effective, and their ingenious combination improves network performance. The proposed ELRNet architecture is advantageous for accurate extraction of buildings with limited hardware resources. In addition, the proposed LFEM feature extraction module can improve the extraction results for buildings in VHR remote sensing images.

Comparative analysis

In this section, the proposed ELRNet is evaluated and compared with two popular baseline networks (SegNet and U-Net) and four state-of-the-art lightweight building-extraction methods (ENet, EDANet, ESFNet, and ERFNet) for the same experimental environment and configuration. The six aforementioned metrics were used to evaluate the effectiveness of the proposed method. The experimental results are displayed in Table 4.

The baseline methods achieved the highest accuracy although they require a large number of computations and parameters. By comparing the proposed method with the baseline methods of SegNet and U-Net, it can be deduced that ELRNet reduces the computational complexity by 55 and 43 times and improves the IoU by 5.21% and 1.94%, respectively. In contrast to the efficient ENet architecture, the IoU of the proposed ELRNet increased by 1.76% and the numbers of parameters decreased by 0.13. Moreover, compared with EDANet and ESFNet, the IoU improved by approximately 3.49% and 3.26% , respectively. In addition, compared with ERFNet, the computational complexity of the proposed method is lower by 11.75 GFLOPs, and the IoU of the proposed network achieves an improvement of 1%. The experiments demonstrated that the proposed architecture provides an excellent tradeoff between reliability and speed.

A comparison of the prediction results obtained using the six networks on the WHU test set is shown in Fig. 10. To verify the effectiveness of the proposed ELRNet, images of various types of small, large, special-shaped, dense, and sparse buildings were selected from the test results and used for comparison. It is observed that the proposed network performs well for various types of building extraction, and the boundary extraction of buildings is clearer and more regular. Therefore, the proposed ELRNet can achieve a higher extraction accuracy with lower computational complexity and fewer parameters.

In summary, a novel lightweight network, ELRNet, was designed to balance accuracy and efficiency in the task of automatic building extraction from high-resolution remote sensing images.

The main highlights of ELRNet can be summarised in the following two aspects: (1) Our network is specially designed for the characteristics of VHR images, which have a lower spatial resolution than other data. In view of the low-resolution characteristics, as few layers as possible were selected in the design of the network layer as well as the downsampling layer to ensure that the image would retain its characteristics when reduced in size to the smallest size in the downsampling layer. After experimental exploration, the current 23-layer network design achieved the best test results on the VHR image datasets. (2) VHR images contain a variety of ground objects, and the process of building extraction is easily affected by surrounding similar ground objects. Our self-designed feature extraction module, LFEM, not only reduces the influence of surrounding noise but also emphasises the difference between fine-grained features and location representation of different layers, to extract more effective information.

However, ELRNet also has some limitations. Because the shapes and styles of buildings in remote sensing images in different regions are different, the lightweight network in our study is insufficient to extract detailed feature information of extremely small buildings. In the future, we will continue to explore improvements to the network model, primarily with respect to two aspects: the feature extraction module and spatial attention mechanism.