Applying a deep learning pipeline to classify land cover from low-quality historical RGB imagery

Pipeline overview

Our objective was to apply a deep learning pipeline to classify low-quality RGB imagery. Our pipeline (Fig. 1) used modern annotated and RGB images to train an initial DeepLabv3+ model. Then, after manually annotating a small selection of a historical RGB image, the model was fine-tuned by training it on this small annotated region. Our pipeline produced two models (a model for classifying modern land cover and a model for classifying historical land cover). These models can be used to estimate modern and historical land cover, which in turn can be compared to calculate land cover change. We applied this land cover pipeline to classify land cover in Metro Vancouver, BC for 1995. Custom code used in this analysis can be found on DOI: 10.17605/OSF.IO/BWJKN.

Images

We used a modern, geo-referenced, annotated land cover mask and two geo-referenced RGB images—one modern and one historical (Figs. 1A, 1D). Each pixel value in the mask indicates the land cover type of the location associated with that pixel. While high-spatial resolution multispectral images are widely available now, aerial RGB images often provide the highest spatial resolution remote sensing of land in the 1990s necessary for understanding fine-scale changes in land cover such as single trees.

For our modern RGB image, we obtained modern satellite imagery taken over Metro Vancouver in 2021 from Maxar Technologies via Google Imagery (RGB, ∼0.5 m spatial resolution, 8 bit unsigned integer, accessed 01 November, 2022) and downsampled to 1 m spatial resolution (Fig. S6 for map of training imagery extent). We cropped the modern mask and RGB image to cover an area of 48,640 m × 28,672 m. We obtained a 5 m spatial resolution annotated land cover mask for Metro Vancouver representing 2019 (Metro Vancouver, 2019). We upsampled the mask to obtain 1 m spatial resolution and reduced the 14 classes into eight classes, including water, buildings, pavement, bare, shrub, grass (including herbs), coniferous trees, and deciduous trees. We obtained a low-quality 1 m spatial resolution (RGB, 16 bit unsigned integer) aerial photos from the Canadian Wildlife Service, Pacific Region (obtained January 2022) (Fig. 2). These photos were taken over Vancouver, BC, by airplane in May and July 1995 by the Selkirk Remote Sensing Ltd. (Selkirk Remote Sensing, 1995) and georeferenced and othorectified by Triathlon Mapping Corporation (Burnaby, BC) (Triathlon Mapping Corporation, 1996).

We cropped the modern mask and RGB images to cover an area of 48,640 m × 28,672 m. More spatially extensive images will produce more training data and thus a more accurate model as long as increasing the spatial extent does not also increase the diversity of land cover types or appearances. Note that the annotated mask has limited accuracy—e.g., Fig. 3A shows how it classifies the grass field in the lower left as barren. Given the limited accuracy of the original input modern land cover classification, accuracy and confusion matrices for the modern land cover should be interpreted cautiously.

To format images for training, we tiled the modern mask and modern RGB image into 512 × 512 pixels (following Su & Chen, 2019), resulting in 5320 tiles for each image type, where each tile of the RGB image represents the same land area as the corresponding mask tile (Fig. 1A). Future work might investigate the optimal tile size. We converted each tile into png format for compatibility with the TensorFlow image decoder.

Modern deep learning model

To train the initial DeepLabv3+ model, we used TensorFlow 2 (version 2.11.0), Keras (version 2.11.0), and DeepLabv3+ within a Python (version 3.10.9) environment. Analyses were conducted on GNU Linux (6.1.12-arch1-1, 64-bit) 16 × AMD Ryzen 7 4800H processors with 15.0 GiB RAM. DeepLabv3+ can be freely downloaded and installed from https://github.com/google-research/deeplab2. We randomly sorted the mask and RGB tiles into training (90%) and validation (10%) datasets (following Su & Chen, 2019), then converted the tiles into tensors. Future research might test the optimality of this 90/10 ratio.

We trained the DeepLabv3+ dilated convolutional neural network on the modern tiles and masks (Fig. 1B). We used a dilation rate ranging from 1 to 18 in order to capture spatial information. Low-level features were initialized with weights from a backbone model (ResNet50 model trained on the ImageNet database; He et al., 2015). We trained the model for 25 epochs using a batch size of 4. We optimized the model using sparse categorical crossentropy as the loss function and the Adam algorithm for gradient-based optimization (Kingma & Ba, 2014).

Modern model validation

To validate the model, we used the model to predict the land cover classes for the 10% of the modern images that were set aside for validation (Fig. 1C). We compared the resulting predictions with the original annotated masks to calculate model accuracy, produce confusion matrices, and visually inspect the accuracy of the predictions (Fig. 3A). We modified model parameters and training dataset size until the accuracy became satisfactory.

Historical annotations

Many deep learning analyses of land cover stop at this point (Su & Chen, 2019; Du et al., 2019). However, because our historical RGB imagery is much lower quality and was produced using different imaging technology than the modern RGB image (Fig. 2), the historical imagery likely does not represent the same feature space nor have the same distribution as the modern RGB training data. Thus following standard methods and using the model trained on the modern data to predict historical land cover classes will likely yield low accuracy (Pan & Yang, 2010). Instead, we manually annotated a small area of the historical image (Fig. 1D) and used transfer-learning to fine-tune the model on this small additional training dataset (Figs. 1E, 1F).

We used the ThRasE plugin (version 22.3.3a; https://github.com/SMByC/ThRasE) in QGIS (version 3.28.0) to manually annotate 58 historical RGB tiles (each tile was 512 m × 512 m, resulting in a total annotated area of ∼ 15 km²). The first author, who carried out the annotation, spent five years in Vancouver, BC, where he conducted many avian and vegetation surveys and ran every street in the city. This first-hand knowledge of the city likely affected the manual annotations. We selected an annotation region that included coniferous forest, broadleaf forest, industrial, residential, and downtown urban core, lakes, and ocean.

To annotate the historical image, we selected an area of the image that was representative of the area of interest. We manually annotated the historical image using the same classes that were contained in the modern annotation mask. To ensure consistency between the classifications, we reviewed the relationship between the modern annotation mask and the modern RGB image. To accelerate manual annotation, the modern model can be used to create initial estimates of historical land cover cover; the modern land cover classification may also provide useful initial estimates that can then be manually corrected. QGIS and the ThRasE plugin make manual annotating easier and are freely available. Label Studio (https://labelstud.io/, accessed 20 March 2023) also provides a free platform for land cover annotation (see e.g., https://www.youtube.com/watch?v=UUP_omOSKuc; accessed 20 March 2023).

After completing annotations, we tiled the historical annotation and spatially-matched historical RGB image. We doubled the size of the annotated dataset by copying each tile and rotating by 90°. Because this annotation dataset is small, such data augmentation via image rotation can increase model accuracy (Perez & Wang, 2017).

Fine tuning

Transfer learning enables the model trained on the modern images to adapt to the lower-quality historical imagery (Pan & Yang, 2010). Thus, we fine-tuned the model trained on the modern images on the historical mask and image tiles (Fig. 1E) using a lower learning rate (10× lower; Shin et al., 2016) and only 20 epochs.

Historical model validation

To validate the fine-tuned model for classifying the historical land cover, we used the fine-tuned model to predict the land cover class for the annotated dataset (using the entire dataset since the annotated dataset is very small; Fig. 1F). We compared the resulting predictions with the manual annotated masks to calculate the model accuracy, produce confusion matrices, and visually inspect the accuracy of the predictions (Fig. 3). To show the effect of fine tuning and data augmentation of the annotated historical images, we tested each of the three models for their capacity to estimate historical land cover, including Model 1: the original modern deep learning model, Model 2: the fine-tuned deep learning model trained without historical data augmentation, and Model 3: the fine-tuned deep learning model trained with historical data augmentation.

Accuracy evaluation indicators

We used overall accuracy (OA) and Cohen’s Kappa, κ, (Cohen, 1960) to evaluate the models, defined as: (1)${\displaystyle \mathrm{accuracy}=\frac{\text{number of matching cells}}{\text{number of cells}},}$ (2)${\displaystyle \kappa =\left(p_0-p_e\right)/\left(1-p_e\right),}$ where p₀ is the observed agreement ratio between two classifiers, and p_e is the expected agreement if both classifiers were random.

Land cover prediction

To predict the land cover across the region of interest, we selected the region of interest and tiled the modern and historical RGB georeferenced images representing this region into 512 × 512 pixel images. We used the first model to create images predicting the land cover classes for each of the modern tiles and the fine-tuned model to create images predicting the land cover classes for each of the historical tiles. We associated the georeference information for each of the tiled RGB images with each of the predicted images and mosaicked the tiles together to create a complete modern and a complete historical land cover image. We used the final models to predict eight land cover classes for Vancouver, BC, Canada for 1995 and 2021. We corrected the land cover for water, since water is easy to manually classify. We compared the distribution of land cover classes within the city to show estimated land cover change over the 26-year period. To show more general changes, we also reduced the eight land classes into three bins: water, built (including buildings, pavement, and bare), and greenery (including shrubs, grass, coniferous trees, and deciduous trees). We used the error rate from the confusion matrices (percentage of classifications that do not match the original annotations) to plot uncertainty of land cover change.

Modern classification

Our land cover classification of the modern image produced an overall accuracy of 0.7037 relative to the 2019 Metro Vancouver land cover classification (Metro Vancouver, 2019). Our land cover class estimates were most consistent with the 2019 Metro Vancouver land cover classification for water (95%), and least consistent for shrubs (7%) (Fig. 4A). Binning the land cover classes into water, built, and greenery yielded improved accuracy, varying from 80% to 95% (Fig. 4B).

Historical classification

Using the model trained only on the modern image produced low accuracy of 55.07%. It particularly struggled to classify water, mischaracterizing many non-water features as water, especially conifers (Fig. 5A). Moreover, pavement and deciduous trees were labelled incorrectly as buildings (Fig. 5A). Fine tuning the model by training on annotated historical imagery produced better predictions, improving the accuracy to 64.74%. In particular, pavement was more accurately labeled as pavement instead of mislabeled as buildings Fig. 5B). Conifers were no longer misclassified as water, though many classes were misclassified as conifers, particularly deciduous trees Fig. 5B). Water was also still overestimated, with grass in particular being misclassified as water (Fig. 5B). Fine tuning the model on augmented historical imagery produced much improved predictions (Fig. 5C). This final model produced an overall accuracy of 75.85% and a kappa coefficient of 68.76 relative to our manual annotation, and a clear diagonal corridor visible in the confusion matrix (Fig. 5C). This model finally correctly classified conifers, with deciduous trees only uncommonly misclassified as conifers. However, shrubs remained poorly classified. Compared to the model only trained on modern imagery, training on augmented annotated historical imagery increased the mean intersection over union (mIoU) by nearly a factor of two (38.63% to 63.62%). Binning the land cover classes into water, built, and greenery yielded further improved accuracy (Fig. 5D). Under our architecture, all models had a computation efficiency of 6 s per step.

Land cover change

Our land cover predictions indicate that between 1995 and 2021, conifer cover decreased by 54%, pavement cover increased by 26%, and deciduous tree cover decreased by 9% (Fig. 6A). Overall, our results indicate that greenery cover decreased by 14% and the built environment cover increased by 10% (Fig. 6B). A contingency matrix of 1995 and 2021 land cover suggests that deciduous trees, grass, and conifers may have been replaced by pavement and buildings (Fig. 7). However, some pavement (12%) and buildings (9%) may have been replaced by deciduous trees (Fig. 7).