Autonomous vehicle surveillance through fuzzy C-means segmentation and DeepSORT on aerial images

Machine learning-based traffic scene analysis

Machine learning has long been used in computer vision, particularly for traffic monitoring. Rafique et al. (2023) developed a vehicle detection model using Haar-like features and an AdaBoost classifier. Tang et al. (2017) performed detection and classification via background subtraction and SIFT extraction, training neural networks and support vector machines (SVMs). Jabri et al. (2018) used Haar features and local binary patterns (LBP) with an AdaBoost classifier, which, when combined, improved results but increased energy consumption (Akhter, Jalal & Kim, 2021). Charouh, Ghogho & Guennoun (2019) proposed a moving vehicle detection method using background subtraction and morphological corrections. Yu et al. (2003) classified and tracked vehicles via image differencing and a Kalman filter. Mu, Hui & Zhao (2016) identified moving vehicles by selecting high sum of absolute differences (SAD) regions and applying SIFT for matching. Chen, Ellis & Velastin (2012) introduced an urban vehicle detection method using SVM and HOG features. However, these approaches are computationally intensive and struggle with complex scenes, affecting model generalizability.

Deep learning-based traffic scene analysis

Traditionally, traffic monitoring relies on manual methods and in-vehicle technologies. However, deep learning-based image processing has outperformed conventional approaches. Lin & Jhang (2022) proposed a car detection method using the YOLOv4 algorithm. Similarly, Al-qaness et al. (2021) developed a model that processes traffic video sequences with a CNN before applying YOLO for detection, though missing vehicles remain a challenge. Muchtar, Afdhal & Nasaruddin (2020) introduced a YOLOv3-based vehicle detection and classification approach, segmenting images with MOG2 before detection. Ammour et al. (2017) presented a vehicle detection and counting method combining a linear SVM classifier with a pre-trained CNN on high-resolution UAV imagery, but its high computational time limits real-time application.

This research aims to enhance modern computer vision technologies. Our system efficiently detects, tracks, and monitors traffic for intelligent, environmentally aware surveillance. By integrating multiple deep learning methodologies, it surpasses existing car detection and autonomous traffic surveillance systems in performance and accuracy.

Innovation over existing approaches

Our model advances vehicle detection and tracking in accuracy, scalability, and applicability. Unlike handcrafted feature-based methods like HOG-SVM and SIFT, which struggle with occlusions and complex backgrounds, we use YOLOv4, optimized for small object detection. Integrating FCM segmentation before detection reduces background interference, improving precision, recall, and F1-score in aerial imagery.

Many existing methods rely on computationally expensive region-based detectors like Faster region-based convolutional neural network (R-CNN), unsuitable for real-time UAV traffic monitoring. Our pipeline, using YOLOv4 and DeepSORT, is optimized for large-scale, real-time processing, ensuring efficient tracking in dynamic conditions. Unlike ground-based CCTV models, our UAV-specific approach integrates DeepSORT with ID assignment and SURF-based recovery for robust multi-frame tracking. Additionally, our vehicle trajectory estimation module enables a comprehensive analysis of traffic flow patterns, often missing in conventional detection models.

Preprocessing

In the preprocessing step, we extract a series of images from the video to achieve better outcomes for vehicle detection and tracking. Afterwards, each extracted frame is resized to 768 × 768 dimensions. To denoise the image, a nonlinear adaptation known as gamma correction (Xu et al., 2009) is applied to every pixel value using

(1) $$O=I^{1/G}$$where I is the input image, G represents the gamma value, which can be obtained using Eq. (2) and O denotes the output image scaled back to [0, 255].

(2) $$G=\log {\displaystyle \frac{0.5\times 255}{S}}$$where $\mathit{S}$ is the mean of the image after converting it to grayscale. Preprocessing plays a crucial role in enhancing image quality before segmentation, detection, and tracking. In our method, we remove Gaussian noise and motion blur common in UAV-captured imagery using median filtering to preserve edges while eliminating noise artifacts. Additionally, gamma correction is applied to adjust brightness variations caused by aerial lighting conditions. The gamma value (G) is selected adaptively using the formula G = mean ( $I_{gray}$)/128, where $I_{gray}$ is the grayscale-converted image. This ensures that contrast enhancement is proportional to the image’s overall brightness, preventing overexposure or underexposure.

Semantic segmentation

Segmentation is a crucial preprocessing step in aerial image analysis, isolating vehicles from complex backgrounds to improve object detection efficiency. Aerial images often include irrelevant elements like buildings, vegetation, and shadows, leading to false detections and higher computational costs (Rafique et al., 2022). Using RF and FCM clustering reduces background noise, allowing the model to focus on vehicles. This enhances detection accuracy, minimizes misclassifications, and improves tracking by ensuring consistent vehicle identification across frames.

Random forest-based semantic segmentation

A random forest is a supervised classifier. The first step in segmentation involves extracting a feature vector from images and labels for model training. Feature extraction uses filters such as Canny edge detection (Yuan & Xu, 2016), Prewitt (Chaple, Daruwala & Gofane, 2015), Scharr (Kumar, Lal & Kumar, 2021), and a median filter. RF combines multiple decision trees; each tree outputs a class label, and the label with the most votes becomes the predicted output, as shown in Fig. 2.

Each feature’s importance on the decision tree is calculated by Eq. (3).

(3) $$f{i}_i={\displaystyle \frac{{\sum }_{j:nodejsplitsonfeaturein{i}_j}}{{\sum }_{k\in allnodes}n{i}_k}}$$where $f{i}_i$ represents the importance score of features $i$, $n{i}_j$ denotes the importance contribution of node j that splits on feature I and $n{i}_k$ is the total importance contribution across all nodes k in the tree. Each calculated feature is normalized by using Eq. (4).

(4) $$normf{i}_i={\displaystyle \frac{f{i}_i}{{\sum }_{j\in allfeatures}f{i}_i}.}$$

Also, the overall importance of each input feature has been calculated by averaging it over all the Random Forest trees as given in Eq. (5).

(5) $$RFf{i}_i={\displaystyle \frac{{\sum }_{j\in alltrees}normf{i}_{ij}}{T}}$$where $RFf{i}_i$ is the overall importance score of feature i in the random forest, $\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}f{i}_{ij}$ represents the normalized importance of feature i in tree j, and T is the total number of trees in the forest.

We trained the model using the leave one subject out (LOSO) validation technique (Pauli, Pohl & Golz, 2021). Figure 3 visualizes the segmentation results: Fig. 3A displays the original frames, while Fig. 3B shows the segmentation using RF.

Segmentation using fuzzy C-means algorithm

FCM is a segmentation algorithm where pixels are linked to multiple clusters (Miao, Zhou & Huang, 2020), reflecting fuzzy logic based on co-occurring elements. The objective function is optimized over iterations to complete the segmentation process (Ahmed, Jalal & Kim, 2020). During this process, clustering centers and membership degrees are updated. The performance index K_FCM calculates the weighted sum of distances between elements and cluster centers, as shown in Eq. (6).

(6) $$K_{FCM}=\left(S,T\right)=\sum _{i=1}^c\sum _{k=1}^N{t}_{ik}^p\parallel u_k-v_i{\parallel }^2,1<p<\mathrm{\infty }$$where $K_{FCM}\left(S,T\right)$ represents the objective function of the FCM clustering algorithm, $t_{ik}$ is the membership degree of data point kkk belonging to cluster i, $u_k$ is the data point, $v_i$ is the center of cluster i, and p is the fuzziness parameter (with 1 < p < $\mathrm{\infty }$). The degree of membership function satisfies the conditions given in Eq. (7).

(7) $$K_{FCM}=\left\{T\in R_{cn}\mid \mathrm{\forall }1⩽i⩽c,1⩽k⩽N{t}_{ik}\in \left[0,1\right];\sum _{i=1}^c{t}_{ik}=1;0<\sum _{i=1}^c{t}_{ik}<N\right\}.$$

$K_{FCM}$ is the feasible membership matrix in the Fuzzy C-Means algorithm, where $t_{ik}$ denotes the membership degree of data point k to cluster i, c is the number of clusters, and N is the total number of data points. The conditions ensure membership values are between 0 and 1, the sum of memberships for each data point across all clusters equals 1, and each cluster has non-zero total membership.

Equations (8) and (9) are used to update the membership matrix and the cluster centres.

(8) $${\mathit{t}}_{\mathit{i}\mathit{k}}^{\mathit{p}}={\displaystyle \frac{1}{{\sum }_{j=1}^c{\left({\displaystyle \frac{d_{ik}^2}{d_{jk}^2}}\right)}^{\frac{1}{p-1}}}}$$where $t_{ik}^p$ represents the membership degree of data point k to cluster i raised to the power of p, $d_{ik}$ is the distance between data point k and cluster center i, and $d_{jk}$ is the distance between data point k and other cluster centers j. The equation calculates the fuzzy membership based on the relative distances, where p is the fuzziness parameter.

(9) $${\mathit{v}}_{\mathit{j}}={\displaystyle \frac{{\sum }_{k=1}^N{t}_{ik}^p{u}_k}{{\sum }_{k=1}^N{t}_{ik}^p}}$$where ${\mathit{v}}_{\mathit{j}}$ represents the center of cluster j, $t_{ik}^p$ is the membership degree of data point k to cluster i raised to the power of p, and $u_k$ is the data point. The equation calculates the weighted average of the data points, with the membership degrees serving as the weights, to determine the new center of cluster j.

Pixels are given high membership values when they are near the centroid of their respective class, and low membership values when they are far from the centroid, thus, the FCM objective function was minimized. The output of the FCM segmentation is shown in Fig. 4. Figure 4A presents the original images whereas the segmented images are shown in Fig. 4B.

For segmentation, we use RF and FCM to separate vehicles from the background. RF segmentation employs supervised learning with features extracted via Canny edge detection, Prewitt, Scharr, and median filters, classified by an ensemble of decision trees trained using LOSO validation. FCM clustering assigns membership values to pixels, enabling soft clustering that minimizes errors, especially in occluded areas. The computational cost and error rates of RF and FCM are compared using Eq. (10).

(10) $$ErrorRate=1-accuracy$$

The RF classifier requires training on the ground truth, increasing the model’s computational complexity. In contrast, FCM can be applied without explicit training, enhancing the algorithm’s generalizability. Table 1 shows the error rates for image segmentation using RF and FCM on the UAVDT and KIS-AIS datasets. Considering both computational time and error rates, FCM proves to be more effective and accurate. Therefore, FCMP results are used for further processing, such as vehicle detection, ID allocation, recovery, counting, and tracking. Table 2 shows comparsion of segmentation techniques.

Table 1:

Error rate comparison of random forest and FCM segmentation.

Datasets	Error rate
	Random forest	FCM
UAVDT	0.37	0.19
KIT-AIS	0.43	0.24

DOI: 10.7717/peerj-cs.2835/table-1

Table 2:

Accuracies comparison of random forest and FCM segmentation.

Datasets	Segmentation accuracies
	Random forest	FCM
UAVDT	0.63	0.81
KIT-AIS	0.57	0.76

DOI: 10.7717/peerj-cs.2835/table-2

Vehicle detection

After image segmentation, vehicle detection is performed using the YOLOv4 deep learning algorithm, which detects objects with high accuracy and speed. YOLOv4’s architecture includes CSPDarknet53, PANet, and the YOLOv3 head, with CNNs splitting images into regions to predict probabilities. This reduces overfitting by using convolution and fewer parameters, as shown in Fig. 5.

(11) $$G\left[i,j\right]=\sum _{u=-k}^k\sum _{v=-k}^{k}h\left[u,v\right]F\left[i+u,j+v\right].$$

G[i, j] represents the output at position (i, j), h[u, v] is the filter value at (u, v), and F[i + u, j + v] is the input at the shifted position. The equation performs a convolution by summing the product of the filter and input values over a kernel-defined neighborhood. The CNN uses a pooling layer for downsampling, as defined in Eq. (12).

(12) $$f^{\left(m\right)}={\left[f_1^m,\dots \dots f_k^m\dots \dots f_K^m\right]}^T,f_k^m=ma{x}_{x\in X_i}x$$where $f_k^m$ is the maximum value of the pixel after each iteration. Convolutional and pooling layers work together to facilitate effective feature learning and selection. In the end, the Mish activation function is implemented to get the output label as given in Eq. (13).

(13) $$f\left(x\right)=x\cdot tanh\left(\varsigma \left(x\right)\right)$$where $\varsigma \left(x\right)=ln\left(1+e^x\right)$ represents the soft plus activation function.

When the performance was compared, YOLOv4 was two times faster than EfficientDet (a competitive recognition model). For vehicle detection, we utilize YOLOv4, which consists of a CSPDarknet53 backbone, a PANet path-aggregation neck, and a YOLOv3 head. We configure YOLOv4 with batch size = 16, momentum = 0.937, learning rate = 0.001, and IoU threshold = 0.5 to balance accuracy and computational efficiency in UAV imagery. Table 4 shows comparsion of YOLOv4 with other methods.

ID allocation and recovery

Each detected vehicle is assigned a unique ID based on SURF features, which provide reliable, scale-invariant, and efficient image comparison (Bay et al., 2008). SURF features are fast to compute, making them suitable for real-time object recognition. The method includes an interest point detector and descriptor (Du, Su & Cai, 2009), with interest points identified using an integral computed by Eq. (14).

(14) $$I_{\mathrm{\Sigma }}\left(x\right)=\sum _{i=0}^x\sum _{j=0}^{y}I\left(i,j\right)$$where $I_{\mathrm{\Sigma }}\left(\mathit{x}\right)$ is the integral image at the location $\mathit{x}={\left(\mathit{x},\mathit{y}\right)}^{\mathit{T}}$ and $I\left(i,j\right)$ is the input image. Moreover, the blob-like structure in the image is calculated by using the Hessian matrix $\mathcal{H}$ as given in Eq. (15).

(15) $$\mathcal{H}\left(\mathrm{x},\sigma \right)=\left[\begin{array}{c}{L}_{xx}\left(x,\sigma \right)L_{xy}\left(x,\sigma \right)\\ L_{xy}\left(x,\sigma \right)L_{yy}\left(x,\sigma \right)\end{array}\right]$$where $\mathcal{H}\left(x,\sigma \right)$ represents the Hessian matrix at point x with scale σ, $L_{xx}\left(x,\sigma \right)$ is the second derivative of the image with respect to x, $L_{xy}\left(x,\sigma \right)$ is the mixed second derivative with respect to x and y, and $L_{yy}\left(x,\sigma \right)$ is the second derivative with respect to y. The Hessian matrix captures the curvature of the image, which is used in various image processing tasks like edge detection and feature extraction. The assigned IDs can be recovered based on the feature-matching score in the succeeding frame detections.

For ID assignment and recovery, we employ SURF to extract key descriptors for each detected vehicle. An ID is successfully recovered if the number of feature matches between frames exceeds a predefined threshold (set to eight in our experiments). If fewer than eight matches are found, the vehicle is registered as a new entry, ensuring a balance between continuity and robustness in tracking.

Vehicle counting

For traffic situation analysis, vehicle counting is performed in each image frame based on vehicle detections. All the detections made by YOLOv4 are recorded using a counter, as given in Eq. (16). Vehicle counting in each frame estimates traffic density on the road at different time intervals, which can be further used to predict spontaneous reactions to traffic jams or other unhappy incidents.

(16) $$VehicleCount=\sum _{i=1}^{N}T$$where T is the vehicle detections in a frame.

Vehicle tracking

To track the movement of vehicles frame by frame, we implemented DeepSORT (Wojke, Bewley & Paulus, 2017; Hou, Wang & Chau, 2019; Kapania et al., 2020; Pramanik et al., 2022). DeepSORT is a tracking algorithm that combines the Kalman filter with deep learning to track objects not only based on their motion and velocity but also on their appearance. The motion information is incorporated using the Mahalanobis distance between the Kalman state and the newly arrived measurement by using Eq. (17).

(17) $$d^{\left(1\right)}\left(i,j\right)={\left(d_j-y_i\right)}^T{S}_i^{-1}\left(d_j-y_i\right)$$where $d^{\left(1\right)}\left(i,j\right)$ represents the Mahalanobis distance between data point $d_j$ and the mean $y_i$ of cluster i, $S_i^{-1}$ is the inverse of the covariance matrix of cluster i, and $\left(d_j-y_i\right)$ is the difference between data point $d_j$ and the cluster center $y_i$. The equation computes the squared distance, weighted by the inverse covariance, which measures how far $d_j$ is from the cluster center in terms of the cluster’s spread.

The appearance information can be calculated using the minimum cosine distance between i-th and j-th detection in appearance space as

(18) $$d^{\left(2\right)}\left(i,j\right)=min\left\{1-r_j^T{r}_k^{\left(i\right)}|r_k^{\left(i\right)}\in {\mathcal{R}}_i\right\}$$where $d^{\left(2\right)}\left(i,j\right)$ represents the minimum distance between data point j and the points in cluster i, $r_j$ is the feature vector of data point j, and $r_k^{\left(i\right)}$ is the feature vector of a point in cluster i. The equation calculates the minimum value of the distance measure $1-r_j^T{r}_k^{\left(i\right)}$, where $r_j^T{r}_k^{\left(i\right)}$ is the dot product, and ${\mathcal{R}}_i$ is the set of all points in cluster i, ensuring the closest match in the feature space.

(19) $$c_{i,j}=\lambda d^{\left(1\right)}\left(i,j\right)+\left(1-\lambda \right)d^{\left(2\right)}\left(i,j\right)$$where $c_{i,j}$ represents the combined distance measure between data point j and cluster i, $d^{\left(1\right)}\left(i,j\right)$ is the Mahalanobis distance (as defined earlier), and $d^{\left(2\right)}\left(i,j\right)$ is the minimum distance measure (as defined earlier). The parameter λ is a weight that balances the contribution of both distance measures, with λ controlling the influence of $d^{\left(1\right)}$ and (1−λ) controlling the influence of $d^{\left(2\right)}$.

We use a pre-trained CNN model with two convolution layers, one max pooling layer, six residual layers connected to a dense layer, and L2 normalization to calculate the appearance features. Figure 6 shows the tracking results in each succeeding frame.

We integrate DeepSORT, an extension of SORT, by adding deep appearance-based feature extraction with Kalman filtering. The Mahalanobis distance associates new detections with tracklets, while a cosine similarity-based re-identification model maintains tracking consistency when objects temporarily exit the frame. Table 5 shows comparsion of DeepSORT with other techniques.

Vehicle trajectories estimation

We track vehicle movement by estimating and plotting each vehicle’s trajectory using the geometric coordinates of detected boxes. The DeepSORT algorithm provides the location data, from which we calculate and mark center points. The algorithm inputs detection coordinates into the DeepSORT tracker, which predicts vehicle locations in subsequent frames. SURF features are extracted and matched to recover vehicle IDs. If matches exceed a threshold, the ID is retrieved; otherwise, a new ID is assigned. Rectangular coordinates and midpoints are used to mark vehicle trajectories. Detailed steps are provided in Algorithm 1.

UAVDT dataset

The UAVDT dataset (Yu et al., 2020) consists of more than 10 h of video captured using a UAV platform in various urban settings. These traffic scenes include intersections, toll plazas, motorways, crossings, and arterial roadways. The .jpg images have a resolution of 1,080 × 540 pixels, while the videos were shot at a frame rate of 30 (fps).

KIT-AIS dataset

The Karlsruher Institut fur Technologie ¨ Aerial Image Sequences (KIT-AIS) dataset (Beheim, 2021) consists of 299 image frames with .jpg format extracted from video sequences. This dataset is provided by German Aerospace. It consists of five different classes, i.e., cars, truck, bus, minibus, and cyclist.

Dataset partitioning and augmentation strategies

Our study uses two publicly available aerial image datasets: UAVDT and KIT-AIS, each containing varied traffic scenarios. Both datasets were split into training, validation, and test sets as follows:

• UAVDT dataset: With its large number of images, we applied an 80-10-10 split (80% for training, 10% for validation, and 10% for testing), ensuring broad training conditions and unbiased performance evaluation.

• KIT-AIS dataset: Due to its smaller size, we enhanced the dataset with random rotation (±15°), horizontal flipping, brightness adjustment, and Gaussian noise addition. This augmented the dataset, creating a 70% training, 15% validation, and 15% test split.

We augmented the KIT-AIS dataset with random rotation (±15°), horizontal flipping, brightness adjustment, and Gaussian noise to simulate real-world variations like lighting, vehicle orientations, and image noise. This improved model generalizability, ensuring robust training and evaluation across diverse aerial traffic scenarios.

Discussion on error propagation and mitigation strategies

Our model demonstrates high accuracy in vehicle detection and tracking but has limitations. Error propagation, due to incorrect segmentation, can lead to missed or misclassified vehicles. This can be mitigated with adaptive error correction, such as feedback loops and motion-based outlier rejection. Scalability is also a concern in high-density urban traffic, where occlusions and overlapping detections complicate ID assignment. Future improvements may include multi-frame aggregation and attention-based deep learning models to reduce ID switches. Additionally, computational efficiency must be optimized for large-scale surveillance, with lighter architectures like YOLOv5 or YOLOv8 and distributed computing to enhance real-time processing. Addressing these challenges will improve model robustness, scalability, and efficiency.