Object detection in optical imaging of the Internet of Things based on deep learning

Image target detection algorithm

(1) IoT system

The IoT represents a versatile technology capable of real-time data collection, encompassing a wide array of parameters including sound, light, heat, electricity, chemical composition, mechanical properties, biological metrics, and geographical location (Lv & Singh, 2021). The IoT system is underpinned by four key technologies, which are systematically classified and depicted in Fig. 1.

IoT systems find extensive application across various industries and domains, encompassing manufacturing, agriculture, transportation, energy management, healthcare, and more. In these contexts, IoT systems assume a pivotal role in facilitating intelligent development and refining resource allocation. They offer the mechanisms to boost operational efficiency, fine-tune resource utilization, and ultimately elevate the quality of life and working environments.

(2) Computer vision

Computer vision (CV) is a theoretical framework for processing visual information. This theory delineates the visual perception process into a multi-level, bottom-up analysis sequence. In this process, a series of distinct representations furnishes detailed information concerning the relevant visual environment (Lv et al., 2021).

Two challenges confront CV: the first pertains to feature extraction, and the second concerns the handling of vast computational datasets. DL offers a promising approach to addressing these challenges. Presently, CV encompasses eight distinct task categories, shown in Fig. 2.

As a pivotal domain in the AI field, CV exhibits extensive applications across various sectors, including industry, healthcare, security, agriculture, retail, human–computer interaction, culture, and the arts. Through the simulation of the human visual system, CV empowers computers to recognize, analyze, and comprehend image and video data, thereby enabling automatic image processing and analysis. The contributions of CV resonate in the form of innovation, efficiency, and convenience across diverse industries, propelling technological advancements and nurturing societal development, underscoring its undeniable significance.

(3) Image target detection

One of the primary challenges within the realm of CV is image object detection. The fundamental goal is to discern and categorize all objects of interest in an image while determining their respective positions. Object recognition has traditionally represented the most formidable hurdle in CV, primarily due to the inherent variability in appearances, shapes, and orientations of different objects, further compounded by factors such as varying illumination, occlusion, and other imaging-related complexities. Figure 3 illustrates the principal quandaries associated with image target detection.

There exist five conventional image object detection techniques, including edge detection, Hough Transform, binarization, projection histogram, and feature extraction. DL-based image target detection algorithms can primarily be divided into two categories: Two Stage and One Stage (Zhao et al., 2019).

Deep learning (DL)

(1) DL

DL finds extensive applications in search technology, machine learning, data mining, machine translation, natural language processing (NLP), multimedia learning, recommendation systems, and personalized technology (Janiesch, Zschech & Heinrich, 2021; Sarker, 2021). Furthermore, DL algorithms demonstrate utility in predicting the geometric structure of RNA molecules (Townshend et al., 2021), inferring and forecasting various outcomes (Ranganathan, 2021), and evaluating collaborative work (Pandian, 2021).

DL research methods can be divided into three categories: neural network systems based on convolution, self-coding neural networks based on multilayer neurons, and deep belief networks with optimized neural network weights. Figure 4 displays the DL model.

(2) CNN

CNNs represent a class of algorithms within DL and also function as a type of artificial neural network, encompassing a series of mathematical operations and connection schemes (Li et al., 2021). CNNs are engineered to emulate the visual perception mechanisms observed in biological organisms (Mamalakis, Barnes & Ebert-Uphoff, 2022). They are adaptable to both supervised and unsupervised learning and offer versatile applications across domains such as computer vision, natural language processing, agriculture (Dhaka et al., 2021), social sciences, healthcare (Xu & Qiu, 2021), and numerous others.

CNN architectures primarily comprise input layers, hidden layers, and output layers. Figure 5 shows the CNN model.

The input layer in CNN primarily serves as the data input interface and can effectively handle multidimensional data. Specifically, a one-dimensional CNN can process one- or two-dimensional (2D) datasets, whereas a 2D CNN can handle 2D or three-dimensional (3D) datasets. When dealing with four-dimensional datasets, a 3D CNN is a suitable choice. In the realm of CV, the 3D CNN is commonly employed.

Within the hidden layer of CNNs, one encounters pivotal components such as the convolutional layer, activation function, pooling layer, and fully connected layer. Among these, the convolutional layer occupies a prominent role. This layer comprises numerous convolution kernels, primarily tasked with the extraction of data features. The dimensions of the convolutional layer are contingent upon the size of the convolution kernel. Figure 6 illustrates a visual representation of convolution kernel operations.

The notation is defined as follows: a represents the bias difference, Q^L denotes the convolution input, Q^L+1 signifies the convolution output, $Q\left(x,y\right)$ represents the pixel of the feature map, K stands for the number of channels of the feature map, F indicates the size of the convolution kernel, W₂ denotes the convolution step, $O_K^{L+1}$ signifies the output size, and P represents the number of filling layers. With these definitions in place, the sum of the input features in the receptive field can be expressed as follows:

(1)${\displaystyle Q^{L+1}\left(x,y\right)=\sum _{K=1}^{K_1}\sum _{X=1}^F\sum _{Y=1}^F\left[Q_K^L\left(W_1+x,W_2+y\right)O_K^{L+1}\left(x,y\right)\right]+a}$ (2)${\displaystyle \left(x,y\right)\in \left\{0,1,2,\dots ,n\right\}{L}_{n+1}=\frac{L_n+2P-F}{W_1}+1}$

Activation functions can help express complex features within CNN. Several common activation functions include the rectified linear unit (ReLU), ReLU with slope, parametric ReLU, randomized ReLU, exponential linear unit (ELU), sigmoid function, and hyperbolic tangent function.

For instance, the ReLU function is calculated as Eq. (3), where $max\left(0,x\right)$ represents the ramp function in algebra. (3)${\displaystyle F\left(x\right)=max\left(0,x\right)=\left\{\begin{array}{c}xx\ge 0\hfill \\ 0x<0\hfill \end{array}\right.}$

The sigmoid function is defined as Eq. (4), where e represents a natural function and y is a variable, with a value range between 0 and 1. (4)${\displaystyle F\left(y\right)=\frac{1}{1+e^{-y}}.}$

The hyperbolic tangent function is given by Eq. (5), where e represents the natural function and x is the variable. (5)${\displaystyle tanhx=\frac{sinhx}{coshx}=\frac{e^x-e^{-x}}{e^x+e^{-x}}}$

The pooling layer within the hidden layer plays a crucial role in feature selection and information filtering. The size of the pooling layer area is determined by the pooling layer size, the step size, and the number of filling layers. The output layer of the CNN is responsible for connecting the hidden layer and outputting the model’s results. Figure 7 presents the calculation process of the pooling layer.

The pool layer is a pivotal component in CNNs, which is used to diminish the size of the feature map and extract essential features. The two most prevalent pooling operations are maximum pooling and average pooling.

Maximum pooling achieves downsampling by selecting the maximum value within a local area, thus retaining the primary features. The maximum pooling is given by Eq. (6). (6)${\displaystyle OP\left(\alpha ,\beta ,\gamma \right)=MaxIP\left(\alpha s+i\beta +k\gamma \right)}$

In Eq. (6), OP(α, β, γ) represents the element of the pooled output feature map. IP(αs + iβ + kγ) corresponds to an element of the input feature map. (i, k) ∈ win, signifying the coordinates within the window. win denotes the window, and s signifies the stride.

Average pooling accomplishes downsampling by calculating the average value within a local area, thus encapsulating the overall feature distribution. The average pooling is aarticulated as Eq. (7). (7)${\displaystyle OP\left(\alpha ,\beta ,\gamma \right)=\frac{1}{winS}\sum _{\left(i,k\right)\in win}IP\left(\alpha s+i\beta +k\gamma \right)}$

Here, winS signifies the number of elements in the window.

By employing both maximum pooling and average pooling, the pooling layer can reduce the size of the feature map, enhance the calculational efficiency, and bolster the robustness and generalization capabilities of the model while retaining crucial features. These operations are widely used in various layers of CNN, facilitating the effective processing of images and other data types.

To fine-tune the CNN target, the utilization of a loss function proves instrumental in predicting weights through coordinate adjustments. Among the prominent loss functions are loss functions are the mean squared error (MSE) loss function, cross entropy, Euclidean distance loss function, Manhattan distance loss function, smooth L1 loss function, and Huber loss function.

MSE serves a frequently employed loss function, assessing the square of the average difference between the predicted value and the actual value. The MSE is calculated using Eq. (8). (8)${\displaystyle MSE=\frac{1}{Z}\sum _{X=1}^Z{\left(L^X-{\ddot{L}}^X\right)}^2}$

In Eq. (8), Z represents the number of samples, L^X denotes the real value of the X sample, and ${\ddot{L}}^X$ signifies the predicted value of the X sample.

The image target detection algorithm based on the neural networks can employ the MSE loss function, and its calculation equation is as follows: (9)${\displaystyle Loss=\sum _{y=0}^{x^2}CoordError+IoUError+ClassError}$

In Eq. (9), Loss represents the loss function; x and y are variables; IoUError denotes the confidence value error; ClassError signifies the classification error; CoordError refers to the coordinate error.

If x and y represent two probabilities, the probability’s cross entropy equation is defined as: (10)${\displaystyle C\left(x,y\right)=-\sum _{n}x\left(n\right)logy\left(n\right)}$

The normalized exponential function is adopted to optimize the cross entropy, and the following equation is derived: (11)${\displaystyle Softmax\left(x\right)=\frac{e^{xn}}{\sum _n{e}^{xn}}}$

Experimental design

(1) A training model for dynamic image target detection built based on the CNN algorithm

This article conducts all the experiments related to building a training model for dynamic image target detection utilizing the CNN algorithm on the Titan algorithm platform. The training of our model is performed using the Jetson TX2 embedded algorithm, with the Pascal VOC 2012 dataset used as the background database to augment the database. Several key configurations are established for the training process, including:

(1) Definition of the training database location for image targets and background data.

(2) Specification of classification labels.

(3) Determination of image dimensions during CNN training.

(4) Selection of the number of samples for each training session.

(5) Assignment of image target categories.

(6) Setting precise values for the image target detection algorithm.

(7) Configuration of the number and size of anchor boxes.

(8) Utilization of three distinct network models: a nine-layer CNN model, a seven-layer CNN model, and a CNN model with residual modules.

(9) Establishment of a target loss function.

(10) Utilization of the Adam optimizer for training algorithm optimization.

The Pascal VOC 2012 dataset is a versatile resource suitable for tasks such as image classification, object detection, and image segmentation. This dataset comprises an extensive collection of 23,080 image datasets and 54,900 project datasets, making it a benchmark dataset for image target detection. This article curates a training set consisting of 3,500 images from this dataset.

(2) CNN simulation models based on CV

This article develops several CNN simulation models grounded in computer vision principles. These models vary in architecture, and their specifications are detailed as follows:

(1) For the nine-layer CNN model, the number of network convolution layer channels follows the progression of 64, 128, 256, and 512, with an output set at 13  × 13  × 5  × 7.

(2) The seven-layer CNN model has an input layer size of 512  × 512  ×35, with the number of convolution kernels specified as 8, 16, 32, 64, and 128.

(3) In the residual module CNN model, the size of the convolution layer is established at 3  ×3  × 140.

(4) In these CNN simulation models, designed within the realm of visual computing, key parameters are set as follows:

(5) Initial learning rate: 0.0001

(6) Optimization method: Adam optimizer

(7) Number of iterations: 10

(8) Learning rate decay: 0.1

(9) Training cessation criterion: When the loss function remains unchanged for three consecutive weeks.

Furthermore, the architecture details are as follows:

(1) The seven-layer CNN model comprises seven layers, each with 16 neurons.

(2) The nine-layer CNN model incorporates nine layers, with each layer containing 32 neurons.

(3) The residual module CNN model consists of seven layers, with each layer featuring eight neurons.

In the evaluation process, this article conducts multiple training sessions to determine the impact of varying network models and loss functions on precision, accuracy, IoU, and FPS. Here, the article clarifies the definitions and significance of these performance metrics:

(1) Precision quantifies the proportion of correctly identified positive samples out of all samples that the model predicts as positive. It serves as an essential measure of the model’s ability to avoid false positives.

(2) Accuracy represents the ratio of correctly predicted samples to the total number of samples across all categories. This metric provides a holistic assessment of the model’s overall performance.

(3) IoU is not a commonly used metric in target detection. Its purpose is to gauge the degree of overlap between the model’s detected target area and the actual target area. The IoU is calculated by dividing the intersection area of the target frame by the union area of the target frame. Higher IoU values indicate that the model’s detection aligns closely with the actual target.

(4) FPS signifies the number of frames the model can process in one second when analyzing images or videos. It serves as a critical metric for assessing the model’s real-time processing capability and performance in tasks requiring quick responses.

These performance indicators collectively provide a comprehensive evaluation of the model’s effectiveness in the domain of target detection. It’s essential to note that each metric emphasizes different aspects of performance, contributing to a comprehensive understanding of the model’s capabilities and effectiveness in specific use cases.

This article is conducted on the Titan algorithm platform, with testing executed using the Jetson TX2 embedded algorithm. The article employs the Pascal VOC 2012 dateset as the background database, with a selection of 3,000 pictures images serving as the model’s test set. TensorFlow framework was employed to implement the sample code for the target detection model. The model architecture encompasses a nine-layer CNN, a seven-layer CNN, and a residual module CNN model. The main function of this code is to define the CNN model, along with its corresponding loss function and optimizer. Throughout the training process, the loss function and accuracy metrics are computed at the conclusion of each epoch.

Based on prior studies, this article sets two labels for the binary classification of images, with the image dimension set at 512 × 512 × 3. The selection of the object detection algorithm’s accuracy and the configuration of anchor frames parameters draw from established practices in the field. In addition, the utilization of the Adam optimizer, as a commonly used training algorithm, is motivated by its demonstrated capacity to facilitate improved convergence and performance across various scenarios. The rationale for selecting the Adam optimizer is rooted in its extensive utilization and effectiveness with gradient descent optimization techniques (Freitas et al., 2019).

(3) Target image detection model of the CNN algorithm based on the attention mechanism.

The attention mechanism is prevalent in DL, enhancing the attention and processing ability of neural networks to input data. By automatically assigning varying weights to distinct segments of the input data, attention mechanisms prioritize crucial information during processing, thereby augmenting the model’s performance and generalization capability. The structural layout of the target image detection model, employing the CNN algorithm integrated with an attention mechanism, is shown in Fig. 8.

This article evaluates the CNN-based target image detection model with an integrated attention mechanism using the KITTI dataset. A total of 3,600 images are designated for the training set, while 3,500 images constitute the test set. To ensure consistency, all images are standardized to a high resolution of 384 × 1,280. The model’s performance is assessed based on the average accuracy of aerial view (AP-R40), which serves as a key performance index. Three levels of recognition difficulty—simple, intermediate, and challenging—are established, and the accuracy and detection time of various algorithmic models are compared across these conditions. Among these metrics, target detection accuracy reflects the algorithm’s ability to correctly identify target objects and can be quantified by such indicators as precision, accuracy, IoU, and FPS. High accuracy suggests that the algorithm excels in precise target identification and localization, minimizing instances of misjudgment and missed detection, thus yielding more reliable detection outcomes. Detection time refers to the duration required for the algorithm to complete the target detection task, with short detection times ensuring swift responsiveness and adaptability to dynamic environments.

The KITTI dataset stands as an open dataset extensively employed in the fields of CV and autonomous driving. It comprises a diverse range of data types, including images, laser point clouds, calibration details, and vehicle trajectories, encompassing various scenes. This dataset is collaboratively developed through the Karlsruhe Institute of Technology in Germany and the Toyota European Research Center. It spans multiple different driving environments, such as urban streets and highways. This expansive dataset offers researchers a wealth of real-world data for studying and developing algorithms and technologies within the realms of target detection, object tracking, stereo vision, and autonomous driving.

This article conducts all experiments for constructing the target image detection model using the CNN algorithm based on the attention mechanism with the PyTorch framework. The ResNet-50 serves as the backbone network to acquire essential visual features related to the target. Additionally, the attention mechanism is used to reduce superfluous information in the image. Notably, the CNN algorithm model is configured with a batch size of eight, an initial learning rate of 0.0002, a learning attenuation rate of 0.1, a decay period of 50, and a weight attenuation coefficient of 0.0001. Optimization is achieved through the Adam optimizer, encompassing a total of 200 training rounds. The model consists of both coding and decoding blocks, each comprising six layers. The number of queries is set to 50, and the depth range spans from 0 to 60 m.

Simulation of simple and salient environments

This article addresses the challenge of image target detection within simple and salient environments. To conduct simulation experiments, a seven-layer CNN model is used. The ensuing simulation results are shown in Fig. 9, showcasing the image target detection performance derived from the seven-layer CNN model within such simple and salient environments.

The analysis of Fig. 9 reveals that in a simple and salient environment, the utilization of a seven-layer CNN model appears to be generally suitable. Notably, the loss function within the image target detection algorithm steadily decreases as the training period extends, reaching stability once the training period reaches 12 units. The confidence curve for specific categories exhibits significant fluctuations throughout the training process, initially declining between 0 and 12 units. Subsequently, from 12: 00 to 89: 00, units into training, there is a discernible overall growth trend, albeit with minor fluctuations. Upon surpassing the 100-unit training threshold, the confidence curve for specific categories shows an upward trend, ultimately reaching a confidence level of 0.92 at the end of training. In this simple and salient environment, the seven-layer CNN model demonstrates high confidence and commendable accuracy. Nevertheless, some images remain undetected. This outcome might be attributed to the limited diversity of objects within the simple and salient environment, which potentially hinders the model’s ability to sufficiently learn the characteristics of specific samples. In addition, factors such as model parameter configurations, data quality in the training dataset, and network structure may also affect the occurrence of undetected situations.

Simulation of complex and salient environments

In the investigation of image target detection within a complex and salient environment, a nine-layer CNN model, a seven-layer CNN model, and a residual module CNN model are employed to conduct simulation experiments. The simulation outcomes are presented in Figs. 10–12, offering insights into the performance of image target detection based on the three CNN models in a complex and salient environment.

Figure 10 reveals that within the nine-layer CNN model in a complex and salient environment, the model employing cross entropy as the loss function achieves an accuracy of 0.77, a precision of 0.97, an IoU score of 0.81, and an FPS value of 29.2. Conversely, when the loss function is the mean square error (MSE), the model attains an accuracy of 0.7, a precision of 0.9, an IoU of 0.76, and a FPS value of 29.5. Comparing the two models, it becomes evident that the nine-layer CNN model employing cross entropy yields higher precision, IoU, and accuracy in image target detection. However, it is noted that the detection speed is marginally lower compared to the nine-layer CNN model utilizing MSE. This observation aligns with the expected characteristics of cross entropy as the loss function in classification problems. While it enhances classification results, it may introduce a slightly complicated training process, thus affecting the detection speed.

Figure 11 provides insights into the performance of the seven-layer CNN model operating in a complex and salient environment. When employing cross entropy as the loss function, the model exhibits an accuracy of 0.87, a precision of 0.91, an IoU score of 0.73, and a FPS value of 30.1. Conversely, when utilizing MSE as the loss function, the model achieves an accuracy of 0.88, a precision of 0.79, an IoU value of 0.73, and a FPS value of 33.8. A comparative analysis between these two models reveals that the seven-layer CNN model using cross entropy as the loss function demonstrates superior accuracy in image target detection. However, it exhibits slightly lower detection speed and overall accuracy in comparison to the seven-layer CNN model utilizing MSE as the loss function. This phenomenon accords with the application of cross entropy in classification problems, which accentuates inter-class differences, ultimately enhancing accuracy. Nonetheless, it may have a minor impact on detection speed and overall model accuracy.

Figure 12 presents a comprehensive analysis of the performance of the residual module CNN model operating in a complex and salient environment under various conditions. When the number of training iterations is set to 1, the model using cross entropy as the loss function demonstrates an accuracy of 0.77, a precision of 1, an IoU score of 0.75, and an FPS value of 19.9. When employing MSE as the loss function, the model achieves an accuracy of 0.85, a precision of 0.98, an IoU of 0.8, and a FPS value of 20.4. Additionally, the model employing a combination of MSE and cross entropy as the loss function exhibits an accuracy of 0.93, a precision of 0.99, an IoU of 0.76, and an FPS value of 26.3. With an increase in the number of training iterations to 2, the residual module CNN model with cross entropy as the loss function achieves an accuracy of 0.92, a precision of 1, an IoU of 0.71, and an FPS value of 20.1. Upon comparing the four models, it becomes evident that when the number of training iterations is set to 1, the residual module CNN model with loss function of MSE and cross entropy simultaneously attains higher precision, accuracy, IoU scores, and FPS values. These metrics were slightly lower than the CNN model using MSE as the loss function alone. The observed improvement in accuracy with increased training iterations is accompanied by a reduction in the IoU score. This phenomenon aligns with the iterative training process of DL models. The choice of different training strategies and loss functions can balance the model’s performance between accuracy and IoU scores.

Simulation of intricate micro-environments

In complex micro-environment, this article conducts comprehensive simulation experiments to evaluate the performance of the residual module CNN model within intricate micro-environments. The experiments are executed across three distinct network states, and the results are visualized in Fig. 13. It illustrates the model’s capabilities for image target detection in various network states in intricate micro-environments.

The analysis of Fig. 13 sheds light on the performance of the residual module CNN model within an intricate micro-environment. When employing cross entropy as the loss function, the model achieves an accuracy of 0.85, a precision of 1, an IoU score of 0.8, and a FPS value of 28.4. Conversely, when utilizing MSE as the loss function, the model attains an accuracy of 0.97, a precision of 0.99, an IoU score of 0.75, and a FPS value of 28.6. The model employing both MSE and cross entropy as loss functions achieves an accuracy of 0.98, a precision of 0.99, an IoU score of 0.83, and an FPS value of 29.9. Comparative analysis reveals that the residual module CNN model employing a combination of MSE and cross entropy as loss functions excels in precision, accuracy, IoU scores, and FPS values. This result is attributed to the distinct effects that different loss functions impart on the model’s optimization process, ultimately striking a balance between accuracy and other metrics.

Analysis of target image detection results of the CNN algorithm model based on the attention mechanism

This article strategically employs a CNN model based on the attention mechanism for comparative experiments to explore the effect of this novel technology on image target detection within the IoT framework. Specific comparative results are presented in Fig. 14, which illustrates the juxtaposition of detection accuracy and detection time across various CNN algorithmic models and those integrating attention mechanisms in target image detection.

Upon scrutinizing Fig. 14, an examination of the accuracy comparison results for different algorithmic models reveals intriguing insights. It is apparent that the CNN model’s detection accuracy, when exposed to multimodal information, is notably the lowest, registering at merely 14.5%. By contrast, CNN models employing non-maximum suppression or real-time monocular 3D detection networks exhibit relatively similar detection accuracies, standing at 19.72% and 20.1%, respectively. Remarkably, the CNN model operating within the depth-constant network framework demonstrates outstanding detection performance, boasting an impressive accuracy rate of 24.59%. Significantly, the CNN model reported here, fortified with an attention mechanism, yields substantial improvements in detection efficacy, showcasing a 0.27% accuracy enhancement compared to the CNN model within the depth-invariant network, which claims the second-best performance. Under the medium and challenging difficulty levels, the accuracy ranking among models remains consistent, with the CNN model incorporating an attention mechanism, maintaining accuracy rates of 17.8% and 14.77%, respectively. Transitioning to the assessment of detection times of various algorithmic models, it becomes evident that CNN models predicated on depth-invariant networks, real-time monocular 3D detection networks, non-maximum suppression, multimodal information, and real-time multimodal 3D share nearly identical detection times of 0.03 s. Conversely, the CNN model based on an attention mechanism slightly extends its detection time to 0.05 s. In summary, the CNN model leveraging an attention mechanism exhibits superior detection performance and higher accuracy in the context of IoT target image analysis. However, as image complexity increases, the model’s accuracy experiences a corresponding decline. Furthermore, a trade-off relationship exists between detection accuracy and processing time, where algorithms achieving higher accuracy invariably incur extended processing times.