Evaluating deep learning variants for cyber-attacks detection and multi-class classification in IoT networks

Contribution

This research aimed to identify cyberattacks in IoT environments using deep-learning models on the CICDIoT2023 dataset. The research’s main contribution is given as follows.

• This research proposed multiple variants of deep learning models to detect cyberattacks in realistic IoT environments on various samples of network stream packets.

• This study contributes to implementing an IoT cyberattacks dataset from CICIoT Dataset 2023, an actual dataset and standard for massive attacks in the Internet of Things environment. This research utilizes 47 features and 33 attacks for assessment.

• This research uses robust scalar and label encoding techniques to preprocess the data and compare variants of deep learning models (DNNs, CNNs and RNNs).

• The research evaluates the performance of the cyberattack detection system using different evaluation metrics. The outcomes show that the proposed approach is effective in accurately predicting cyberattacks.

Organization

The article’s organization is as follows: “Related Work” contains a study on relevant ML and DL techniques for cyberattack detection. “Proposed Methodology” describes the proposed approach that uses the CIC IoT dataset, data preparation, and deep learning models. “Results and Discussion” explains and discusses the findings. “Conclusion and Future Scope” contains the study’s conclusion and recommendations for further investigation.

Experimental dataset

This study used the CICIoT2023 Dataset, an actual dataset and benchmark for massive attacks in the Internet of Things environment. There are two distinct varieties of files for the CICIoT2023 dataset: pcap and csv. The initial data generated and gathered in various scenarios inside the CIC IoT network is in pcap files. All sent packets are included in these files, which can be used to design additional functionality. A fixed-size packet window summarizes the features taken from the original pcap files and is added to those files. A series of packets delivering data between two hosts is used to extract the features (Neto et al., 2023).

This research utilizes the 47 features for assessment. Using the retrieved features, we aggregate the values recorded in intervals size of 100 (Mirai UDPPlain, Mirai Greeth Flood, Mirai GREIP Flood, DoS UDP Flood, DoS TCP Flood, DDoS UDP Flood, DoS HTTP Flood, DDoS TCP Flood, DoS SYN Flood, DDoS UDP Fragmentation, DDoS SYN Flood, DDoS HTTP Flood, DDoS SynonymousIP Flood, DDoS RSTFIN Flood, DDoS ICMP Fragmentation, DDoS ICMP Flood, DDoS SlowLoris, DDoS ACK Fragmentation and DDoS PSHACK Flood) and 10 (Dictionary brute force, Ping Sweep, Vulnerability Scan, Host Discovery, Backdoor Malware, Command Injection, OS Scan, SQL Injection, DNS spoofing, XSS, Browser Hijacking, Uploading Attack, MITM ARP spoofing, Benign Traffic and Port Scan) packages to reduce fluctuating data sizes (such as Command_Injection and DDoS). Subsequently, the csv file dataset is used for this research. The generated csv datasets indicate the features of every data block. Additionally, each attack used in this research has unique properties. For instance, a DDoS assault generates more network streams than a spoofing assault, usually less (Neto et al., 2023).

Data preprocessing

The model’s performance is enhanced, and more accurate features are produced due to the critical data preparation phase. In this stage, categorical data is preprocessed into integer values using a robust scalar and label encoding technique.

Robust scalar: This study uses the data preparation method RobustScaler. Compared to other scaling techniques like the StandardScaler or MinMaxScaler, which are more susceptible to outliers, the RobustScaler is specifically created to scale the characteristics of a dataset. The RobustScaler scales the data utilizing the median and Interquartile range (IQR), as opposed to the mean and standard deviation used by the StandardScaler. The median is less dependent on outliers than the mean, while the IQR is a measure of the data’s dispersion that is likewise less affected by outliers.

Label encoder: Label encoding renders the input numerical labels into a machine-learning algorithm (Sharma et al., 2020).

Data splitting: After performing the data preprocessing steps, the models are built. The data is divided into training and testing data with a ratio of 80% training data and 20% testing data to enhance accuracy and efficacy for this phase. After splitting, the model is trained using the different variants of the deep learning model.

Deep neural network

This study inspected DNN’s aptitude for spotting cyberattacks in IoT settings. According to Deng & Yu (2014), the deep learning method incorporates the learning area that employs irregular data in numerous stages through organizational frameworks. Deep learning synthesizes graphical design, neural networks, and pattern recognition. The deep learning model performs well for prediction on large data sets. According to Huang et al. (2013), DNNs are exceptional to other machine learning classifiers. The proposed deep learning technique examines the organic patterns on a sample of network stream packets.

Additionally, deep learning logically supports multi-task training, which uses a single-layer deep neural network to consider all building forms’ properties. This network mainly consists of self-learning units with two or more layers. DNN uses hidden units among the input and output layers. The hidden unit p can use a logistic function to translate the scalar variable $y_q$ of subsequent layers to the input $x_q$ below. In a DNN network, (4) and (5) can predict the output of $i$th neuron $y_i$ as following Eqs. (1) and (2):

(1) $$y_i=f({\xi }_i),$$

(2) $$f({\xi }_i)=\vartheta +{\mathrm{\Sigma }}_{h\epsilon {\tau }_i}-1W_i{X}_j,$$

$f({\xi }_i)$ represents the transfer function and ${\xi }_i$ is the potential of $i$th-neuron. The transfer function is shown in the following Eq. (3):

(3) $$f({\xi }_i)=\frac{1}{1+exp(-{\xi }_i)}$$

The sum of squared errors can represent the entire objective cost function where the output neurons are used to determine the target values, $y_o$, and $\widehat{y_o}$ as shown in Eq. (4).

(4) $$C=\mathrm{\Sigma }1/2(y_o-\widehat{y_o})$$

Different variants of DNN models are utilized for this research. This study employs a sequential DNN1 algorithm with a single input layer. The first input layer has 256 units and employs the relu activation method. The hidden layer comprises two dense and one dropout layer. The dense layer comprises 256 and 34 units with relu and softmax activation functions, and the dropout layer has 256 units. The output layer is the next activation function, relu and softmax, used to tackle the categorical categorization obstacle.

For the next DNN variant model, this study uses the sequential DNN2 model. The input layer is 1 with 256 units and employs the same method as DNN1. The hidden layer comprises two dense and two dropout layers. The dropout layers comprised 256 and 128 units. The dense layer contains 128 and 34 units with relu and softmax activation functions, respectively. This research uses the sequential DNN3 algorithm with one input layer as a third DNN variant model. The input layer is 1 with 256 units and employs a similar method as DNN1. The hidden layer is next, which comprises three dense and three dropout layers. The dropout layers comprised 256, 128 and 64 units. The dense layer contains 128, 64 and 34 units with relu and softmax activation functions, respectively. The DNN model has utilized Adam as an optimizer to compute and decrease the damage utilizing sparse_categorical_crossentropy.

Convolutional neural network

Convolutional neural network(CNN) is a subclass of deep neural networks that have demonstrated exceptional performance in several computer vision applications. This research uses the CNN model for text classification. The CNN model comprises three consequential layers: a max-pooling layer, a convolutional layer, and a fully connected layer. Different variants of CNN models are utilized for this research. This study uses a sequential CNN1 model with a single input layer. The model has 10 layers: one convolutional layer, three Leaky_ReLU layers, one max-pooling layer, one dropout layer, three dense layers, and one flattened layer. The convolutional block is merged with a 1D convolutional neural network, one Leaky_ReLU layer, one max-pooling layer, and a dropout layer with a 30% dropout rate. The consequent attribute maps tend to be flattened after the convolutional and pooling layers. Following flattening, the attributes proceed through two Leaky_ReLU layers, three dense layers with 64, 32, and 34 units using the softmax function.

This research uses the sequential CNN2 model as a second CNN variant model. The model has 14 layers: two convolutional layers, four Leaky_ReLU layers, two max-pooling layers, two dropout layers, three dense layers, and one flattened layer. The convolutional block is merged with a 1D convolutional neural network, Leaky_ReLU layer, max-pooling layer, and a dropout layer with a 30% dropout rate. After the convolutional and pooling layers, subsequent attribute maps often become flat. Following flattening, the attributes proceed through two Leaky_ReLU layers, three dense layers with 64, 32, and 34 units using the softmax function.

This study uses the sequential CNN3 model for the next CNN variant model. The model has 18 layers: three convolutional layers, five Leaky_ReLU layers, three max-pooling layers, three dropout layers, three dense layers, and one flattened layer. The convolutional block is merged with a 1D CNN, Leaky_ReLU layer, max-pooling layer, and a dropout layer with a 30% dropout rate. The consequent attribute maps tend to be flattened after the convolutional and pooling layers. Following flattening, the attributes proceed through two Leaky_ReLU layers, three dense layers with 64, 32, and 34 units using the softmax function. The output layer employed the Adam optimizer in the final section to compute and reduce the loss employing categorical_crossentropy.

Recurrent neural network

A recurrent neural network (RNN) is a genre of neural network architecture constructed to work with data sequences. RNNs retain a hidden state that gathers data from earlier time steps in the sequence, unlike standard feedforward neural networks, which analyze each data point independently (Alzubaidi et al., 2021).

Various variants of RNN models are utilized for this research. This study uses a sequential RNN1 system with a single input layer. The shape of the input layer is 1 with 32 units, and the relu activation function is used. The dropout regularisation is used with the input layer. Next, add the output layers comprising one flattened layer, three dense layers and two Leaky_ReLU layers. The dense layer with 34, 16 and 32 utilizes the activation functions softmax and a fully connected layer.

The sequential RNN2 algorithm that utilizes a single input layer is the following RNN variant. The input layer used the same method and function as RNN1 variants. The dropout regularisation is used with the input layer. Next, add the second RNN layer with 32 units and use the dropout regularisation layer. The output layers comprise one flattened layer, three dense layers and two Leaky_ReLU layers. The dense layer with 34, 16 and 32 utilizes the activation functions softmax and a fully connected layer.

This study uses the sequential RNN3 model with one input layer as a third DNN variant model. The input layer used the same method and function as RNN1 variants. The output layers comprise one flattened layer, three dense layers and two Leaky_ReLU layers. The dense layer with 32, 16, and 34 uses the softmax activation functions and is a fully connected layer. The output layer employed the Adam optimizer in the final section to compute and reduce the loss employing categorical_crossentropy.

Algorithm 1 illustrates the method of the proposed model for cyberattack detection. A proposed method for anticipating cyberattacks in an IoT environment using deep learning techniques is described in the presented methodology. The algorithm proceeds by specifying the input as cyberattack data and the intended output as predictors of cyberattacks. We propose a function called $CICIoTDataset$ that performs data preprocessing operations on the data, such as employing the RobustScaler for scaling features and label encoding for categorical labels to a numerical form. The processed data is returned with the symbols $x$ (features) and $y$ (labels). The function known as $TrainModel$ is defined. To do so, the data are divided into training and testing sets ( $x_{t}rain,x_{t}est,y_{t}rain,y_{t}est$). Next, three different kinds of NN models are built: a DNN, a CNN, and a RNN. It then returns the created $model$. The DNN and RNN model used the hyperparameter $keras.backend.clear\mathrm{\_}session()$ and hidden initializer $random\mathrm{\_}uniform(SEED)$. It is explained how $DDoSPrediction$ works. An iterative process involves a specified number of epochs $E\mathrm{\_}p$ and batch sizes $B\mathrm{\_}s$. The model is applied to generate predictions (x) from the input data $Data$ for every epoch and batch size. The $crossentropy$ loss is determined between the actual labels X and the expected results. After that, the algorithm outputs the expected outcomes.

Deep neural network

The results of a study on the performance of different deep neural network (DNN) models are presented in Table 2. Several significant performance measures were used in the evaluation to determine these models’ effectiveness in classification tasks. The table thoroughly summarizes the findings, facilitating comprehension of each model’s performance on 33 cyberattacks. Model DNN1 achieved 87.42% accuracy, 86.94% precision, 87.42% recall, and 86.26% F1-score. Model DNN2 demonstrated 84.73% accuracy, 85.69% precision, 84.73% recall, and 84.39% F1-score. Model DNN3 exhibited 88.64% accuracy, 91.20% precision, 88.64% recall, and an F1-score of 88.51%. Concerning accurate classification, handling false positives and negatives, and achieving a balance between accuracy and recall, these measures collectively provide insights into the effectiveness of each DNN model. In contrast to the other models, Model DNN2 performs poorly in accuracy, with Model DNN3 emerging as the best model in precision and F1-score.

Figure 2 visualizes the results of the DNN1 model. Figure 2A graph illustrates the accuracy for both training and validation. The training accuracy was $0.676\mathrm{\%}$ at the $0^{th}$ epoch and ranged among drops and boosts until roughly $0.851\mathrm{\%}$ at the end. Validation accuracy starts at $0.775\mathrm{\%}$ at the $0^{th}$ epoch and varies among losses and gains until it comes near $0.875\mathrm{\%}$ at the last epoch. Figure 2B loss curve shows the training and validation loss. training loss started at $0.799\mathrm{\%}$ and fell to $0.01\mathrm{\%}$ at the last epoch. Validation loss started at $0.0299\mathrm{\%}$ at the $0^{th}$ epoch and fell to $0.01\mathrm{\%}$ by the ${30}^{th}$ epoch.

Figure 3 demonstrates the results of the DNN2 model. Figure 3A graph displays the training and validation accuracy. the training accuracy was $0.651\mathrm{\%}$, and it varied between drops and gains until about $0.848\mathrm{\%}$ at the $0^{th}$ epoch. Validation accuracy ranges among losses and gains until it peaks at $0.848\mathrm{\%}$ at the last epoch. Figure 3B graph displays the training and validation loss. Training loss decreased to $0.10$, and validation loss remained $0.001\mathrm{\%}$.

Figure 4 represents the outcomes of the DNN3 model. Figure 4A graph represents the training and validation accuracy. The training accuracy peaked at about $0.87\mathrm{\%}$ accuracy at the last epoch. Validation accuracy peaked at $0.875\mathrm{\%}$ accuracy at the last epoch. Figure 4B presents the training and validation loss. Training loss decreased to $0.01$ at the last epoch. Validation loss remains the same $0.01\mathrm{\%}$ at the last epoch.

Convolutional neural network

Table 3 demonstrates the performance results of various convolutional neural network (CNN) models in a classification assessment of 33 cyberattacks in an IoT environment. Experiments have been conducted on CNN1, CNN2, and CNN3 models. The proportion of effectively classified occurrences among all instances is how accurately a classification is made. CNN3 exhibited the highest accuracy, scoring 96.37%, ahead of CNN2 (94.30%) and CNN1 (95.49%). Precision measures the percentage of precise positive predictions of favorable outcomes. CNN3 had the best precision, scoring 96.15%, while CNN2 and CNN1 followed in at 94.74% and 95.17%, respectively. A recall measures the proportion of real positive instances correctly anticipated as positive. Recall for CNN3 was 96.37%, CNN1 was 95.49%, and CNN2 was 94.30%. The F1-score is a balanced indicator of a model’s performance because it is the harmonic mean of precision and recall. CNN1 had an F1-score of 94.48 percent, CNN3 had a 95.51%, and CNN2 had 93.36%. Based on these outcomes and the offered assessment metrics, the CNN3 model performs best for cyberattack detection.

Figure 5 demonstrates the results of the CNN1 model. Figure 5A shows the training and validation accuracy graph. The training accuracy was $0.68\mathrm{\%}$ at the $0^{th}$ epoch and fluctuated between falls and increases until roughly $0.94\mathrm{\%}$ at the ${30}^{th}$epoch. Validation accuracy starts at $0.76\mathrm{\%}$ at the $0^{th}$ epoch and varies between gains and losses until it comes near $0.95\mathrm{\%}$ at the ${30}^{th}$ epoch. The graph in Figure 5B represents the training and validation loss. Training loss is about $0.01$ at the last epoch. Validation loss decreased to $0.01\mathrm{\%}$ at the last epoch.

Figure 6 represents the outcomes of the CNN2 model. Figure 6A depicts the training and validation accuracy graph. Training accuracy rises to $0.94\mathrm{\%}$, and validation accuracy is $0.73\mathrm{\%}$. The training and validation loss is represented on the graph in Fig. 6b. Training loss decreases to $0.01$, and validation loss decreases to $0.01\mathrm{\%}$.

Figure 7 demonstrates the results of the CNN3 model. The training and validation accuracy is represented by the graph in Fig. 7a. The training accuracy is $0.90\mathrm{\%}$; following several cycles of gains and losses, it goes up about $0.955\mathrm{\%}$. Validation accuracy is $0.94\mathrm{\%}$. The training and validation loss is depicted on the graph in Fig. 7b. Training loss decreased to $0.13$, and validation loss peaked at about $0.10\mathrm{\%}$.

Recurrent neural network

The comparison analysis of three RNN models, RNN1, RNN2, and RNN3, is shown in Table 4. The RNN1 model had a 96.52% accuracy rate. It had a high level of precision (96.25%), showing that a substantial amount of the positive predictions were accurate. A sizable percentage of effective captures of positive experiences is indicated by the recall value (96.52%). The F1-score (95.73%) indicates an adequate balance between recall and precision, letting the model perform well. A 96.01% accuracy was attained using the RNN2 model. A good capacity for making accurate positive predictions is indicated by the precision number (95.77%). This model detected many positive cases, with a recall value of 96.00%. The F1-score (95.65%) indicates a balanced trade-off between recall and precision, which adds to its strong performance. The RNN3 model had a 95.89% accuracy rate. The accuracy (95.60%) shows a noteworthy capacity for making precise optimistic predictions. The model has successfully detected several positive instances, as indicated by the recall value (95.89%). The F1-score (95.03%) shows that precision and recall are often well-balanced.

Figure 8 represents the results of the RNN1 model. Figure 8A graph represents the training and validation accuracy. Training accuracy is about $0.60\mathrm{\%}$, and validation accuracy is $0.75\mathrm{\%}$. The training and validation loss is depicted on the graph in Fig. 8b. Training loss seems to decrease to $0.01$, and validation loss to $0.01\mathrm{\%}$.

Figure 9 demonstrates the results of the RNN2 model. The training and validation accuracy is represented by the curve in Fig. 9a. The training accuracy reaches about $0.94\mathrm{\%}$ accuracy at the last epoch. Validation accuracy increases to $0.95\mathrm{\%}$ accuracy at the last epoch. The training and validation loss is depicted on the graph in Fig. 9b. Training loss started from $0.24\mathrm{\%}$ at $0^{th}$ epoch and declined to $0.10$, and validation loss reached about $0.10\mathrm{\%}$.

The RNN3 model is visualized in Fig. 10. Figure 10A graph represents the training and validation accuracy. The training accuracy is about $0.95\mathrm{\%}$ accuracy at the last epoch. Validation accuracy is $0.71\mathrm{\%}$ at the start and increases at $0.95\mathrm{\%}$ accuracy at the last epoch. Figure 10B shows the training and validation loss. Training loss decreased to $0.10$, and validation loss declined to $0.10\mathrm{\%}$.

Comparison with existing study

Table 5 presents a comparison with the benchmark study (Neto et al., 2023). It can be seen that Neto et al. (2023) have an accuracy of 99.43%, a precision of 70.54%, a recall of 91.05%, and an F-score of 71.92%, whereas the proposed approach attained an accuracy of 96.52%, a precision of 96.25%, a recall of 96.52%, and an F-score of 95.73%. In this context, the Proposed Approach outperforms the base article regarding accuracy, precision, recall, and F-score, suggesting our approach is better for cyberattack detection in a realistic IoT environment.