LSTMDD: an optimized LSTM-based drift detector for concept drift in dynamic cloud computing

Introduction

Cloud computing is a paradigm that facilitates the provision of various services to users via the internet (Zaharia et al., 2011; Saraswat & Tripathi, 2020; Jangjou & Sohrabi, 2022). It offers five essential characteristics: on-demand self-service, ubiquitous network access, resource pooling, rapid elasticity (enabling flexible resource allocation based on demand), and measured services (Puthal et al., 2015; Sunyaev & Sunyaev, 2020; Kamanga et al., 2023). Thus, cloud providers face numerous challenges in ensuring high-quality services.

Efficient resource utilization is also a significant challenge that can directly impact energy consumption (Bharany et al., 2022; Rahimikhanghah et al., 2022; Mansouri et al., 2023). Despite the various methodologies available, this field of study is still emerging due to heterogeneous workloads that fluctuate with cloud-supported services. To address this problem, cloud providers are progressively implementing advancements in strategies like virtualization and server consolidation to maximize resource efficiency and minimize energy usage. Cloud usage prediction aims to allocate resources and adjust server capacity more efficiently. By predicting demand, cloud providers can reduce idle or underutilized servers, optimize energy consumption, manage costs, reduce carbon footprints, and enhance service quality. Therefore, adopting green and sustainable computing practices represents the future direction for cloud providers. According to a study conducted by Google, the energy consumption attributed to idle resources within data centers accounts for over 50% of the total energy usage (Bohrer et al., 2002; Barroso & Hölzle, 2007). Notably, energy consumption scales linearly with resource utilization (Fan, Weber & Barroso, 2007; Reiss et al., 2012). Underutilized or zombie servers pose a significant concern for cloud providers regarding their reputation, as they can result in resource inefficiency and a decline in service quality. Furthermore, implementing proactive monitoring systems can aid in identifying and decommissioning zombie and underutilized servers, promoting a more sustainable and dependable cloud infrastructure.

Organizations are increasingly embracing cloud computing because of its ability to provide resources rapidly and its pay-as-you-go pricing model. Consequently, the workload associated with cloud adoption varies across different organizations (Badshah et al., 2023; El-Rashidy et al., 2023). The workload usage traces generate log data rapidly and exhibit high dynamism. In order to comprehend changes in distribution, it is imperative to analyze these usage patterns, commonly known as concept drift (Lu et al., 2018). The accurate prediction of efficient resource usage is of utmost importance; however, the presence of drift poses a significant challenge to the model’s performance. Utilizing a static model is not a viable solution due to the impact of non-stationary traces over an extended duration on the model’s performance. Hence, using an online learning model becomes necessary to address the phenomenon of concept drift (Agrahari & Singh, 2022). Although online learning models offer advantages in adapting to drift in distribution, they also entail certain drawbacks. The stability-plasticity dilemma can be conceptualized as a dynamic balance between two fundamental aspects: stability, which involves the retention and preservation of valuable knowledge, and plasticity, which necessitates the capacity to effectively adapt to new and unfamiliar information (Bayram, Ahmed & Kassler, 2022). It is crucial to maintain a careful equilibrium in actively identifying changes to prevent needless adjustments to the model that can result in higher computing costs while considering domain expertise.

Monitoring & control and analytic & diagnostic are two crucial concept drift domains covered in the article (Žliobaitė, Pechenizkiy & Gama, 2016). Monitoring and control involve overseeing and regulating tasks and sudden changes caused by complex processes and sensor data input. Diagnostics and analytics often use relational data analysis and observe gradual drift; abrupt changes are rare. After reviewing the literature, it can be concluded that cloud usage traces can be classified in both categories. In the publications mentioned earlier, authors (Fehling et al., 2014; Amiri, Mohammad-Khanli & Mirandola, 2018) have examined the patterns of cloud workload and noted that they can be categorized as static, periodic, one-time occurrences, unexpected, and subject to constant change as a result of user routines. Hence, the assertions above indicate sudden and gradual shifts in cloud workload patterns. Cloud behavior can best be observed with cluster usage traces that allow work on real workloads and their operational complexity. Google released two public usage traces in 2014, and 2019 represents the best benchmark for cloud research. In Chen et al. (2014), this dataset was used to design the model mentioned above and is considered a baseline for performing this research. The Kolmogorov-Smirnov (KS) test detects drift in the dataset. A KS statistical test was applied to the Google usage trace to see drift in the distribution with positive results (dos Reis et al., 2016). In Žliobaitė, Pechenizkiy & Gama (2016) the article, they performed statistical analysis on Google usage trace and identified a multi-model and skewed distribution. We conducted a normal probability plot on the CPU and memory resources of Google data, as shown in Fig. 1. It is a non-Gaussian distribution that does not follow the red line. Moreover, the cloud-generated data stream is time-critical for prediction, mainly because of the changing speed of the data (Gonçalves et al., 2014). There are three significant types of drift concerning overtime, termed abrupt, gradual, and reoccurring (Webb et al., 2016; Khamassi et al., 2018). The abrupt changes are sudden drifts, while gradual drifts are over a more extended period. The reoccurring drifts repeat themselves after time intervals. Real drift refers to changes in only the label class of data concerning time. In contrast, changes in the distribution of input features are the result of virtual drift (Webb et al., 2016). Changes in the distribution of usage load concerning time are called sudden and gradual drift. This drift could be related to any resource, while our focus is the virtual drift.

The problem statement concerns monitoring and detecting concept drift in cloud resource usage, particularly emphasizing non-Gaussian continuous resource values. Drift detectors based on data distribution, such as (Webb et al., 2017), do not provide a constant quantitative measure of drift, whereas (Yu & Webb, 2019) limits the restriction to the type of distribution. A specialized concept-drift mapping technique is required to address this challenge. The article’s primary motivation is the model’s involvement in the drift detector to gain more information about drift occurrences. This can help create smaller and optimum buffers for cloud data streams. The article focuses on detecting drift using past knowledge of data distribution in continuous data flow. Note that this updated research question emphasizes detecting and adapting to concept drift in the non-Gaussian distribution in dynamic cloud computing environments and proposes using a drift detector to facilitate proactive retraining. We also compare the performance of existing drift detectors for sudden and gradual drift. Finally, we offer a framework for drift detection to handle the dynamic cloud environment. The framework adopts the changes and learns from the data series to facilitate cloud providers by keeping the following points in mind: How can we effectively detect concept drift in non-Gaussian distributed cloud computing environments and manage adaptive retraining of machine learning models to maintain high accuracy? Specifically, we propose incorporating a focus-based LSTM model with class weight to address the class imbalance in the data and applying a genetic algorithm-based hyperparameter tuning approach to optimize model performance. Additionally, the proposed drift detector can monitor changes in the distribution of cloud computing data and trigger retraining based on the drift when necessary.

The key contributions of the proposed framework are listed below:

• The objective of this study is to discover the optimum drift detector within the context of cloud computing by utilizing usage trace data obtained from Google.

• The examination of individual drift detectors within the context of sudden and gradual drift occurrences.

• The proposed approach integrates LSTM models, attention mechanisms, and genetic algorithms to tackle the problem of concept drift detection in non-Gaussian cloud environments.

• Evaluating the performance of the proposed approach against existing state-of-the-art methods for detecting concept drift in cloud environments. To help establish the proposed approach’s relative effectiveness and highlight areas for further improvement.

The following sections are in this article. In “Literature Review”, the literature review covers different types of existing drift detectors. The materials and method are covered in “Materials and Method”. The dataset subsection explains the type of dataset used. Then, the following subsection proposes an architecture and model for the cloud drift detector. The results and discussion “Conclusion” with evaluation setup, parameter configuration, and results are discussed. The conclusion summarizes the findings and presents future work.

Literature review

The utilization of online learning is increasingly prevalent in various domains due to the vulnerability of systems to concept drift. Multiple factors, such as modifications in user behavior, environmental changes, or alterations in the system itself, can trigger concept drift. The occurrence of concept drift can significantly degrade the performance of machine learning systems. Therefore, detecting and adapting to concept drift is crucial. This literature review explores the types of concept drift, methods for detecting and adjusting to it, challenges faced, and future research directions in this field.

Two primary methodologies can be employed to demonstrate the occurrence of drift in our data. One approach involves using visualization techniques to discern patterns, while another involves applying drift detector algorithms. In the literature, detectors are classified into three categories: Sequential analysis-based statistical, Data distribution-based drift (DDB), and Learner output-based drift detection. Both sequential analysis-based and data distribution tests employ statistical tests to identify the presence of drift. Sequential analysis-based statistics concentrate mainly on analyzing changes in probability within sequential data, while data distribution-based statistics compare the distributions of two distinct time windows. Two contributions can be compared using either parametric or non-parametric tests. Non-parametric methods are advantageous due to their reduced reliance on prior knowledge of the Probability Density Function(PDF) of data, enhancing their applicability in real-world scenarios. However, this characteristic also renders them less responsive to slight variations.

The concept drift detector uses the results of the base learner. Drift detector selection is based on the drift type in the distribution. It can be used with an individual or an ensemble of classifiers. These methods mostly use a base learner to detect the drift. The drift Detection Method (DDM) (Gama et al., 2004), Early Drift Detection Method (EDDM) (Baena-Garcıa et al., 2006), Adaptive Windowing (ADWIN) (Bifet & Gavalda, 2007), Statistical Test of Equal Proportions(STEPD) (Nishida & Yamauchi, 2007), EWMA for Concept Drift Detection (ECDD) (Ross et al., 2012), DDMs Based on Hoeffding’s (HDDMA) (Frías-Blanco et al., 2015), DDMs Based on Hoeffding’s (HDDMW) (Frías-Blanco et al., 2015), SEED Drift Detector (SEED) (Huang et al., 2014), Sequential Drift (SeqDrift) 1 and 2 (Pears, Sakthithasan & Koh, 2014), and Reactive Drift Detection Method (RDDM) (Barros et al., 2017) are drift detection techniques discussed and compared below in Table 1.

Drift detectors are further classified into three main categories based on their working methodology. The earner output-based drift detection method uses the base learner’s output to detect drift. DDM and EDDM are the most basic learner-based drift detections. It has typically been developed considering classification problems only. When concept drift is detected, they do not provide information about the data necessary to adapt the learner to the new environment. The sequential technique can only deal with a single time series and regression.

Page-Hinkley and H-ICI-based CDT is the most basic used in sequential analysis learners. Its concept drifts over a single time series. Data distribution only works with variable correlation but requires unnecessary model adaptation. Data distribution can help both deal with regression and multivariate data. It can also be used to consider the correlation between attributes; thus, it can extract more detail about the drift. It can further help adopt much data with drift Magnitude, frequency, and duration. The Concept Drift Map is from a Data distribution-based drift detector. It can detect concept drifts of variables irrelevant to the learner output and cause unnecessary ML model adaptations. A concept drift map is more like a visualization technique than a detection one. Table 2 summarizes the characteristics of all three categories of the drift detection technique. Most of the latest work has recently been done in data distribution-based drift detection.

• Drift Detection Method (DDM): It comes under the category of a learner-based drift detector. A drift detector that utilizes a base learner as its fundamental component. The base learner is employed to make predictions. The error rate is computed by using the classification outcomes. The error rate and standard deviation determine the estimation of drift warnings and dirt occurrences. The number of errors exhibits an inverse relationship with the accuracy of the classification. It is most effective in cases of sudden drift. Let $s_i$ be the value of the standard deviation $i$ at a given time, $p_i$ be an error rate that depends on $i$, $p_{min}$ be the minimum value of $p_i$, and $s_{min}$ be the minimum value of $s_i$. Then the following drift warning Eq. (1) holds:

  (1) $$(p_{min}+2\cdot s_{min})\le p_i+s_i$$

  Then, the following drift detection Eq. (2) holds:

  (2) $$p_{min}+(3\cdot s_{min})\le p_i+s_i$$

• Early Drift Detection Method (EDDM): It is also a learner-based drift detector. Calculate the distance error rate based on the classification results. The number of examples between the two classification errors means that the drift’s distance error rate increases in the case of no concept. The drift warning and occurrence are observed based on the error rate and standard deviation. It works best for a slow, gradual drift. Let $p_i$ be the distance error rate and $s$ be the standard deviation. EDDM is calculated using the following Eqs. (3) and (4):

  (3) $$\frac{(p_i+2)}{(p_{max}+(2\cdot s_{max}))}<\alpha , (\mathrm{D}\mathrm{r}\mathrm{i}\mathrm{f}\mathrm{t}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{n}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n})$$ (4) $$\frac{(p_i+2s_i)}{(p_{max}+(2\cdot s_{max}))}<\beta , (\mathrm{D}\mathrm{r}\mathrm{i}\mathrm{f}\mathrm{t}\mathrm{o}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n})$$where $\alpha$ is 0.95 and $\beta$ is 0.9.

• Adaptive Windowing (ADWIN): Its name clearly shows that it uses the sliding window concept and compares two distributions for change. It opens the older window when the change is detected. The window length has been increased based on the fact that no changes have occurred. Two sub-windows are selected to measure the difference.

• Statistical Test of Equal Proportions (STEPD): It separately calculates the accuracy of all recent instances to compare two windows. It notifies the presence of the drift by checking for a significant decrease in inaccuracies over the period. The Chi-square test and standard deviation percentile are used to calculate the significance level. A value below the significance level will indicate the drift and drift warning concepts.

• EWMA for Concept Drift Detection (ECDD): EWMA stands for an exponential moving average. Probability is calculated based on the learner’s accuracy and the estimated time between false positive detection. Two estimates are calculated and compared. Estimate one is calculated with more weights on the recent examples. Estimate two considers equal importance for all data. The two estimates are compared to detect concept drift. The difference between the two estimates is compared with two thresholds. The first threshold is set to alarm the drift warning, and the second is to detect actual drift occurrence.

• DDMs based on Hoeffding’s (HDDMA & HDDMW): HDDMA is based on the moving average, and HDDMW is based on the weighted average. It does not need any assumptions about the PDF.

• SEED drift detector (SEED): SEED is based on Hoeffding’s inequality of ADWIN to work faster and more efficiently. It compares two sub-windows by comparing the average. In the case of a higher value, the previous sub-window is dropped.

• Reactive Drift Detection Method (RDDM):

  RDDM is proposed to overcome the problem of DMM and achieve higher global accuracy. It recalculates the drift warning and identification to remove the effect of previous instances.

The majority of research is devoted to deep learning algorithms for resource usage prediction, such as ensembles (Mehmood, Latif & Malik, 2018), time series (Gutterman et al., 2019), LSTM (Bi et al., 2021) and evolution algorithm (Malik et al., 2022). In Shu, Cai & Xiong (2021), Saba et al. (2023), Banerjee et al. (2023), cloud services are optimized using machine learning to improve resource usage. Deep learning approaches also visualize the drift in cloud (Mehmood & Latif, 2022). The most recent detectors in literature are the advancements of existing drift detectors. Later, a concept drift map was established to understand how drift occurs when all qualities are mentioned in Table 2. The drift mechanism’s efficiency depends on its domain and type. In Webb et al. (2017) understanding, a probability-based idea drift map using Naive Bayes covers the above qualities, but the discrete values make it insufficient for the cloud domain. Thus, continuous values must be binned before determining distance. Translation from a constant process to a categorical conclusion could be more efficient. Transcoding destroys vital data. Thus, discretizing the previous dataset may not work for the present distribution. To overcome strong data type assumptions, (Yu & Webb, 2019) an adaptive concept drift map for continuous values may be proposed. These compare the average mean and maximum values of two Gaussian distributions revealed drift. The research mentioned above highlights a research gap, allowing one to detect deviations when dealing with non-Gaussian distributions and continuous variables. A simple drift detector based on simple Gaussian statistical techniques might also not work perfectly for a complicated domain. Thus, a drift detector based on some knowledge can perform well for a high-intensity stream. In deep learning concept drift detection, resilient methods for changing data distributions are becoming more popular. Traditional ensemble learning and sliding window approaches are well-studied. The Meta-cognitive Online Sequential Extreme Learning Machine (Mirza & Lin, 2016) addresses domain knowledge, imbalanced datasets, and delayed versus quick concept transitions. However, approaches like Dynamic Extreme Learning Machine (Xu & Wang, 2017) and OS-ELM (Yang et al., 2019) stress data flexibility and measure the learner performance degradation. Deep learning models that need dynamic real-world environments fuel this trend. The distribution of real-world data will always differ from model training. The (Cai & Li, 2023) binary classifier detects comparable changes in image distribution when detecting out-of-distribution.

LSTM has been widely used for natural language processing tasks (Jang et al., 2020; Jain et al., 2020; Xie et al., 2020; Li, Xu & Shi, 2019; Zhai et al., 2023), but it has the potential as a drift detector for time-series data. Recent studies (Li et al., 2022; Wang, Qi & Liu, 2019; Fields, Hsieh & Chenou, 2019) worked on the drift-sensitive LSTM method. However, drift is the change that should be part of a model and should not be mitigated for better prediction. The LSTM mechanism can learn the temporal patterns of the data and detect changes in the data distribution, which is yet to be explored. Their promising results in other domains show their power in detecting concept drift in time-series datasets.

Materials and method

This section will cover the dataset, proposed model, parameter configuration, and evaluation mechanism. The dataset section will cover the synthetic dataset used, while cloud data will explain cloud usage traces. The proposed model shows how drift detectors can be used in the cloud. In the section on evaluation methods and techniques to compare the results of these drift detectors. In the subsection, the algorithm’s settings are defined in parameter configuration.

Cloud google dataset

Google provides the Google usage trace dataset for research purposes to enable a deeper understanding of the cloud’s underlying patterns. The trace exhibits a skewed distribution related to non-Gaussian distribution (Žliobaitė, Pechenizkiy & Gama, 2016). Job duration, network traffic caused by other applications running on the same host with similar parameters, and other factors may impact the drift process. The trace includes CPU, memory, and disk utilization as resources, and task mapping is done using job and machine IDs. CPU actual and requested usage is measured in the core count or core seconds/second, and memory and disk space are measured in bytes. The features used in drift detection include start and end times, unmapped and page cache memory usage, maximum memory usage, cycles per instruction (CPI), memory accesses per instruction (MAI), sampling rate, and aggregation type.

The Kolmogorov-Smirnov (KS) statistical test is a non-parametric method used on the Google dataset to detect drift in distribution. This test compares two distributions, and if the resulting p-value is significantly more than 0.05, the null hypothesis that the distributions are the same is accepted (dos Reis et al., 2016). A higher statistic value indicates a rejection of the null hypothesis. For better visualization, two CPU distribution samples, with one showing changes over time in Fig. 2 and the other comparing two windows of CPU distribution in Fig. 3, may be considered. A sudden drift was observed at the end of the plot, with a spike in CPU usage causing a significant change in the distribution.

Preparing the dataset for a forecasting model involves data pre-processing, which is conducted in three steps: normalization, handling of missing values, and qualitative coordinates. Regarding normalization, CPU, memory, and disk resource usage measurements are normalized based on the most significant capacity machine on the cluster, where the maximum value is represented as one and the lowest as 0 (Hu et al., 2014). Qualitative coordinates are used to transform real continuous values into a limited set of qualitative classes, particularly for measuring resource usage values, which are continuous and need to be categorized into a limited set of classes. Discretization is applied to convert a continuous set of values into a limited set of discrete values, given the difficulty in detecting which actual value (cores or bytes) refers to the normalized values.

Proposed model and architecture

In dynamic cloud environments, dealing with concept drift using an offline model becomes impossible. There are several types of workload patterns: static, periodic, one-time, sudden, unpredictable, and continuously changing. It will lead to gradual and sudden drifts that can occur in the cloud usage workload trace. However, data distribution-based drift detectors can detect drifts and maintain continuous quantitative measurement. They must be capable of providing a detailed description of the nature and form of ridges. Therefore, an online learning algorithm with a drift detector is required to handle different types of substances and update itself with non-stationary data. Using an outdated model can lead to incorrect resource utilization, so a new solution must be able to handle distribution changes optimally. However, a drift detector with a prediction mechanism has yet to be covered in the literature. LSTM models can store short- and long-term information, but most literature on drift detection is based on the current window size and does not utilize long-term details. Therefore, an online learning workload prediction model with a drift detector is needed to evaluate its accuracy from the cloud provider’s perspective of resource allocation and to improve cloud resource utilization. Finally, it is worth noting that cloud usage workload often has a non-Gaussian distribution (Žliobaitė, Pechenizkiy & Gama, 2016). Figure 4 presents the proposed model for drift detection in cloud data infrastructure. The study offers a framework consisting of three phases: Resource Usage Trace Generation Phase, Online Phase, and Offline Phase.

During the Trace Generation Phase, cloud users generate usage traces when requesting and utilizing cloud resources. These requests are then forwarded to the resource allocator to fulfill the client’s needs. The Resource Usage Monitor captures information such as the client’s timestamp, CPU, Memory, job arrival times, and other relevant data. The Resource Usage Trace maintains storage of these usage traces for further use.

The online learning mechanism has two phases, namely the online and offline phases, which include a data distribution analyzer and a drift detector. In the online step, the trace data is analyzed to make predictions and train a new model in case of sudden changes. The distribution analyzer creates the distribution of data chunks available in the buffer based on the window size. The new model is a base learner trained offline on buffered data. More simply, it is responsible for learning about drift window size, while the drift detector raises alarms when drift occurs. Sudden drift occurs within a shorter period compared to gradual drift. Detecting sudden drifts requires smaller window sizes, while gradual drifts require larger window sizes. In the event of a drift, the online mechanism, which consists of a live model accompanied by a data stream, will be replaced by a new model, also referred to as a fine-tuned model. The offline phase means retraining the new model on buffer data available for the online phase later. The threshold is set to detect the occurrence of gradual drift, and the data is collected until the drift is detected.

LSTM is a type of neural network that can store both short-term and long-term information, making it suitable for prediction in non-stationary environments. It comprises three gates—input, forget, and output—each with defined usage. The memory cell of LSTM is shown in Fig. 5, where the cell state $c_t$ contains both short and long memory. The input gate controls the amount of information that can pass from the input $x_t$. The forget gate decides the amount of past information retained, controlled by $f_t$. The output state $o_t$ decides the amount of information flow from $c_t$ to $h_t$. The hidden state $h_t$ predicts based on the output, input, and current cell state to make the final prediction. The training set calculates the prediction error and threshold for detecting gradual drift.

A new trace can be created based on the classifier prediction results, and a drift detector can be used to make a drift prediction. LSTM is known for overcoming the problem of vanishing gradient. Each cell design of LSTM is to deal with present and past information. In sudden and gradual drift, the main difference between them is time. Thus, LSTM is the best choice in case of gradual drift in time series data; it can maintain information for a more extended period and abrupt drift in a shorter period. The proposed model employs a sliding window approach to detect changes in the data. At each time step, the model trains an LSTM with hyperparameter tuning on a fixed window of historical data. The LSTM model is trained to predict the next value in the time series. The predicted value is compared with the actual value, and the divergence between the two is computed. If the divergence exceeds a threshold, the model is considered to have detected a drift in the data. The threshold is dynamically updated based on the divergence values observed in the historical data. The proposed Attention Mechanism-based LSTM with class weight is defined using the following steps summarized in the Algorithm 1:

Algorithm 1:

Attention mechanism-based LSTM with class weight.

Require: Training data $X_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and labels $Y_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$, testing data $X_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ and labels $Y_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$, batch size b, epochs e, number of LSTM units u, dropout rate d, learning rate α, class weight w, and optimizer o

Ensure: Trained model

1:     Procedure ATTENTIONLAYER ( $\mathrm{i}\mathrm{n}\mathrm{p}\mathrm{u}{\mathrm{t}}_{\mathrm{s}}\mathrm{h}\mathrm{a}\mathrm{p}\mathrm{e},\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{u}\mathrm{r}{\mathrm{n}}_{\mathrm{s}}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s}=\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e}$)

2:        $b\leftarrow$ Initialize biases with zeros

3:        return Weighted sum of input

4:     end Procedure

5:     Procedure CREATE_MODEL ( $\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}=u,\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}{\mathrm{t}}_{\mathrm{r}}\mathrm{a}\mathrm{t}\mathrm{e}=\mathrm{d},\mathrm{o}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{r}=\mathrm{o},\mathrm{l}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{n}\mathrm{i}\mathrm{n}{\mathrm{g}}_{\mathrm{r}}\mathrm{a}\mathrm{t}\mathrm{e}=\alpha$)

6:        LSTM layer with u units and input shape

7:        Attention layer

8:        Dropout layer with dropout rate d

9:        Dense layer with sigmoid activation

10:       Compile model with binary cross-entropy loss and optimizer o with learning rate α

11:        return Compiled model

12:     end Procedure

13:      $model\leftarrow$ CREATE_MODEL( $\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{t}\mathrm{s}=u,\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}{\mathrm{t}}_{\mathrm{r}}\mathrm{a}\mathrm{t}\mathrm{e}=\mathrm{d},\mathrm{o}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{r}=\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{m},\mathrm{l}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{n}\mathrm{i}\mathrm{n}{\mathrm{g}}_{\mathrm{r}}\mathrm{a}\mathrm{t}\mathrm{e}=\alpha$)

14:      $model.\mathrm{f}\mathrm{i}\mathrm{t}(X_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}},Y_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}},\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}{\mathrm{h}}_{\mathrm{s}}\mathrm{i}\mathrm{z}\mathrm{e}=\mathrm{b},\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{c}\mathrm{h}\mathrm{s}=\mathrm{e},\mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}{\mathrm{s}}_{\mathrm{w}}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}=\mathrm{w})$

15:      return Trained model

DOI: 10.7717/peerj-cs.1827/table-10

Results and discussion

Result analysis

These experiments are repeated with the same configuration, and results are summarized in the remaining Table 5. Each drift detector is executed to observe its performance on each sudden and gradual dataset. DDM is renowned in research for the detection of sudden drift. Abrupt drift occurs suddenly and changes the data in a short period. In experimentation, it has also performed well in sudden and gradual drift cases. In the case of abrupt and gradual drift, EDDM performed the worst for all the datasets, although it tends to perform well for gradual drift. Distance between the classification error means.

Table 5:

Drift detectors average error.

Dataset	DDM	EDDM	ECDD	STEPD	Adwin	HDDMA	HDDMW	LSTMDD
Sine	0.016	0.047	0.031	0.0312	0.0156	0.251	0.282	0.106
Stagger	0.718	4.583	0.721	0.616	0.721	0.746	0.737	0.086
Mixed	0.499	1.655	0.498	0.464	0.498	0.498	0.500	0.680
Hyper	0.249	0.509	0.249	0.509	0.249	0.249	0.252	0.391
Google	0.007	0.006	0.007	0.006	0.008	0.008	0.008	0.005

DOI: 10.7717/peerj-cs.1827/table-5

Configurations

The algorithms from the Machine Learning for Data Streams (MOA) tool have used the same optimized parameters settings provided in the existing article (Gonçalves et al., 2014) and summarized in Table 6. A minimum number of instances allowed before the drift or change is detected is represented by “N”. A minimum number of instances not correctly classified before the drift warning is “ $e$”. Standard deviations that will be used to warn are referred to as “W”. Similarly, other abbreviations are the number of standard deviations to detect drift as “D” & “S”, “ $W1$” is the window size, max global error as “G”, and frequency of window reduction is as “M”.

Table 6:

Drift detector parameters configuration.

Drift detector	N	D	W	Other parameters
DDM	30	2.5	2	–
EDDM	30	0.9	0.95	$e$ =15
ECDD	30	–	0.5	$a$ = 40, $l$ = 0.1
STEPD	20	0.03	–	$m1$ = 0.08
ADWIN	–	–	–	$g$ = 0.02, M = 16

DOI: 10.7717/peerj-cs.1827/table-6

Results

Table 7 summarizes results for abrupt drift datasets. In sine datasets, except EDDM, HDDMA, and HBBMW, all others have shown promising results. All others showed impressive results for sudden drift in the second dataset except for EDDM. ADWIN compares the two subsequent windows, and the proper selection of windows improves the results. It performed well for both by showing performance near DDM, but its performance depends upon the position of drift and window sizes. LSTM-Drift detector could detect drift well in case of both sudden and gradual datasets.

Table 7:

Drift detector summary for abrupt drift.

Drift type	Abrupt
	Average error	Dataset	Average time	Dataset
DDM	Low	All	Low	Stagger
ADWIN	Low	Sine	High	Stagger
EDDM	High	ALL	High	All
HDDMA & HDDMW	High	Sine	Low	All
LSTMDD	Low	ALL	Low	All

DOI: 10.7717/peerj-cs.1827/table-7

Drift detector performance on the gradual drift is summarized in Table 8. DDM was again able to give good results for gradual drift for both gradual datasets. The remaining drift detectors, including LSTMDD, performed nearly the same in gradual datasets. In Google usage trace, sudden and gradual drift occurred, and LSTMDD performed better than other datasets. LSTM networks are well-equipped to handle both short and gradual drift in datasets. When confronted with fast drift instances with a sudden and brief change in the CPU usage pattern, LSTMs quickly adapt to these changes due to their ability to retain crucial information over time. The memory cells within LSTM units enable the network to adjust its internal state and capture the new patterns associated with the drift, making them adept at detecting and responding to short-lived changes. For gradual drift, where the CPU usage pattern changes gradually over an extended period, LSTMs are also effective. By leveraging their capacity to capture long-term dependencies and remember past patterns, LSTMs can recognize the subtle shifts that occur over time and adapt to the evolving trends associated with gradual drift. By training on historical CPU usage data that encompasses instances of drift, LSTM networks can learn to identify and accurately classify short and gradual drift instances by modeling the temporal dynamics of the CPU usage sequence. Fine-tuning hyperparameters and optimizing the model architecture further enhance the LSTM’s performance in detecting drift.

Table 8:

Drift detector summary for gradual drift.

Drift Type	Gradual
	Average error	Dataset	Average time	Dataset
DDM	Low	All	Low	All
Other DD	Average	All	Average	All
EDDM	High	All	High	Mixed
LSTMDD	Low	ALL	Low	All

DOI: 10.7717/peerj-cs.1827/table-8

Integrating an attention mechanism in an LSTM-based architecture enhances the model’s ability to handle imbalanced datasets. By allocating attention weights to specific parts of the input sequence, the model can focus on informative instances, particularly those belonging to the minority class or representing abnormal behavior in the CPU usage data. Thus, this helps mitigate the effects of class imbalance and enables the model to learn discriminating features and adapt to varying imbalance ratios. The attention mechanism provides a means to overcome the bias towards the majority class and improve the model’s performance in accurately detecting anomalies in imbalanced datasets.

The evaluation metric defined above to access the performance of LSTMDD against all databases is summarized in Table 9. Accuracy is not the best metric in the drift detection case, so precision and recall are the more critical evaluation metrics. Following the class balancing, each resource level’s confusion matrix is displayed in Fig. 6 for further explanation. Sampling was used to balance the classes because drift-occurring events are infrequent. Similarly, when determining the percentage of true drift cases mistakenly categorized as non-drift instances, FNR matters more than FPR. Low FNR is shown by the model in Table 9 for each of the three resource types. LSTMDD performed well on CPU usage drift detection with a macro f-score 0.98.

Table 9:

LSTMDD model’s results on datasets.

Dataset	Accuracy	Precision	Recall	F-score	FPR	FNR
Sine	0.85	0.85	0.85	0.85	0.15	0.13
Stagger	0.76	0.78	0.76	0.75	0.39	0.09
Mixed	0	0	0	0	0	0
Hyper	86	0.86	0.86	0.85	0.09	0.18
CPU google	0.98	1	0.98	0.98	0.019	0.0004
Memory google	0.98	0.97	0.99	0.98	0.028	0.003
Disk google	0.98	0.98	0.99	0.98	0.011	0.0001

DOI: 10.7717/peerj-cs.1827/table-9

Conclusion

Cloud usage constantly changes and requires an adaptive model because a static model, even trained with over a hundred years of data, will not be enough. Different drift detectors for adaptive learning mechanisms are required to deal with these sudden and gradual drifts. We presented a summary of the execution of different types of detectors for synthetic and real cloud datasets. The concept of drift detector selection depends on the domain for better detection, so generalization is impossible. In this article, we have investigated using LSTM models for concept drift detection in the cloud domain. This approach has shown promise in various applications, but its effectiveness for detecting concept drift in cloud environments still needs clarification. This article presents an attention mechanism-based LSTM using a genetic hyper-tuning drift detector to detect sudden and gradual drifts in the cloud. LSTMs’ ability to retain crucial information while capturing both short-lived and gradual changes is adept at detecting and responding to drift in data. At the same time, integrating an attention mechanism enhances their performance in handling imbalanced datasets. The DDM, EDDM, ADWIN, STEPD, ECDD, HDDMA, HDDMW, SEED, SEQ 1 and 2, and RMMD drift detectors are compared with the proposed LSTMDD. It performed well for datasets of larger size, with real drifts, with an f-score of 98% for all resource types.

In the future, we will enhance this model with built-in adaptive learning for the cloud to achieve better efficiency. The ensemble is best known for providing the advantage of multiple algorithms according to the requirement. The researcher can further explore the ensemble of drift detectors to improve the performance. We are exploring the potential use of the proposed approach for other applications beyond concept drift detection in cloud computing. For example, it may be possible to adapt the approach in other distributed computing systems where concept drift is a concern.