A survey of streaming data anomaly detection in network security

Overview of the review process

This investigation utilized a systematic methodology to select high-quality articles on streaming data anomaly detection, with pronounced emphasis on network security applications. It established a structured review framework comprising: (1) comprehensive search across multiple databases, (2) the application of stringent inclusion criteria, and (3) the meticulous filtering and critical appraisal of the resultant literature.

Data sources

The systematic review was conducted from September 2023 to October 2024, focusing on publications from 2015–2024. The present study conducted an exhaustive and detailed search of relevant literature, including research articles from journals, conferences, books, and magazines. To guarantee comprehensive coverage, multiple databases were utilized, including Google Scholar, IEEE Xplore, Springer, ScienceDirect, Scopus, Web of Science, and ACM Digital Library. Furthermore, a selection of highly reputable conferences such as USENIX, WWW, SIGKDD, VLDB, SIGMOD, NDSS, S&P, ICDM, and CCS were considered. Figure 2 visually presents the distribution of research from different resource types. As depicted, the corpus of literature under review comprises 884 conference articles, 715 journal articles, and 110 books included in the study.

Search keywords and scope

The study search was confined to computer science and engineering domains, utilizing a two-phase keyword strategy:

• Initial phase: (Stream* data OR Stream* OR Real-time data) AND (Anomaly detection OR Outlier detect* OR Deviation detection OR Novelty detection).

• Refinement phase: Added domain-specific terms (IoT OR ICS OR SDN OR GRID OR WSN OR Cloud OR Sensor OR Embedded Systems OR Intrusion Detection).

where “∗” indicates wildcard matching to capture variations of the terms.

Selection criteria

To ensure the quality and relevance of the selected literature, the study established specific inclusion criteria as delineated in Table 1. These criteria guided both initial screening and detailed evaluation phases.

Filtering process

The study employed a systematic selection process, illustrated in Fig. 3, encompassing three principal stages:

1. Initial screening: In the initial phase, it used the keywords mentioned in the first strategy to screen out a total of 2,620 publications. Subsequently, based on the crude detection in the secondary strategy’s criteria, it screened out 909 publications, leaving 1,711 publications. From 1,711 identified articles, this study removed publications lacking academic rankings and those irrelevant based on abstract review, yielding 252 publications.

2. Quality assessment: Further evaluation based on methodological rigor, citation impact, and relevance to streaming anomaly detection reduced the selection to 105 high-quality articles.

3. Final classification: Detailed examination identified 42 articles specifically focused on network security applications of streaming anomaly detection, which formed the core of the analytical review.

Dataset classification

Given that real-world datasets are not universally designed for streaming data, recent research has principally adopted two methodologies. The initial approach identifies anomalies by classifying infrequent occurrences as statistical outliers. The second strategy, congruent with the real-time characteristics of streaming data, pinpointing anomalies that conform to the protocol’s real-time criteria. The classification of the datasets is as follows:

• Statistical dataset: It refers to datasets composed of multiple continuous or discrete features presented in tabular form Moustafa, Turnbull & Choo (2018).

• Sequential dataset: It refers to datasets where there is implicit time correlation or dependency between consecutive data points (Karim, Majumdar & Darabi, 2020).

• Spatial dataset: It refers to datasets with specific locations and spatial relationships (Wang et al., 2020a).

This study presents a compilation of commonly used datasets, with information including the dataset name, data entries, classified attributes, the number of attacks, and the statistically derived anomaly thresholds, as shown in Table 2.

Measurement technique

Streaming data measurements in large-scale, high-speed networks is crucial for anomaly detection. A multitude of algorithms estimate streaming data for anomaly detection using approximate computation and compression techniques under the stringent constraints of limited memory and computational resources. This study provides a systematic summarization of prevalent streaming data measurement techniques as follows:

• Frequency estimation: Frequency estimation estimates the frequency of elements in streaming data using hash functions and counters. Prominent examples include space-saving, count-min sketch (CMS). CMS uses multiple hash functions and a 2D array to estimate frequencies (Cormode & Muthukrishnan, 2005). Given its exceptional space efficiency, rapid query performance, and robust fault tolerance, this approach is widely adopted across diverse applications. Tong & Prasanna (2017) employed CMS and K-ary sketch for heavy hitter detection and heavy change detection. Bhatia et al. (2022) used CMS for detecting anomaly micro-clusters in streaming data.

• Quantile estimation: Quantile estimation achieves quantile estimation by maintaining quantile summaries or quantile estimators within error boundaries, such as t-digest (Radke et al., 2018) and the Greenwald-Khanna algorithm (Lall, 2015).

• Change detection: Change detection identifies change points or anomalies that deviate from the normal pattern by analyzing patterns, trends, or statistical features in streaming data. Among the most commonly employed algorithms are the cumulative sum (CUSUM) control chart and the Page-Hinkley (PH) test. Martínez-Rego et al. (2015) adopted Bernoulli CUSUM for change detection. Duarte, Gama & Bifet (2016) availed of PH to detect changes in the data generation process and respond to them using pruning rules.

• Cardinality estimation: Cardinality estimation makes use of hash functions or bit arrays to map elements to specific positions and estimate cardinality based on statistical information. In contrast to frequency estimation, cardinality estimation is exclusively focused on the total number of distinct elements in streaming data, such as LogLog, HyperLogLog, and bloom filter. It is worth noting that HyperLogLog estimates cardinality based on the maximum leading zero count (LZC) in an array, while LogLog estimates cardinality based on the maximum zero count in an array. Xiao et al. (2023) proposed three HyperLogLog-based algorithms to estimate streaming distribution and reduce estimation errors.

• Similarity calculation: Similarity calculation is the relationship between data objects by comparing their similarities. Locality sensitive hashing (LSH) is a prevalently used algorithm. LSH functions by selecting a family of hash functions engineered to map similar data points to the same bucket with a high probability. Notable variants include MinHash, LSH Forest, and random projection (RP). Zeng et al. (2023b) used double locality sensitive hashing bloom filter (DLSH) to improve accuracy and efficiency. Yang et al. (2022) proposed DLSHiForest to address the inherent property of infinite, correlated, and concept drift in traditional static anomaly detection algorithms. Pham et al. (2014) applied RP to obtain compressed data and solve the scalability issue. Lai et al. (2022) replaced the entropy estimation calculation with a simple lookup process using RP.

• Sampling: Sampling algorithms approximate the analysis of entire streaming data by selecting a subset of elements from the streaming data. These include sticky sampling (SS) and reservoir sampling (RS). SS prioritizes sampling based on data priority, while RS is a random sampling technique. Wang et al. (2023) adopted weighted RS to model the distribution characteristics of historical reliability streaming data in mobile edge computing (MEC). Yu et al. (2018) applied RS algorithm to represent the vectors of vertices in dynamic network computation.

Detection algorithm

Research on streaming data anomaly detection has evolved through multiple scholarly trajectories. Comprehensive studies have systematically examined diverse detection methodologies, with each addressing distinct aspects: Wang, Bah & Hammad (2019) taxonomized approaches into distance-based, clustering-based, density-based, ensemble-based, and learning-based categories (2000–2019); Boukerche, Zheng & Alfandi (2020) analyzed algorithmic efficiency parameters and high-dimensional processing challenges while proffering a novel classification framework; Din et al. (2021) specifically addressed concept evolution phenomena in streaming classification; Bhaya & Alasadi (2016) evaluated streaming mining techniques for network traffic anomaly detection and Souiden, Brahmi & Toumi (2016) furnished comparative assessment frameworks for algorithm selection across various contexts. Further specialized research has emerged along two distinct lines. The first focuses on specific application scenarios: Fahy, Yang & Gongora (2022) addressed the label scarcity problem in dynamic streams with concept drift; Stahmann & Rieger (2021) investigated anomaly detection requirements in Industry 4.0 manufacturing environments with millisecond-frequency sensor data. The second line explores specialized methodological frameworks: Krawczyk et al. (2017) examined ensemble learning approaches for non-stationary stream environment; Faria et al. (2016) analyzed offline/online phase integration and noise-anomaly differentiation in novelty detection; Barbariol et al. (2022) evaluated tree-based methods, particularly iForest variants; Clever et al. (2022) constructed a structured framework for streaming classification workflows. Additionally, comprehensive reviews have addressed cross-cutting challenges in streaming data processing: Gurjar & Chhabria (2015) examined concept evolution in streaming classification with methods for unknown class detection; while (Chauhan & Shukla, 2015) explored K-Means applications for clustering-based anomaly detection in high-volume, concept-drifting streaming data.

Notwithstanding these significant contributions of existing research, the rapid advancement of big data technologies, machine learning, and deep learning has precipitated the emergence of numerous innovative methodologies in recent years. Accordingly, this study proffers a new taxonomic framework that synthesizes both established classical methods and recently developed approaches into two primary categories: traditional machine learning and deep learning. Within the traditional machine learning paradigm, models are further classified according to their algorithmic principles: statistical models, distance models, clustering models, density models, isolation models, and frequent item mining models. Analogously, deep learning approaches are categorized as reconstruction models, generative models, predictive models, and representation learning models. Employing this structured framework, the present study undertakes a systematic review and comparative evaluation of these distinct methodological classes.

Anomaly and output type

To facilitate systematic analysis and treatment of anomalies, it is essential to establish a comprehensive classification method that encompasses both anomaly types and output types.

• Classification of anomaly types: Anomaly values are typically classified into three types: point anomaly, contextual anomaly, and collective anomaly (Gorunescu, 2011). Point anomaly refers to isolated data points that are markedly different from other data points. Contextual anomaly are data points that deviate from normal behavior or patterns within a specific context compared to other data points in given context. Collective anomaly refers to anomalies relative to the entire dataset.

• Classification of output types: Output result types are generally divided into label and score (Chandola, Banerjee & Kumar, 2009). Anomaly labels allow for direct determination of whether each point is an anomaly based on the model’s output. Anomaly scores provide further insight into which points exhibit a higher degree of anomaly.

Evaluation criteria

This research proposes a comprehensive set of evaluation criteria for assessing existing research. As shown in Table 6, these criteria encompass a broad spectrum of research outcomes. Specifically, the evaluation criteria mainly include efficiency metrics, evaluation metrics, task metrics, and anomaly interpretability.

• Efficiency metrics: Efficiency metrics serve to quantify the efficiency of algorithm, particularly with respect to their algorithmic complexity.

• Evaluation metrics: Evaluation metrics are utilized to evaluate the efficacy of algorithm. They primarily encompass the following: false alarm rate (FAR), recall, detection rate (DR), true positive rate (TPR), true negative rate (TNR), receiver operating characteristic curve (ROC), and area under curve (AUC). In addition to the above evaluation metrics, additional indices, such as the Youden’s index (Harush, Meidan & Shabtai, 2021) and the Kappa coefficient (Xing, Demertzis & Yang, 2020; Jain, Kaur & Saxena, 2022) are also used as reference standards.

• Task metrics: Task metrics assess algorithm’s capability in managing concept drift and feature drift, as well as its aptitude for handling high-dimensional or large-scale streaming datasets.

• Anomaly explanation: Anomaly explanation indicates whether algorithm can provide explanations.

Detection algorithm discussion

The vast majority of traditional machine learning models are lightweight, meaning they have compact model size and low memory requirements. Therefore, these algorithms are widely used in resource-constrained environments.

Statistical models, valued for their parsimony and straightforward implementation, have demonstrated considerable utility across a multitude of applications. Hunt & Willett (2018) adapted gaussian mixture model for online anomaly detection in wide-area motion imagery and email databases. Tao & Michailidis (2019) used higher-order statistical techniques to detect false data injection (FDI) attacks in power systems. Ma, Aminian & Kirby (2019) applied ARIMA to detect anomalous traffic in WSN. Nevertheless, a principal constraint of such statistical methodologies is their inherent reliance on specific distributional assumptions. This prerequisite can substantially limit their effectiveness when confronted with the complexity of real-world data, where such assumptions may not be valid.

Distance models facilitate localization and identification of anomalies by leveraging distance or similarity measures. Zhu et al. (2020) employed min heap to compute upper bounds or lower bounds of distances between objects and their k-th nearest neighbors for anomaly detection in IoT. Miao et al. (2018) implemented online OCSVM in distributed WSN. However, distance-based models often face the curse of dimensionality and high computational costs when dealing with high dimensional or large-scale data. It is essential to adopt suitable dimensionality reduction methods and optimize distance metrics to ease these challenges.

Cluster models adeptly adapt to intrinsic structural dynamics, rendering them well-suited for addressing distributional shifts in streaming data. Jain, Kaur & Saxena (2022), ZareMoodi, Kamali Siahroudi & Beigy (2019) used cluster-based models to address concept drift. Wang et al. (2022) analyzed the influence of collective anomalies based on cluster. Zou et al. (2023) combined grid cluster with gaussian model to improve the ability to distinguish noise from anomalies. Harush, Meidan & Shabtai (2021) integrated cluster-based methods with deep learning to classify contextual information in real time. Bah et al. (2019) employed micro-clusters to refine the search space for outlier detection. Wang et al. (2020b) utilized the centers of micro-cluster within each class as inputs for detecting unknown classes by projecting micro-cluster centers onto fixed positions on orthogonal axes in the feature space, forming clear classification boundaries.

Density models effectively address local anomaly detection in high-dimensional or large-scale streaming data with with reduced memory requirements. Liu et al. (2020) introduced a KDE-based outlier detection method that substantially accelerates processing speed through an upper-bound pruning strategy. Chen, Wang & Yang (2021) employed entropy-weighted index calculation and reachable distance factor discrimination methods, achieving a 15% improvement in accuracy while requiring only 1% of the runtime compared to the traditional LOF. Density-based models can adapt to the evolution of streaming data, but they have high computational complexity and are challenging to apply to high-dimensional sparse or large-scale streaming data.

Isolation models exhibit robust real-time capability (Shao et al., 2020). However, they are characterized by high computational complexity and sensitive to model parameters. Consequently, the development of more efficient isolation frameworks is imperative to improve detection speed.

Frequency item mining algorithms can uncover common patterns in streaming data, thereby enhancing the recognition and pinpointing of anomalous samples. Cai et al. (2020a, 2020b), Hao et al. (2019) stored uncertain streaming data information in matrices to detect outliers. Compared to conventional static data frequent item mining algorithms, these approaches are more suitable for processing large-scale uncertain streaming data. Owing to its ability to handle complex and large-scale data, automatically learn relevant features from streaming data, and provide automatic feature extraction, scalability, adaptability and handling of complex relationships in streaming data, deep learning-based algorithms are becoming increasingly popular.

Reconstruction models effectively learn high-level feature representations of data without the need for manual feature design. These models provide reconstruction errors or losses for anomalous data, making the models and results easier to understand and interpret. Yoo, Kim & Kim (2019), Zeng et al. (2023a) used AE to detect anomalies by analyzing the reconstruction errors between the reconstructed and original sequences. Nevertheless, reconstruction models have high complexity and require enough normal data samples as training sets to establish the distribution model for normal data.

Generative models not only capture complex streaming data distributions but also generate new samples based on the learned distributions. The GAN architecture stands as a paradigmatic exemplar in this domain, demonstrating remarkable efficacy in anomaly detection. Li et al. (2019) advanced the conventional GAN framework by integrating LSTM-RNN as its foundational architecture, thereby exploiting the intricate spatiotemporal correlations and multivariate dependencies inherent in sequential data streams. Extending this trajectory of innovation, Hallaji, Razavi-Far & Saif (2022) engineered an advanced approach that incorporates the dynamic temporal characteristics of streaming data directly into the GAN detector framework, yielding substantial improvements in intrusion detection precision within IoT ecosystems. In conjunction with GAN-based methodologies, restricted Boltzmann machines (RBMs) represent an equally viable analytical tool within the proposed classification paradigm. Xing, Demertzis & Yang (2020) pioneered enhancements in anomaly detection fidelity through the introduction of real-time evolving spiking RBM architectures, which demonstrate particular aptitude for processing high-velocity data streams. Further advancing this architectural lineage, Talapula et al. (2023) orchestrated a sophisticated fusion of RBMs with deep belief criteria and metaheuristic algorithms, specifically calibrated for generative streaming log data analysis, thereby substantially augmenting the model’s discriminative capabilities for anomaly identification.

Prediction models with their powerful expressive capabilities and adaptability can capture temporal dependencies and patterns. Wang et al. (2023) combined LSTM prediction models with AE to compress time series data into low-dimensional feature models. Compared to employing a single prediction-based model, combining with other types of deep learning models improves the efficiency and effectiveness of real-time streaming data detection. Liu et al. (2021) used LSTM+ for real-time monitoring and correction of streaming data in IoT. Similarly, Cheng et al. (2019) stacked distance-based models such as KNN and SVM, with probabilistic models, includeing decision trees and Bayesian classifiers and combined them with TCN for sequence problems.

Representation models can capture the underlying features and intrinsic structures of streaming data. Munir et al. (2018) used a CNN-based representation learning model, termed DeepAnT, to address periodic and seasonal anomalies that cannot be solved by distance-based and density-based anomaly detection techniques in traditional machine learning. Garg et al. (2019) improved the standard CNN employing a uniform distribution method for anomaly detection in heterogeneous streaming data.

Measurement algorithm discussion

Frequency estimation can estimate the occurrence frequency of various elements in streaming data. Consistent hashing and counters can efficiently estimate element frequencies in a single pass, characterized by minimal time and space complexity. These approaches yield error bounds for frequency approximation, meeting the real-time detection requirements. However, there is a certain estimation error compared to exact calculations due to the use of approximate counters.

Quantile estimation elucidates the global distribution of streaming data by calculating the approximate quantiles in real time. However, in cases where the data distribution in streaming is extremely uneven, the estimation of tail quantiles may be less accurate (Liu et al., 2018).

Change detection employs chi-square tests or the KL divergence to detect the statistical distribution and aggregate metrics of streaming data. However, this approach is sensitive to parameter settings (Kuncheva, 2011).

Cardinality estimation leverages probabilistic data structures to efficiently estimate the cardinality of streaming data in a single pass. However, compared to frequency estimation, real-time performance may be inferior (Jie et al., 2022).

Similarity calculation utilizes LSH or other algorithms to rapidly retrieve neighbors and compute approximate similarities in sublinear time in streaming data. However, similarity calculations for high-dimensional and large-scale streaming data may be time-consuming, requiring appropriate similarity measurement methodologies and thresholds (Wu et al., 2024).

Sampling techniques represent a critical dimension in the study taxonomy of streaming data anomaly detection methods, as they extract representative subsets from streaming data, significantly reducing storage and processing requirements. While effective sampling demands meticulously designed random or stratified strategies, sketch algorithms have emerged as particularly promising within this category due to their sublinear complexity and efficiency advantages over traditional item-by-item processing approaches. Recent advancements in sketch algorithms demonstrate their increasing relevance to the framework of streaming anomaly detection. Liu et al. (2016) developed a universal monitoring framework that exemplifies how sketching can provide system-wide visibility while preserving computational efficiency. Building on this foundation, Yang et al. (2018) addressed the challenge of varying traffic conditions through adaptive sketching methods, a critical capability for detecting anomalies in dynamically evolving streams. To address the need for scalable detection in high-throughput network scenarios, Huang et al. (2017) proposed a robust measurement framework. The evolution of sketch-based algorithms has focused primarily on two aspects central to effective anomaly detection: accuracy and performance. Liu et al. (2015) tackled the fundamental challenge of hash collisions by reconstructing and estimating infected host cardinality using overlapping techniques of hash bit strings based on vectorized Bloom Filter. This approach significantly enhances detection accuracy, addressing one of the key challenges identified in the study framework. More recently, Zeng et al. (2023b) advanced this concept by establishing a multi-layer dLSHBF model using Bloom Filter, which effectively avoids element conflicts by reducing data hash encoding length. Similarly, Xiao et al. (2023) employed a multi-level design methodology combined with TailCut’s register compression technique (Xiao et al., 2020) to alleviate hash collisions between streams, demonstrating how hash function selection critically impacts detection algorithm performance. Within this study comprehensive framework of streaming data anomaly detection methodologies, these sketch-based approaches represent a particularly valuable direction for network security applications, offering an optimal balance between computational efficiency and detection accuracy.

Limitations of current research

This study surveys the latest literature on anomaly detection in streaming data and discusses some issues in the research direction of streaming data in the field of network security. During the research process, this study verified that the existing literature in this field has not adequately addressed the relevant problems, thus requiring more attention. Based on the previous statements and research, the following general limitations of existing research literature can be identified:

1. The field of cybersecurity has not adequately addressed the issue of high-dimensional streaming data.

2. The interpretability of detected anomalies in streaming data is poor.

3. There is still significant room for improvement in the efficiency of these models or methods.

4. There is a lack of methods for handling anomalies in multi-type data.

5. There is a scarcity of datasets appropriate for streaming data in the field of network security.

Future research directions

Anomaly detection in high-dimensional streaming data: Anomaly detection in high-dimensional streaming data typically requires a significant amount of feature engineering. Deep learning can automatically learn relevant features, with its multi-layer structure, proficiently manage high-dimensional streaming data enhance its effectiveness by learning more abstract representations through the hierarchical extraction and combination of features. However, the application of deep learning algorithms in the field of cybersecurity, especially in high-dimensional streaming data, has not been extensively studied. Therefore, applying deep learning to anomaly detection in streaming data is a direction worth exploring. It is worth noting that high-dimensional streaming data requires more hidden layers for learning input features, resulting in a linear increase in model computational complexity with the increase of hidden layers.

Detection of anomalies in multi-type data: Different applications generate diverse streaming data, such as text streaming, image streaming, video streaming, etc. Detecting anomalies in these heterogeneous streaming data can be cumbersome. Therefore, there is a need to research new algorithms or techniques to handle these complex data formats to better understand streaming data in the real world.

Interpretability of anomaly: Recent studies have underscored the significance of anomaly interpretation, particularly in the context of streaming data. The aim is to unearth plausible explanations for detected abnormal patterns. This will help in understanding and evaluating relevant detection results and enhance the reliability of anomaly evaluation. So far, the existing literature has focused on interpreting anomalies in low-dimensional streaming data, thus requiring further investigation in this area.

Improvement of detection algorithm efficiency: The characteristics of streaming data require anomaly detection algorithms to produce results at a lower computational cost, making the efficiency of algorithms crucial. Due to limitations in time and storage space, it is worth considering compressing the streaming data first and then combining traditional machine learning algorithms or deep learning models for anomaly detection. This approach can improve efficiency while ensuring detection accuracy.

Application of large language models: Large language models (LLMs) offer a transformative approach to anomaly detection in network security streams by enabling the analysis of textual and semi-structured data at a deep semantic level. By processing diverse data sources, such as network logs, system alerts, or command-line sequences, LLMs can harness their sophisticated contextual comprehension to detect subtle and intricate anomalies that often elude conventional detection methodologies. For example, they are capable of identifying emerging phishing campaigns or multi-stage insider threats by analyzing the narrative flow and contextual coherence of communication streams. Despite this immense potential, their practical application remains an underexplored research area, primarily due to significant challenges. The high computational requirements and inference latency of LLMs pose a barrier to real-time analysis, while the need for curated, domain-specific datasets for effective fine-tuning presents another obstacle. Therefore, future research should be strategically directed towards enhancing the viability of LLMs for this task. Investigating lightweight architectures, such as distilled or pruned models, and exploring sophisticated transfer learning methodologies represent promising pathways to harness the power of LLMs for real-time, adaptive anomaly detection in cybersecurity.