Traffic congestion anomaly detection is of paramount importance in intelligent traffic systems. The goals of transportation agencies are two-fold: to monitor the general traffic conditions in the area of interest and to locate road segments under abnormal congestion states. Modeling congestion patterns can achieve these goals for citywide roadways, which amounts to learning the distribution of multivariate time series (MTS). However, existing works are either not scalable or unable to capture the spatial-temporal information in MTS simultaneously. To this end, we propose a principled and comprehensive framework consisting of a data-driven generative approach that can perform tractable density estimation for detecting traffic anomalies. Our approach first clusters segments in the feature space and then uses conditional normalizing flow to identify anomalous temporal snapshots at the cluster level in an unsupervised setting. Then, we identify anomalies at the segment level by using a kernel density estimator on the anomalous cluster. Extensive experiments on synthetic datasets show that our approach significantly outperforms several state-of-the-art congestion anomaly detection and diagnosis methods in terms of Recall and F1-Score. We also use the generative model to sample labeled data, which can train classifiers in a supervised setting, alleviating the lack of labeled data for anomaly detection in sparse settings.
more »
« less
Anomaly Detection Using Support Vector Machines for Time Series Data
Analysis of large data sets is increasingly important in business and scientific research. One of the challenges in such analysis
stems from uncertainty in data, which can produce anomalous results. This paper proposes a method for detecting an anomaly
in time series data using a Support Vector Machine (SVM). Three different kernels of the SVM are analyzed to predict anomalies
in the UCR time series benchmark data sets. Comparison of the three kernels shows that the defined parameter values of the
Radial Basis Function (RBF) kernel are critical for improving the validity and accuracy in anomaly detection. Our results show
that the RBF kernel of the SVM can be used to advantage in detecting anomalies.
more »
« less
- Award ID(s):
- 1818884
- NSF-PAR ID:
- 10289945
- Date Published:
- Journal Name:
- Journal of robotics networking and artificial life
- ISSN:
- 2405-9021
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Traffic congestion anomaly detection is of paramount importance in intelligent traffic systems. The goals of transportation agencies are two-fold: to monitor the general traffic conditions in the area of interest and to locate road segments under abnormal congestion states. Modeling congestion patterns can achieve these goals for citywide roadways, which amounts to learning the distribution of multivariate time series (MTS). However, existing works are either not scalable or unable to capture the spatial-temporal information in MTS simultaneously. To this end, we propose a principled and comprehensive framework consisting of a data-driven generative approach that can perform tractable density estimation for detecting traffic anomalies. Our approach first clusters segments in the feature space and then uses conditional normalizing flow to identify anomalous temporal snapshots at the cluster level in an unsupervised setting. Then, we identify anomalies at the segment level by using a kernel density estimator on the anomalous cluster. Extensive experiments on synthetic datasets show that our approach significantly outperforms several state-of-the-art congestion anomaly detection and diagnosis methods in terms of Recall and F1-Score. We also use the generative model to sample labeled data, which can train classifiers in a supervised setting, alleviating the lack of labeled data for anomaly detection in sparse settings.more » « less
-
more » « less
Abstract Deep generative learning cannot only be used for generating new data with statistical characteristics derived from input data but also for anomaly detection, by separating nominal and anomalous instances based on their reconstruction quality. In this paper, we explore the performance of three unsupervised deep generative models—variational autoencoders (VAEs) with Gaussian, Bernoulli, and Boltzmann priors—in detecting anomalies in multivariate time series of commercial-flight operations. We created two VAE models with discrete latent variables (DVAEs), one with a factorized Bernoulli prior and one with a restricted Boltzmann machine (RBM) with novel positive-phase architecture as prior, because of the demand for discrete-variable models in machine-learning applications and because the integration of quantum devices based on two-level quantum systems requires such models. To the best of our knowledge, our work is the first that applies DVAE models to anomaly-detection tasks in the aerospace field. The DVAE with RBM prior, using a relatively simple—and classically or quantum-mechanically enhanceable—sampling technique for the evolution of the RBM’s negative phase, performed better in detecting anomalies than the Bernoulli DVAE and on par with the Gaussian model, which has a continuous latent space. The transfer of a model to an unseen dataset with the same anomaly but without re-tuning of hyperparameters or re-training noticeably impaired anomaly-detection performance, but performance could be improved by post-training on the new dataset. The RBM model was robust to change of anomaly type and phase of flight during which the anomaly occurred. Our studies demonstrate the competitiveness of a discrete deep generative model with its Gaussian counterpart on anomaly-detection problems. Moreover, the DVAE model with RBM prior can be easily integrated with quantum sampling by outsourcing its generative process to measurements of quantum states obtained from a quantum annealer or gate-model device.
-
Many network/graph structures are continuously monitored by various sensors that are placed at a subset of nodes and edges. The multidimensional data collected from these sensors over time create large-scale graph data in which the data points are highly dependent. Monitoring large-scale attributed networks with thousands of nodes and heterogeneous sensor data to detect anomalies and unusual events is a complex and computationally expensive process. This paper introduces a new generic approach inspired by state-space models for network anomaly detection that can utilize the information from the network topology, the node attributes (sensor data), and the anomaly propagation sets in an integrated manner to analyze the entire network all at once. This article presents how heterogeneous network sensor data can be analyzed to locate the sources of anomalies as well as the anomalous regions in a network, which can be impacted by one or multiple anomalies at any time instance. Experimental results demonstrate the superior performance of our proposed framework in detecting anomalies in attributed graphs. Summary of Contribution: With the increasing availability of large-scale network sensors and rapid advances in artificial intelligence methods, fundamentally new analytical tools are needed that can integrate data collected from sensors across the networks for decision making while taking into account the stochastic and topological dependencies between nodes, sensors, and anomalies. This paper develops a framework to intelligently and efficiently analyze complex and highly dependent data collected from disparate sensors across large-scale network/graph structures to detect anomalies and abnormal behavior in real time. Unlike general purpose (often black-box) machine learning models, this paper proposes a unique framework for network/graph structures that incorporates the complexities of networks and interdependencies between network entities and sensors. Because of the multidisciplinary nature of the paper that involves optimization, machine learning, and system monitoring and control, it can help researchers in both operations research and computer science domains to develop new network-specific computing tools and machine learning frameworks to efficiently manage large-scale network data.more » « less
-
Abstract We present the Arctic atmospheric river (AR) climatology based on twelve sets of labels derived from ERA5 and MERRA-2 reanalyses for 1980–2019. The ARs were identified and tracked in the 3-hourly reanalysis data with a multifactorial approach based on either atmospheric column-integrated water vapor (IWV) or integrated water vapor transport (IVT) exceeding one of the three climate thresholds (75th, 85th, and 95th percentiles). Time series analysis of the AR event counts from the AR labels showed overall upward trends from the mid-1990s to 2019. The 75th IVT- and IWV-based labels, as well as the 85th IWV-based labels, are likely more sensitive to Arctic surface warming, therefore, detected some broadening of AR-affected areas over time, while the rest of the labels did not. Spatial exploratory analysis of these labels revealed that the AR frequency of occurrence maxima shifted poleward from over-land in 1980–1999 to over the Arctic Ocean and its outlying Seas in 2000–2019. Regions across the Atlantic, the Arctic, to the Pacific Oceans trended higher AR occurrence, surface temperature, and column-integrated moisture. Meanwhile, ARs were increasingly responsible for the rising moisture transport into the Arctic. Even though the increase of Arctic AR occurrence was primarily associated with long-term Arctic surface warming and moistening, the effects of changing atmospheric circulation could stand out locally, such as on the Pacific side over the Chukchi Sea. The changing teleconnection patterns strongly modulated AR activities in time and space, with prominent anomalies in the Arctic-Pacific sector during the latest decade. Besides, the extreme events identified by the 95th-percentile labels displayed the most significant changes and were most influenced by the teleconnection patterns. The twelve Arctic AR labels and the detailed graphics in the atlas can help navigate the uncertainty of detecting and quantifying Arctic ARs and their associated effects in current and future studies.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
