skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Online Multivariate Anomaly Detection and Localization for High-Dimensional Settings
This paper considers the real-time detection of abrupt and persistent anomalies in high-dimensional data streams. The goal is to detect anomalies quickly and accurately so that the appropriate countermeasures could be taken in time before the system possibly gets harmed. We propose a sequential and multivariate anomaly detection method that scales well to high-dimensional datasets. The proposed method follows a nonparametric, i.e., data-driven, and semi-supervised approach, i.e., trains only on nominal data. Thus, it is applicable to a wide range of applications and data types. Thanks to its multivariate nature, it can quickly and accurately detect challenging anomalies, such as changes in the correlation structure. Its asymptotic optimality and computational complexity are comprehensively analyzed. In conjunction with the detection method, an effective technique for localizing the anomalous data dimensions is also proposed. The practical use of proposed algorithms are demonstrated using synthetic and real data, and in variety of applications including seizure detection, DDoS attack detection, and video surveillance.  more » « less
Award ID(s):
2040572
PAR ID:
10418526
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
Sensors
Volume:
22
Issue:
21
ISSN:
1424-8220
Page Range / eLocation ID:
8264
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; (, Sensors)
    This paper considers the problem of real-time detection and classification of power quality disturbances in power delivery systems. We propose a sequential and multivariate disturbance detection method (aiming for quick and accurate detection). Our proposed detector follows a non-parametric and supervised approach, i.e., it learns nominal and anomalous patterns from training data involving clean and disturbance signals. The multivariate nature of the method enables joint processing of data from multiple meters, facilitating quicker detection as a result of the cooperative analysis. We further extend our supervised sequential detection method to a multi-hypothesis setting, which aims to classify the disturbance events as quickly and accurately as possible in a real-time manner. The multi-hypothesis method requires a training dataset per hypothesis, i.e., per each disturbance type as well as the ’no disturbance’ case. The proposed classification method is demonstrated to quickly and accurately detect and classify power disturbances. 
    more » « less
  2. ; ; (, Electronics)
    In this paper, we address the problem of detecting and learning anomalies in high-dimensional data-streams in real-time. Following a data-driven approach, we propose an online and multivariate anomaly detection method that is suitable for the timely and accurate detection of anomalies. We propose our method for both semi-supervised and supervised settings. By combining the semi-supervised and supervised algorithms, we present a self-supervised online learning algorithm in which the semi-supervised algorithm trains the supervised algorithm to improve its detection performance over time. The methods are comprehensively analyzed in terms of computational complexity, asymptotic optimality, and false alarm rate. The performances of the proposed algorithms are also evaluated using real-world cybersecurity datasets, that show a significant improvement over the state-of-the-art results. 
    more » « less
  3. ; ; ; ; ; ; ; ; ; ; et al (, Computing and Software for Big Science)
    Abstract The CMS detector is a general-purpose apparatus that detects high-energy collisions produced at the LHC. Online data quality monitoring of the CMS electromagnetic calorimeter is a vital operational tool that allows detector experts to quickly identify, localize, and diagnose a broad range of detector issues that could affect the quality of physics data. A real-time autoencoder-based anomaly detection system using semi-supervised machine learning is presented enabling the detection of anomalies in the CMS electromagnetic calorimeter data. A novel method is introduced which maximizes the anomaly detection performance by exploiting the time-dependent evolution of anomalies as well as spatial variations in the detector response. The autoencoder-based system is able to efficiently detect anomalies, while maintaining a very low false discovery rate. The performance of the system is validated with anomalies found in 2018 and 2022 LHC collision data. In addition, the first results from deploying the autoencoder-based system in the CMS online data quality monitoring workflow during the beginning of Run 3 of the LHC are presented, showing its ability to detect issues missed by the existing system. 
    more » « less
  4. ; ; ; (, Journal of the Royal Statistical Society Series C: Applied Statistics)
    Summary Decentralized waste water treatment facilities monitor many features that are complexly related. The ability to detect the onset of a fault and to identify variables accurately that have shifted because of the fault are vital to maintaining proper system operation and high quality produced water. Various multivariate methods have been proposed to perform fault detection and isolation, but the methods require data to be independent and identically distributed when the process is in control, and most require a distributional assumption. We propose a distribution-free retrospective change-point-detection method for auto-correlated and non-stationary multivariate processes. We detrend the data by using observations from an in-control time period to account for expected changes due to external or user-controlled factors. Next, we perform the fused lasso, which penalizes differences in consecutive observations, to detect faults and to identify shifted variables. To account for auto-correlation, the regularization parameter is chosen by using an estimated effective sample size in the extended Bayesian information criterion. We demonstrate the performance of our method compared with a competitor in simulation. Finally, we apply our method to waste water treatment facility data with a known fault, and the variables identified by our proposed method are consistent with the operators’ diagnosis of the fault's cause. 
    more » « less
  5. ; ; (, Journal of the American Statistical Association)
    The estimation of large precision matrices is crucial in modern multivariate analysis. Traditional sparsity assumptions, while useful, often fall short of accurately capturing the dependencies among features. This article addresses this limitation by focusing on precision matrix estimation for multivariate data characterized by a flexible yet unknown group structure. We introduce a novel approach that begins with the detection of this unknown group structure, clustering features within the low-dimensional space defined by the leading eigenvectors of the sample covariance matrix. Following this, we employ group-wise multivariate response linear regressions, guided by the identified group memberships, to estimate the precision matrix. We rigorously establish the theoretical foundations of our proposed method for both group detection and precision matrix estimation. The superior numerical performance of our approach is demonstrated through comprehensive simulation experiments and a comparative analysis with established methods in the field. Additionally, we apply our method to a real breast cancer dataset, showcasing its practical utility and effectiveness. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email