An official website of the United States government
The problem of detecting anomalies in time series from network measurements has been widely studied and is a topic of fundamental importance. Many anomaly detection methods are based on packet inspection collected at the network core routers, with consequent disadvantages in terms of computational cost and privacy. We propose an alternative method in which packet header inspection is not needed. The method is based on the extraction of a normal subspace obtained by the tensor decomposition technique considering the correlation between different metrics. We propose a new approach for online tensor decomposition where changes in the normal subspace can be tracked efficiently. Another advantage of our proposal is the interpretability of the obtained models. The flexibility of the method is illustrated by applying it to two distinct examples, both using actual data collected on residential routers.
more »
« less
Tensor-based flow reconstruction from optimally located sensor measurements
Reconstructing high-resolution flow fields from sparse measurements is a major challenge in fluid dynamics. Existing methods often vectorize the flow by stacking different spatial directions on top of each other, hence confounding the information encoded in different dimensions. Here, we introduce a tensor-based sensor placement and flow reconstruction method which retains and exploits the inherent multidimensionality of the flow. We derive estimates for the flow reconstruction error, storage requirements and computational cost of our method. We show, with examples, that our tensor-based method is significantly more accurate than similar vectorized methods. Furthermore, the variance of the error is smaller when using our tensor-based method. While the computational cost of our method is comparable to similar vectorized methods, it reduces the storage cost by several orders of magnitude. The reduced storage cost becomes even more pronounced as the dimension of the flow increases. We demonstrate the efficacy of our method on three examples: a chaotic Kolmogorov flow, in situ and satellite measurements of the global sea surface temperature and three-dimensional unsteady simulated flow around a marine research vessel.
more »
« less
- Award ID(s):
- 1745654
- PAR ID:
- 10420332
- Date Published:
- Journal Name:
- Journal of Fluid Mechanics
- Volume:
- 962
- ISSN:
- 0022-1120
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Seismic data are often incomplete due to equipment malfunction, limited source and receiver placement at near and far offsets, and missing crossline data. Seismic data contain redundancies because they are repeatedly recorded over the same or adjacent subsurface regions, causing the data to have a low-rank structure. To recover missing data, one can organize the data into a multidimensional array or tensor and apply a tensor completion method. We can increase the effectiveness and efficiency of low-rank data reconstruction based on tensor singular value decomposition (tSVD) by analyzing the effect of tensor orientation and exploiting the conjugate symmetry of the multidimensional Fourier transform. In fact, these results can be generalized to any order tensor. Relating the singular values of the tSVD to those of a matrix leads to a simplified analysis, revealing that the most square orientation gives the best data structure for low-rank reconstruction. After the first step of the tSVD, a multidimensional Fourier transform, frontal slices of the tensor form conjugate pairs. For each pair, a singular value decomposition can be replaced with a much cheaper conjugate calculation, allowing for faster computation of the tSVD. Using conjugate symmetry in our improved tSVD algorithm reduces the runtime of the inner loop by 35%–50%. We consider synthetic and real seismic data sets from the Viking Graben Region and the Northwest Shelf of Australia arranged as high-dimensional tensors. We compare the tSVD-based reconstruction with traditional methods, projection onto convex sets and multichannel singular spectrum analysis, and we see that the tSVD-based method gives similar or better accuracy and is more efficient, converging with runtimes that are an order of magnitude faster than the traditional methods. In addition, we verify that the most square orientation improves recovery for these examples by 10%–20% compared with the other orientations.more » « less
-
Coded aperture X-ray CT (CAXCT) is a new low-dose imaging technology that promises far-reaching benefits in industrial and clinical applications. It places various coded apertures (CA) at a time in front of the X-ray source to partially block the radiation. The ill-posed inverse reconstruction problem is then solved using l1-norm-based iterative reconstruction methods. Unfortunately, to attain high-quality reconstructions, the CA patterns must change in concert with the view-angles making the implementation impractical. This paper proposes a simple yet radically different approach to CAXCT, which is coined StaticCodeCT, that uses a single-static CA in the CT gantry, thus making the imaging system amenable for practical implementations. Rather than using conventional compressed sensing algorithms for recovery, we introduce a new reconstruction framework for StaticCodeCT. Namely, we synthesize the missing measurements using low-rank tensor completion principles that exploit the multi-dimensional data correlation and low-rank nature of a 3-way tensor formed by stacking the 2D coded CT projections. Then, we use the FDK algorithm to recover the 3D object. Computational experiments using experimental projection measurements exhibit up to 10% gains in the normalized root mean square distance of the reconstruction using the proposed method compared with those attained by alternative low-dose systems.more » « less
-
Recent works have shown that imposing tensor structures on the coefficient tensor in regression problems can lead to more reliable parameter estimation and lower sample complexity compared to vector-based methods. This work investigates a new low-rank tensor model, called Low Separation Rank (LSR), in Generalized Linear Model (GLM) problems. The LSR model – which generalizes the well-known Tucker and CANDECOMP/PARAFAC (CP) models, and is a special case of the Block Tensor Decomposition (BTD) model – is imposed onto the coefficient tensor in the GLM model. This work proposes a block coordinate descent algorithm for parameter estimation in LSR-structured tensor GLMs. Most importantly, it derives a minimax lower bound on the error threshold on estimating the coefficient tensor in LSR tensor GLM problems. The minimax bound is proportional to the intrinsic degrees of freedom in the LSR tensor GLM problem, suggesting that its sample complexity may be significantly lower than that of vectorized GLMs. This result can also be specialised to lower bound the estimation error in CP and Tucker-structured GLMs. The derived bounds are comparable to tight bounds in the literature for Tucker linear regression, and the tightness of the minimax lower bound is further assessed numerically. Finally, numerical experiments on synthetic datasets demonstrate the efficacy of the proposed LSR tensor model for three regression types (linear, logistic and Poisson). Experiments on a collection of medical imaging datasets demonstrate the usefulness of the LSR model over other tensor models (Tucker and CP) on real, imbalanced data with limited available samples. License: Creative Commons Attribution 4.0 International (CC BY 4.0)more » « less
-
Reconstructing a three-dimensional ocean sound speed field (SSF) from limited and noisy measurements presents an ill-posed and challenging inverse problem. Existing methods used a number of pre-specified priors (e.g., low-rank tensor and tensor neural network structures) to address this issue. However, the SSFs are often too complex to be accurately described by these pre-defined priors. While utilizing neural network-based priors trained on historical SSF data may be a viable workaround, acquiring SSF data remains a nontrivial task. This work starts with a key observation: Although natural images and SSFs admit fairly different characteristics, their denoising processes appear to share similar traits—as both remove random components from more structured signals. This observation allows us to incorporate deep denoisers trained using extensive natural images to realize zero-shot SSF reconstruction, without any extra training or network modifications. To implement this idea, an alternating direction method of multipliers (ADMM) algorithm using such a deep denoiser is proposed, which is reminiscent of the plug-and-play scheme from medical imaging. Our plug-and-play framework is tailored for SSF recovery such that the learned denoiser can be simultaneously used with other handcrafted SSF priors. Extensive numerical studies show that the new framework largely outperforms state-of-the-art baselines, especially under widely recognized challenging scenarios, e.g., when the SSF samples are taken as tensor fibers. The code is available at https://github.com/OceanSTARLab/DeepPnP.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1017/jfm.2023.269
