skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: The Benefit of Distraction: Denoising Camera-Based Physiological Measurements Using Inverse Attention
Attention networks perform well on diverse computer vision tasks. The core idea is that the signal of interest is stronger in some pixels ("foreground"), and by selectively focusing computation on these pixels, networks can extract subtle information buried in noise and other sources of corruption. Our paper is based on one key observation: in many real-world applications, many sources of corruption, such as illumination and motion, are often shared between the "foreground" and the "background" pixels. Can we utilize this to our advantage? We propose the utility of inverse attention networks, which focus on extracting information about these shared sources of corruption. We show that this helps to effectively suppress shared covariates and amplify signal information, resulting in improved performance. We illustrate this on the task of camera-based physiological measurement where the signal of interest is weak and global illumination variations and motion act as significant shared sources of corruption. We perform experiments on three datasets and show that our approach of inverse attention produces state-of-the-art results, increasing the signal-to-noise ratio by up to 5.8 dB, reducing heart rate and breathing rate estimation errors by as much as 30 %, recovering subtle waveform dynamics, and generalizing from RGB to NIR videos without retraining.  more » « less
Award ID(s):
1801372
PAR ID:
10301742
Author(s) / Creator(s):
Date Published:
Journal Name:
Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021
Page Range / eLocation ID:
4955-4964
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; (, PLOS ONE)
    Giove, Federico (Ed.)
    Resting-state blood-oxygen-level-dependent (BOLD) signal acquired through functional magnetic resonance imaging is a proxy of neural activity and a key mechanism for assessing neurological conditions. Therefore, practical tools to filter out artefacts that can compromise the assessment are required. On the one hand, a variety of tailored methods to preprocess the data to deal with identified sources of noise (e.g., head motion, heart beating, and breathing, just to mention a few) are in place. But, on the other hand, there might be unknown sources of unstructured noise present in the data. Therefore, to mitigate the effects of such unstructured noises, we propose a model-based filter that explores the statistical properties of the underlying signal (i.e., long-term memory). Specifically, we consider autoregressive fractional integrative process filters. Remarkably, we provide evidence that such processes can model the signals at different regions of interest to attain stationarity. Furthermore, we use a principled analysis where a ground-truth signal with statistical properties similar to the BOLD signal under the injection of noise is retrieved using the proposed filters. Next, we considered preprocessed (i.e., the identified sources of noise removed) resting-state BOLD data of 98 subjects from the Human Connectome Project. Our results demonstrate that the proposed filters decrease the power in the higher frequencies. However, unlike the low-pass filters, the proposed filters do not remove all high-frequency information, instead they preserve process-related higher frequency information. Additionally, we considered four different metrics (power spectrum, functional connectivity using the Pearson’s correlation, coherence, and eigenbrains) to infer the impact of such filter. We provided evidence that whereas the first three keep most of the features of interest from a neuroscience perspective unchanged, the latter exhibits some variations that could be due to the sporadic activity filtered out. 
    more » « less
  2. ; ; ; ; ; ; ; ; (, Optics Continuum)
    High-speed widefield fluorescence imaging of neural activity in vivo is fundamentally limited by fluctuations in recorded signal due to background contamination and stochastic noise. In this study, we show background and shot noise-reduced imaging of the ultrafast genetically encoded Ca2+indicator GCaMP8f in CA1 pyramidal neurons using periodic structured illumination (SI) with computational image reconstruction. We implement what we believe to be a novel reconstruction method for data acquired using periodic structured illumination, termed pseudo-HiLo (pHiLo), that combines a pseudo-widefield (pWF) reconstruction with individual SI frames to perform a HiLo reconstruction. We compare this new technique to interleaved optical sectioning structured illumination microscopy (OS-SIM) and pWF reconstruction. We quantify the performance of each reconstruction by evaluating contrast, transient peak-to-noise ratio (PNR), pairwise correlation coefficients between ΔF/F time courses extracted from individual in-focus cells, and correlation coefficients between each cell with surrounding cell-free background pixels. We additionally incorporate a self-supervised deep learning method for real-time noise suppression (DeepCAD-RT) into our data preprocessing pipeline. At 500 Hz frame rates, we demonstrate a 75% increase in PNR using the denoised pHiLo reconstruction compared to pWF. Utilizing DeepCAD-RT, we show significant PNR improvements using both structured illumination (SI) reconstruction methods with OS-SIM showing a 59% increase in PNR after denoising. Both pHiLo and OS-SIM reconstructions result in a ≈65% decrease in the mean correlation coefficient of the ΔF/F time courses between ROIs in comparison with pWF, indicating the potential to remove background fluorescent transients from out-of-focus cells. 
    more » « less
  3. ; ; (, Biomedical Optics Express)
    Camera-based physiological measurement enables vital signs to be captured unobtrusively without contact with the body. Remote, or imaging, photoplethysmography involves recovering peripheral blood flow from subtle variations in video pixel intensities. While the pulse signal might be easy to obtain from high quality uncompressed videos, the signal-to-noise ratio drops dramatically with video bitrate. Uncompressed videos incur large file storage and data transfer costs, making analysis, manipulation and sharing challenging. To help address these challenges, we use compression specific supervised models to mitigate the effect of temporal video compression on heart rate estimates. We perform a systematic evaluation of the performance of state-of-the-art algorithms across different levels, and formats, of compression. We demonstrate that networks trained on compressed videos consistently outperform other benchmark methods, both on stationary videos and videos with significant rigid head motions. By training on videos with the same, or higher compression factor than test videos, we achieve improvements in signal-to-noise ratio (SNR) of up to 3 dB and mean absolute error (MAE) of up to 6 beats per minute (BPM). 
    more » « less
  4. ; ; ; ; (, Journal of Electronic Imaging)
    Single-photon sensitive image sensors have recently gained popularity in passive imaging applications where the goal is to capture photon flux (brightness) values of different scene points in the presence of challenging lighting conditions and scene motion. Recent work has shown that high-speed bursts of single-photon timestamp information captured using a single-photon avalanche diode camera can be used to estimate and correct for scene motion thereby improving signal-to-noise ratio and reducing motion blur artifacts. We perform a comparison of various design choices in the processing pipeline used for noise reduction, motion compensation, and upsampling of single-photon timestamp frames. We consider various pixelwise noise reduction techniques in combination with state-of-the-art deep neural network upscaling algorithms to super-resolve intensity images formed with single-photon timestamp data. We explore the trade space of motion blur and signal noise in various scenes with different motion content. Using real data captured with a hardware prototype, we achieved superresolution reconstruction at frame rates up to 65.8 kHz (native sampling rate of the sensor) and captured videos of fast-moving objects. The best reconstruction is obtained with the motion compensation approach, which achieves a structural similarity (SSIM) of about 0.67 for fast moving rigid objects. We are able to reconstruct subpixel resolution. These results show the relative superiority of our motion compensation compared to other approaches that do not exceed an SSIM of 0.5. 
    more » « less
  5. ; ; (, PLOS ONE)
    Mariño, Inés P. (Ed.)
    In many physiological systems, real-time endogeneous and exogenous signals in living organisms provide critical information and interpretations of physiological functions; however, these signals or variables of interest are not directly accessible and must be estimated from noisy, measured signals. In this paper, we study an inverse problem of recovering gas exchange signals of animals placed in a flow-through respirometry chamber from measured gas concentrations. For large-scale experiments (e.g., long scans with high sampling rate) that have many uncertainties (e.g., noise in the observations or an unknown impulse response function), this is a computationally challenging inverse problem. We first describe various computational tools that can be used for respirometry reconstruction and uncertainty quantification when the impulse response function is known. Then, we address the more challenging problem where the impulse response function is not known or only partially known. We describe nonlinear optimization methods for reconstruction, where both the unknown model parameters and the unknown signal are reconstructed simultaneously. Numerical experiments show the benefits and potential impacts of these methods in respirometry. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email