skip to main content

Explore Research Products in the NSF-PAR

Title: PPG3D: Does 3D head tracking improve camera-based PPG estimation?
Over the last few years, camera-based estimation of vital signs referred to as imaging photoplethysmography (iPPG) has garnered significant attention due to the relative simplicity, ease, unobtrusiveness and flexibility offered by such measurements. It is expected that iPPG may be integrated into a host of emerging applications in areas as diverse as autonomous cars, neonatal monitoring, and telemedicine. In spite of this potential, the primary challenge of non-contact camera-based measurements is the relative motion between the camera and the subjects. Current techniques employ 2D feature tracking to reduce the effect of subject and camera motion but they are limited to handling translational and in-plane motion. In this paper, we study, for the first-time, the utility of 3D face tracking to allow iPPG to retain robust performance even in presence of out-of-plane and large relative motions. We use a RGB-D camera to obtain 3D information from the subjects and use the spatial and depth information to fit a 3D face model and track the model over the video frames. This allows us to estimate correspondence over the entire video with pixel-level accuracy, even in the presence of out-of-plane or large motions. We then estimate iPPG from the warped video data that ensures per-pixel correspondence over the entire window-length used for estimation. Our experiments demonstrate improvement in robustness when head motion is large.  more » « less
Award ID(s):
1801372
NSF-PAR ID:
10301748
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)
Page Range / eLocation ID:
1194 to 1197
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ( , Biomedical Optics Express)

    Camera-based heart rate measurement is becoming an attractive option as a non-contact modality for continuous remote health and engagement monitoring. However, reliable heart rate extraction from camera-based measurement is challenging in realistic scenarios, especially when the subject is moving. In this work, we develop a motion-robust algorithm, labeled RobustPPG, for extracting photoplethysmography signals (PPG) from face video and estimating the heart rate. Our key innovation is to explicitly model and generate motion distortions due to the movements of the person’s face. We use inverse rendering to obtain the 3D shape and albedo of the face and environment lighting from video frames and then render the human face for each frame. The rendered face is similar to the original face but does not contain the heart rate signal; facial movements alone cause pixel intensity variation in the generated video frames. Finally, we use the generated motion distortion to filter the motion-induced measurements. We demonstrate that our approach performs better than the state-of-the-art methods in extracting a clean blood volume signal with over 2 dB signal quality improvement and 30% improvement in RMSE of estimated heart rate in intense motion scenarios.

     
    more » « less
  2. ( , Journal of the American Helicopter Society)
    The paper discusses an intelligent vision-based control solution for autonomous tracking and landing of Vertical Take-Off and Landing (VTOL) capable Unmanned Aerial Vehicles (UAVs) on ships without utilizing GPS signal. The central idea involves automating the Navy helicopter ship landing procedure where the pilot utilizes the ship as the visual reference for long-range tracking; however, refers to a standardized visual cue installed on most Navy ships called the ”horizon bar” for the final approach and landing phases. This idea is implemented using a uniquely designed nonlinear controller integrated with machine vision. The vision system utilizes machine learning based object detection for long-range ship tracking and classical computer vision for the estimation of aircraft relative position and orientation utilizing the horizon bar during the final approach and landing phases. The nonlinear controller operates based on the information estimated by the vision system and has demonstrated robust tracking performance even in the presence of uncertainties. The developed autonomous ship landing system was implemented on a quad-rotor UAV equipped with an onboard camera, and approach and landing were successfully demonstrated on a moving deck, which imitates realistic ship deck motions. Extensive simulations and flight tests were conducted to demonstrate vertical landing safety, tracking capability, and landing accuracy. The video of the real-world experiments and demonstrations is available at this URL. 
    more » « less
  3. ( , 77th Annual National Forum of the Vertical Flight Society)
    The paper discusses a machine learning vision and nonlinear control approach for autonomous ship landing of vertical flight aircraft without utilizing GPS signal. The central idea involves automating the Navy helicopter ship landing procedure where the pilot utilizes the ship as the visual reference for long-range tracking, but refers to a standardized visual cue installed on most Navy ships called the ”horizon bar” for the final approach and landing phases. This idea is implemented using a uniquely designed nonlinear controller integrated with machine vision. The vision system utilizes machine learning based object detection for long-range ship tracking, and classical computer vision for object detection and the estimation of aircraft relative position and orientation during the final approach and landing phases. The nonlinear controller operates based on the information estimated by the vision system and has demonstrated robust tracking performance even in the presence of uncertainties. The developed autonomous ship landing system is implemented on a quad-rotor vertical take-off and landing (VTOL) capable unmanned aerial vehicle (UAV) equipped with an onboard camera and was demonstrated on a moving deck, which imitates realistic ship deck motions using a Stewart platform and a visual cue equivalent to the horizon bar. Extensive simulations and flight tests are conducted to demonstrate vertical landing safety, tracking capability, and landing accuracy while the deck is in motion. 
    more » « less
  4. ; ( , Earthquake Spectra)

    We have conducted three-dimensional (3D) 0–7.5 Hz physics-based wave propagation simulations to model the seismic response of the Long Valley Dam (LVD), which has formed Lake Crowley in Central California, to estimate peak ground motions and settlement of the dam expected during maximum credible earthquake (MCE) scenarios on the nearby Hilton Creek Fault (HCF). We calibrated the velocity structure, anelastic attenuation model, and the overall elastic properties of the dam via linear simulations of a Mw3.7 event as well as the Mw6.2 Chalfant Valley earthquake of 1986, constrained by observed ground motions on and nearby the LVD. The Statewide California Earthquake Center (SCEC) Community Velocity Model CVM-S4.26.M01 superimposed with a geotechnical layer using [Formula: see text] information tapered from the surface to a 700-m depth was used in the simulations. We found optimal fit of simulated and observed ground motions at the LVD using frequency-independent attenuation of [Formula: see text] ([Formula: see text] in m/s). Using the calibrated model, we simulated 3D nonlinear ground motions at the LVD for Mw6.6 rupture scenarios on the HCF using an Iwan-type, multi-yield-surface technique. We use a two-step method where the computationally expensive nonlinear calculations were carried out in a small domain with the plane wave excitation along the bottom boundary obtained from a full-domain 3D linear finite-fault simulation. Our nonlinear MCE simulation results show that peak ground velocities (PGVs) and peak ground accelerations (PGAs) as high as 72 cm/s and 0.55 g, respectively, can be expected at the crest of the LVD. Compared with linear ground motion simulation results, our results show that Iwan nonlinear damping reduces PGAs on the dam crest by up to a factor of 8 and increasingly depletes the high-frequency content of the waves toward the dam crest. We find horizontal relative displacements of the material inside the dam of up to [Formula: see text] and up to [Formula: see text] of vertical subsidence, equivalent to 1% of the dam height.

     
    more » « less
  5. ; ; ( , IEEE Transactions on Neural Networks and Learning Systems)
    Disentangling the sources of visual motion in a dynamic scene during self-movement or ego motion is important for autonomous navigation and tracking. In the dynamic image segments of a video frame containing independently moving objects, optic flow relative to the next frame is the sum of the motion fields generated due to camera and object motion. The traditional ego-motion estimation methods assume the scene to be static, and the recent deep learning-based methods do not separate pixel velocities into object- and ego-motion components. We propose a learning-based approach to predict both ego-motion parameters and object-motion field (OMF) from image sequences using a convolutional autoencoder while being robust to variations due to the unconstrained scene depth. This is achieved by: 1) training with continuous ego-motion constraints that allow solving for ego-motion parameters independently of depth and 2) learning a sparsely activated overcomplete ego-motion field (EMF) basis set, which eliminates the irrelevant components in both static and dynamic segments for the task of ego-motion estimation. In order to learn the EMF basis set, we propose a new differentiable sparsity penalty function that approximates the number of nonzero activations in the bottleneck layer of the autoencoder and enforces sparsity more effectively than L1- and L2-norm-based penalties. Unlike the existing direct ego-motion estimation methods, the predicted global EMF can be used to extract OMF directly by comparing it against the optic flow. Compared with the state-of-the-art baselines, the proposed model performs favorably on pixelwise object- and ego-motion estimation tasks when evaluated on real and synthetic data sets of dynamic scenes. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email