skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: MiSleep: Human Sleep Posture Identification from Deep Learning Augmented Millimeter-Wave Wireless Systems
In this work, we proposeMiSleep, a deep learning augmented millimeter-wave (mmWave) wireless system to monitor human sleep posture by predicting the 3D location of the body joints of a person during sleep. Unlike existing vision- or wearable-based sleep monitoring systems,MiSleepis not privacy-invasive and does not require users to wear anything on their body.MiSleepleverages knowledge of human anatomical features and deep learning models to solve challenges in existing mmWave devices with low-resolution and aliased imaging, and specularity in signals.MiSleepbuilds the model by learning the relationship between mmWave reflected signals and body postures from thousands of existing samples. Since a practical sleep also involves sudden toss-turns, which could introduce errors in posture prediction,MiSleepdesigns a state machine based on the reflected signals to classify the sleeping states into rest or toss-turn, and predict the posture only during the rest states. We evaluateMiSleepwith real data collected from Commercial-Off-The-Shelf mmWave devices for 8 volunteers of diverse ages, genders, and heights performing different sleep postures. We observe thatMiSleepidentifies the toss-turn events start time and duration within 1.25 s and 1.7 s of the ground truth, respectively, and predicts the 3D location of body joints with a median error of 1.3 cm only and can perform even under the blankets, with accuracy on par with the existing vision-based system, unlocking the potential of mmWave systems for privacy-noninvasive at-home healthcare applications.  more » « less
Award ID(s):
2144505 2018966 1910853
PAR ID:
10492330
Author(s) / Creator(s):
;
Publisher / Repository:
ACM
Date Published:
Journal Name:
ACM Transactions on Internet of Things
ISSN:
2691-1914
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies)
    We propose MiShape, a millimeter-wave (mmWave) wireless signal based imaging system that generates high-resolution human silhouettes and predicts 3D locations of body joints. The system can capture human motions in real-time under low light and low-visibility conditions. Unlike existing vision-based motion capture systems, MiShape is privacy non-invasive and can generalize to a wide range of motion tracking applications at-home. To overcome the challenges with low-resolution, specularity, and aliasing in images from Commercial-Off-The-Shelf (COTS) mmWave systems, MiShape designs deep learning models based on conditional Generative Adversarial Networks and incorporates the rules of human biomechanics. We have customized MiShape for gait monitoring, but the model is well adaptive to any tracking applications with limited fine-tuning samples. We experimentally evaluate MiShape with real data collected from a COTS mmWave system for 10 volunteers, with diverse ages, gender, height, and somatotype, performing different poses. Our experimental results demonstrate that MiShape delivers high-resolution silhouettes and accurate body poses on par with an existing vision-based system, and unlocks the potential of mmWave systems, such as 5G home wireless routers, for privacy-noninvasive healthcare applications. 
    more » « less
  2. ; ; ; ; ; ; ; ; ; (, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies)
    Sleep is a vital physiological state that significantly impacts overall health. Continuous monitoring of sleep posture, heart rate, respiratory rate, and body movement is crucial for diagnosing and managing sleep disorders. Current monitoring solutions often disrupt natural sleep due to discomfort or raise privacy and instrumentation concerns. We introduce PillowSense, a fabric-based sleep monitoring system seamlessly integrated into a pillowcase. PillowSense utilizes a dual-layer fabric design. The top layer comprises conductive fabrics for sensing electrocardiogram (ECG) and surface electromyogram (sEMG), while the bottom layer features pressure-sensitive fabrics to monitor sleep location and movement. The system processes ECG and sEMG signals sequentially to infer multiple sleep variables and incorporates an adversarial neural network to enhance posture classification accuracy. We fabricate prototypes using off-the-shelf hardware and conduct both lab-based and in-the-wild longitudinal user studies to evaluate the system's effectiveness. Across 151 nights and 912.2 hours of real-world sleep data, the system achieves an F1 score of 88% for classifying seven sleep postures, and clinically-acceptable accuracy in vital sign monitoring. PillowSense's comfort, washability, and robustness in multi-user scenarios underscore its potential for unobtrusive, large-scale sleep monitoring. 
    more » « less
  3. ; (, Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing)
    This paper proposes MilliPose, a system that facilitates full human body silhouette imaging and 3D pose estimation from millimeterwave (mmWave) devices. Unlike existing vision-based motion capture systems, MilliPose is not privacy-invasive and is capable of working under obstructions, poor visibility, and low light conditions. MilliPose leverages machine-learning models based on conditional Generative Adversarial Networks and Recurrent Neural Network to solve the challenges of poor resolution, specularity, and variable reflectivity with existing mmWave imaging systems. Our preliminary results show the efficacy of MilliPose in accurately predicting body joint locations under natural human movement. 
    more » « less
  4. ; ; ; (, Proceedings of the ACM on Measurement and Analysis of Computing Systems)
    mmWave signals form a critical component of 5G and next-generation wireless networks, which are also being increasingly considered for sensing the environment around us to enable ubiquitous IoT applications. In this context, this paper leverages the properties of mmWave signals for tracking 3D finger motion for interactive IoT applications. While conventional vision-based solutions break down under poor lighting, occlusions, and also suffer from privacy concerns, mmWave signals work under typical occlusions and non-line-of-sight conditions, while being privacy-preserving. In contrast to prior works on mmWave sensing that focus on predefined gesture classification, this work performs continuous 3D finger motion tracking. Towards this end, we first observe via simulations and experiments that the small size of fingers coupled with specular reflections do not yield stable mmWave reflections. However, we make an interesting observation that focusing on the forearm instead of the fingers can provide stable reflections for 3D finger motion tracking. Muscles that activate the fingers extend through the forearm, whose motion manifests as vibrations on the forearm. By analyzing the variation in phases of reflected mmWave signals from the forearm, this paper designs mm4Arm, a system that tracks 3D finger motion. Nontrivial challenges arise due to the high dimensional search space, complex vibration patterns, diversity across users, hardware noise, etc. mm4Arm exploits anatomical constraints in finger motions and fuses them with machine learning architectures based on encoder-decoder and ResNets in enabling accurate tracking. A systematic performance evaluation with 10 users demonstrates a median error of 5.73° (location error of 4.07 mm) with robustness to multipath and natural variation in hand position/orientation. The accuracy is also consistent under non-line-of-sight conditions and clothing that might occlude the forearm. mm4Arm runs on smartphones with a latency of 19 ms and low energy overhead. 
    more » « less
  5. ; ; ; ; ; ; ; ; ; ; et al (, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies)
    In-bed postures offer valuable information about an individual's sleep quality and overall health conditions, particularly for patients with sleep apnea. However, current in-bed posture classification systems lack privacy-friendly and easy-to-install options. Furthermore, existing solutions do not consider variations between patients and are typically trained only once, neglecting the utilization of time consistency and unlabeled data from new patients. To address these limitations, this paper builds on a seismic sensor to introduce a novel sleep posture framework, which comprises two main components, namely, the Multi-Granularity Supervised Contrastive Learning (MGSCL) module and the ensemble Online Adaptation (oa) module. Unlike most existing contrastive learning frameworks that operate at the sample level, MGSCL leverages multi-granular information, operating not only at the sample level but also at the group level. The oa module enables the model to adapt to new patient data while ensuring time consistency in sleep posture predictions. Additionally, it quantifies model uncertainty to generate weighted predictions, further enhancing performance. Evaluated on a dataset of 100 patients collected at a clinical research center, MGSCLoa achieved an average accuracy of 91.67% and an average F1 score of 91.53% with only 40 seconds of labeled data per posture. In a Phase 2 evaluation with 11 participants over 13 nights in home settings, the framework reached an average accuracy of 85.37% and a weighted F1 score of 83.59% using just 3 minutes of labeled data per common posture for each participant. These results underscore the potential of seismic sensor-based in-bed posture classification for assessing sleep quality and related health conditions. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email