skip to main content

Title: FaceSense: Sensing Face Touch with an Ear-worn System
Face touch is an unconscious human habit. Frequent touching of sensitive/mucosal facial zones (eyes, nose, and mouth) increases health risks by passing pathogens into the body and spreading diseases. Furthermore, accurate monitoring of face touch is critical for behavioral intervention. Existing monitoring systems only capture objects approaching the face, rather than detecting actual touches. As such, these systems are prone to false positives upon hand or object movement in proximity to one's face (e.g., picking up a phone). We present FaceSense, an ear-worn system capable of identifying actual touches and differentiating them between sensitive/mucosal areas from other facial areas. Following a multimodal approach, FaceSense integrates low-resolution thermal images and physiological signals. Thermal sensors sense the thermal infrared signal emitted by an approaching hand, while physiological sensors monitor impedance changes caused by skin deformation during a touch. Processed thermal and physiological signals are fed into a deep learning model (TouchNet) to detect touches and identify the facial zone of the touch. We fabricated prototypes using off-the-shelf hardware and conducted experiments with 14 participants while they perform various daily activities (e.g., drinking, talking). Results show a macro-F1-score of 83.4% for touch detection with leave-one-user-out cross-validation and a macro-F1-score of 90.1% for touch more » zone identification with a personalized model. « less
Authors:
 ;  ;  ;  ;  ;  ;  ;  
Award ID(s):
1846541 2037267 2132112
Publication Date:
NSF-PAR ID:
10303235
Journal Name:
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Volume:
5
Issue:
3
ISSN:
2474-9567
Sponsoring Org:
National Science Foundation
More Like this
  1. Across a plethora of social situations, we touch others in natural and intuitive ways to share thoughts and emotions, such as tapping to get one’s attention or caressing to soothe one’s anxiety. A deeper understanding of these human-to-human interactions will require, in part, the precise measurement of skin-to-skin physical contact. Among prior efforts, each measurement approach exhibits certain constraints, e.g., motion trackers do not capture the precise shape of skin surfaces, while pressure sensors impede skin-to-skin contact. In contrast, this work develops an interference-free 3D visual tracking system using a depth camera to measure the contact attributes between the bare hand of a toucher and the forearm of a receiver. The toucher’s hand is tracked as a posed and positioned mesh by fitting a hand model to detected 3D hand joints, whereas a receiver’s forearm is extracted as a 3D surface updated upon repeated skin contact. Based on a contact model involving point clouds, the spatiotemporal changes of hand-to-forearm contact are decomposed as six, high-resolution, time-series contact attributes, i.e., contact area, indentation depth, absolute velocity, and three orthogonal velocity components, together with contact duration. To examine the system’s capabilities and limitations, two types of experiments were performed. First, to evaluatemore »its ability to discern human touches, one person delivered cued social messages, e.g., happiness, anger, sympathy, to another person using their preferred gestures. The results indicated that messages and gestures, as well as the identities of the touchers, were readily discerned from their contact attributes. Second, the system’s spatiotemporal accuracy was validated against measurements from independent devices, including an electromagnetic motion tracker, sensorized pressure mat, and laser displacement sensor. While validated here in the context of social communication, this system is extendable to human touch interactions such as maternal care of infants and massage therapy.« less
  2. Heating, ventilation and air-conditioning (HVAC) systems have been adopted to create comfortable, healthy and safe indoor environments. In the control loop, the technical feature of the human demand-oriented supply can help operate HVAC effectively. Among many technical options, real time monitoring based on feedback signals from end users has been frequently reported as a critical technology to confirm optimizing building performance. Recent studies have incorporated human thermal physiology signals and thermal comfort/discomfort status as real-time feedback signals. A series of human subject experiments used to be conducted by primarily adopting subjective questionnaire surveys in a lab-setting study, which is limited in the application for reality. With the help of advanced technologies, physiological signals have been detected, measured and processed by using multiple technical formats, such as wearable sensors. Nevertheless, they mostly require physical contacts with the skin surface in spite of the small physical dimension and compatibility with other wearable accessories, such as goggles, and intelligent bracelets. Most recently, a low cost small infrared camera has been adopted for monitoring human facial images, which could detect the facial skin temperature and blood perfusion in a contactless way. Also, according to latest pilot studies, a conventional digital camera can generate infraredmore »images with the help of new methods, such as the Euler video magnification technology. Human thermal comfort/discomfort poses can also be detected by video methods without contacting human bodies and be analyzed by the skeleton keypoints model. In this review, new sensing technologies were summarized, their cons and pros were discussed, and extended applications for the demand-oriented ventilation were also reviewed as potential development and applications.« less
  3. Mobile devices typically rely on entry-point and other one-time authentication mechanisms such as a password, PIN, fingerprint, iris, or face. But these authentication types are prone to a wide attack vector and worse 1 INTRODUCTION Currently smartphones are predominantly protected a patterned password is prone to smudge attacks, and fingerprint scanning is prone to spoof attacks. Other forms of attacks include video capture and shoulder surfing. Given the increasingly important roles smartphones play in e-commerce and other operations where security is crucial, there lies a strong need of continuous authentication mechanisms to complement and enhance one-time authentication such that even if the authentication at the point of login gets compromised, the device is still unobtrusively protected by additional security measures in a continuous fashion. The research community has investigated several continuous authentication mechanisms based on unique human behavioral traits, including typing, swiping, and gait. To this end, we focus on investigating physiological traits. While interacting with hand-held devices, individuals strive to achieve stability and precision. This is because a certain degree of stability is required in order to manipulate and interact successfully with smartphones, while precision is needed for tasks such as touching or tapping a small target on themore »touch screen (Sitov´a et al., 2015). As a result, to achieve stability and precision, individuals tend to develop their own postural preferences, such as holding a phone with one or both hands, supporting hands on the sides of upper torso and interacting, keeping the phone on the table and typing with the preferred finger, setting the phone on knees while sitting crosslegged and typing, supporting both elbows on chair handles and typing. On the other hand, physiological traits, such as hand-size, grip strength, muscles, age, 424 Ray, A., Hou, D., Schuckers, S. and Barbir, A. Continuous Authentication based on Hand Micro-movement during Smartphone Form Filling by Seated Human Subjects. DOI: 10.5220/0010225804240431 In Proceedings of the 7th International Conference on Information Systems Security and Privacy (ICISSP 2021), pages 424-431 ISBN: 978-989-758-491-6 Copyrightc 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved still, once compromised, fail to protect the user’s account and data. In contrast, continuous authentication, based on traits of human behavior, can offer additional security measures in the device to authenticate against unauthorized users, even after the entry-point and one-time authentication has been compromised. To this end, we have collected a new data-set of multiple behavioral biometric modalities (49 users) when a user fills out an account recovery form in sitting using an Android app. These include motion events (acceleration and angular velocity), touch and swipe events, keystrokes, and pattern tracing. In this paper, we focus on authentication based on motion events by evaluating a set of score level fusion techniques to authenticate users based on the acceleration and angular velocity data. The best EERs of 2.4% and 6.9% for intra- and inter-session respectively, are achieved by fusing acceleration and angular velocity using Nandakumar et al.’s likelihood ratio (LR) based score fusion.« less
  4. Heating, ventilation and air-conditioning (HVAC) systems have been adopted to create comfortable, healthy and safe indoor environments. In the control loop, the technical feature of the human demand-oriented supply can help operate HVAC effectively. Among many technical options, real time monitoring based on feedback signals from end users has been frequently reported as a critical technology to confirm optimizing building performance. Recent studies have incorporated human thermal physiologysignals and thermal comfort/discomfort status as real-time feedback signals. A series of human subject experiments used to be conducted by primarily adopting subjective questionnaire surveys in a lab-setting study, which is limited in the application for reality. With the help of advanced technologies, physiological signals have been detected, measured and processed by using multiple technical formats, such as wearable sensors. Nevertheless, they mostly require physical contacts with the skin surface in spite of the small physical dimension and compatibility with other wearable accessories, such as goggles, and intelligent bracelets. Most recently, a low cost small infrared camera has been adopted for monitoring human facial images, which could detect the facial skin temperature and blood perfusion in a contactless way. Also, according to latest pilot studies, a conventional digital camera can generate infrared imagesmore »with the help of new methods, such as the Euler video magnification technology. Human thermal comfort/discomfort poses can also be detected by video methods without contacting human bodies and be analyzed by the skeleton keypoints model. In this review, new sensing technologies were summarized, their cons and pros were discussed, and extended applications for the demand-oriented ventilation were also reviewed as potential development and applications.« less
  5. Vital signs (e.g., heart and respiratory rate) are indicative for health status assessment. Efforts have been made to extract vital signs using radio frequency (RF) techniques (e.g., Wi-Fi, FMCW, UWB), which offer a non-touch solution for continuous and ubiquitous monitoring without users’ cooperative efforts. While RF-based vital signs monitoring is user-friendly, its robustness faces two challenges. On the one hand, the RF signal is modulated by the periodic chest wall displacement due to heartbeat and breathing in a nonlinear manner. It is inherently hard to identify the fundamental heart and respiratory rates (HR and RR) in the presence of higher order harmonics of them and intermodulation between HR and RR, especially when they have overlapping frequency bands. On the other hand, the inadvertent body movements may disturb and distort the RF signal, overwhelming the vital signals, thus inhibiting the parameter estimation of the physiological movement (i.e., heartbeat and breathing). In this paper, we propose DeepVS, a deep learning approach that addresses the aforementioned challenges from the non-linearity and inadvertent movements for robust RF-based vital signs sensing in a unified manner. DeepVS combines 1D CNN and attention models to exploit local features and temporal correlations. Moreover, it leverages a two-stream schememore »to integrate features from both time and frequency domains. Additionally, DeepVS unifies the estimation of HR and RR with a multi-head structure, which only adds limited extra overhead (<1%) to the existing model, compared to doubling the overhead using two separate models for HR and RR respectively. Our experiments demonstrate that DeepVS achieves 80-percentile HR/RR errors of 7.4/4.9 beat/breaths per minute (bpm) on a challenging dataset, as compared to 11.8/7.3 bpm of a non-learning solution. Besides, an ablation study has been conducted to quantify the effectiveness of DeepVS.« less