skip to main content


Title: FaceSense: Sensing Face Touch with an Ear-worn System
Face touch is an unconscious human habit. Frequent touching of sensitive/mucosal facial zones (eyes, nose, and mouth) increases health risks by passing pathogens into the body and spreading diseases. Furthermore, accurate monitoring of face touch is critical for behavioral intervention. Existing monitoring systems only capture objects approaching the face, rather than detecting actual touches. As such, these systems are prone to false positives upon hand or object movement in proximity to one's face (e.g., picking up a phone). We present FaceSense, an ear-worn system capable of identifying actual touches and differentiating them between sensitive/mucosal areas from other facial areas. Following a multimodal approach, FaceSense integrates low-resolution thermal images and physiological signals. Thermal sensors sense the thermal infrared signal emitted by an approaching hand, while physiological sensors monitor impedance changes caused by skin deformation during a touch. Processed thermal and physiological signals are fed into a deep learning model (TouchNet) to detect touches and identify the facial zone of the touch. We fabricated prototypes using off-the-shelf hardware and conducted experiments with 14 participants while they perform various daily activities (e.g., drinking, talking). Results show a macro-F1-score of 83.4% for touch detection with leave-one-user-out cross-validation and a macro-F1-score of 90.1% for touch zone identification with a personalized model.  more » « less
Award ID(s):
1846541 2037267 2132112 2322879
NSF-PAR ID:
10303235
Author(s) / Creator(s):
 ;  ;  ;  ;  ;  ;  ;  
Date Published:
Journal Name:
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Volume:
5
Issue:
3
ISSN:
2474-9567
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Synopsis

    Every stage of organismal life history is being challenged by global warming. Many species are already experiencing temperatures approaching their physiological limits; this is particularly true for ectothermic species, such as lizards. Embryos are markedly sensitive to thermal insult. Here, we demonstrate that temperatures currently experienced in natural nesting areas can modify gene expression levels and induce neural and craniofacial malformations in embryos of the lizard Anolis sagrei. Developmental abnormalities ranged from minor changes in facial structure to significant disruption of anterior face and forebrain. The first several days of postoviposition development are particularly sensitive to this thermal insult. These results raise new concern over the viability of ectothermic species under contemporary climate change. Herein, we propose and test a novel developmental hypothesis that describes the cellular and developmental origins of those malformations: cell death in the developing forebrain and abnormal facial induction due to disrupted Hedgehog signaling. Based on similarities in the embryonic response to thermal stress among distantly related species, we propose that this developmental hypothesis represents a common embryonic response to thermal insult among amniote embryos. Our results emphasize the importance of adopting a broad, multidisciplinary approach that includes both lab and field perspectives when trying to understand the future impacts of anthropogenic change on animal development.

     
    more » « less
  2. Across a plethora of social situations, we touch others in natural and intuitive ways to share thoughts and emotions, such as tapping to get one’s attention or caressing to soothe one’s anxiety. A deeper understanding of these human-to-human interactions will require, in part, the precise measurement of skin-to-skin physical contact. Among prior efforts, each measurement approach exhibits certain constraints, e.g., motion trackers do not capture the precise shape of skin surfaces, while pressure sensors impede skin-to-skin contact. In contrast, this work develops an interference-free 3D visual tracking system using a depth camera to measure the contact attributes between the bare hand of a toucher and the forearm of a receiver. The toucher’s hand is tracked as a posed and positioned mesh by fitting a hand model to detected 3D hand joints, whereas a receiver’s forearm is extracted as a 3D surface updated upon repeated skin contact. Based on a contact model involving point clouds, the spatiotemporal changes of hand-to-forearm contact are decomposed as six, high-resolution, time-series contact attributes, i.e., contact area, indentation depth, absolute velocity, and three orthogonal velocity components, together with contact duration. To examine the system’s capabilities and limitations, two types of experiments were performed. First, to evaluate its ability to discern human touches, one person delivered cued social messages, e.g., happiness, anger, sympathy, to another person using their preferred gestures. The results indicated that messages and gestures, as well as the identities of the touchers, were readily discerned from their contact attributes. Second, the system’s spatiotemporal accuracy was validated against measurements from independent devices, including an electromagnetic motion tracker, sensorized pressure mat, and laser displacement sensor. While validated here in the context of social communication, this system is extendable to human touch interactions such as maternal care of infants and massage therapy. 
    more » « less
  3. Mobile devices typically rely on entry-point and other one-time authentication mechanisms such as a password, PIN, fingerprint, iris, or face. But these authentication types are prone to a wide attack vector and worse 1 INTRODUCTION Currently smartphones are predominantly protected a patterned password is prone to smudge attacks, and fingerprint scanning is prone to spoof attacks. Other forms of attacks include video capture and shoulder surfing. Given the increasingly important roles smartphones play in e-commerce and other operations where security is crucial, there lies a strong need of continuous authentication mechanisms to complement and enhance one-time authentication such that even if the authentication at the point of login gets compromised, the device is still unobtrusively protected by additional security measures in a continuous fashion. The research community has investigated several continuous authentication mechanisms based on unique human behavioral traits, including typing, swiping, and gait. To this end, we focus on investigating physiological traits. While interacting with hand-held devices, individuals strive to achieve stability and precision. This is because a certain degree of stability is required in order to manipulate and interact successfully with smartphones, while precision is needed for tasks such as touching or tapping a small target on the touch screen (Sitov´a et al., 2015). As a result, to achieve stability and precision, individuals tend to develop their own postural preferences, such as holding a phone with one or both hands, supporting hands on the sides of upper torso and interacting, keeping the phone on the table and typing with the preferred finger, setting the phone on knees while sitting crosslegged and typing, supporting both elbows on chair handles and typing. On the other hand, physiological traits, such as hand-size, grip strength, muscles, age, 424 Ray, A., Hou, D., Schuckers, S. and Barbir, A. Continuous Authentication based on Hand Micro-movement during Smartphone Form Filling by Seated Human Subjects. DOI: 10.5220/0010225804240431 In Proceedings of the 7th International Conference on Information Systems Security and Privacy (ICISSP 2021), pages 424-431 ISBN: 978-989-758-491-6 Copyrightc 2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved still, once compromised, fail to protect the user’s account and data. In contrast, continuous authentication, based on traits of human behavior, can offer additional security measures in the device to authenticate against unauthorized users, even after the entry-point and one-time authentication has been compromised. To this end, we have collected a new data-set of multiple behavioral biometric modalities (49 users) when a user fills out an account recovery form in sitting using an Android app. These include motion events (acceleration and angular velocity), touch and swipe events, keystrokes, and pattern tracing. In this paper, we focus on authentication based on motion events by evaluating a set of score level fusion techniques to authenticate users based on the acceleration and angular velocity data. The best EERs of 2.4% and 6.9% for intra- and inter-session respectively, are achieved by fusing acceleration and angular velocity using Nandakumar et al.’s likelihood ratio (LR) based score fusion. 
    more » « less
  4. Heating, ventilation and air-conditioning (HVAC) systems have been adopted to create comfortable, healthy and safe indoor environments. In the control loop, the technical feature of the human demand-oriented supply can help operate HVAC effectively. Among many technical options, real time monitoring based on feedback signals from end users has been frequently reported as a critical technology to confirm optimizing building performance. Recent studies have incorporated human thermal physiology signals and thermal comfort/discomfort status as real-time feedback signals. A series of human subject experiments used to be conducted by primarily adopting subjective questionnaire surveys in a lab-setting study, which is limited in the application for reality. With the help of advanced technologies, physiological signals have been detected, measured and processed by using multiple technical formats, such as wearable sensors. Nevertheless, they mostly require physical contacts with the skin surface in spite of the small physical dimension and compatibility with other wearable accessories, such as goggles, and intelligent bracelets. Most recently, a low cost small infrared camera has been adopted for monitoring human facial images, which could detect the facial skin temperature and blood perfusion in a contactless way. Also, according to latest pilot studies, a conventional digital camera can generate infrared images with the help of new methods, such as the Euler video magnification technology. Human thermal comfort/discomfort poses can also be detected by video methods without contacting human bodies and be analyzed by the skeleton keypoints model. In this review, new sensing technologies were summarized, their cons and pros were discussed, and extended applications for the demand-oriented ventilation were also reviewed as potential development and applications. 
    more » « less
  5. null (Ed.)
    Heating, ventilation and air-conditioning (HVAC) systems have been adopted to create comfortable, healthy and safe indoor environments. In the control loop, the technical feature of the human demand-oriented supply can help operate HVAC effectively. Among many technical options, real time monitoring based on feedback signals from end users has been frequently reported as a critical technology to confirm optimizing building performance. Recent studies have incorporated human thermal physiologysignals and thermal comfort/discomfort status as real-time feedback signals. A series of human subject experiments used to be conducted by primarily adopting subjective questionnaire surveys in a lab-setting study, which is limited in the application for reality. With the help of advanced technologies, physiological signals have been detected, measured and processed by using multiple technical formats, such as wearable sensors. Nevertheless, they mostly require physical contacts with the skin surface in spite of the small physical dimension and compatibility with other wearable accessories, such as goggles, and intelligent bracelets. Most recently, a low cost small infrared camera has been adopted for monitoring human facial images, which could detect the facial skin temperature and blood perfusion in a contactless way. Also, according to latest pilot studies, a conventional digital camera can generate infrared images with the help of new methods, such as the Euler video magnification technology. Human thermal comfort/discomfort poses can also be detected by video methods without contacting human bodies and be analyzed by the skeleton keypoints model. In this review, new sensing technologies were summarized, their cons and pros were discussed, and extended applications for the demand-oriented ventilation were also reviewed as potential development and applications. 
    more » « less