skip to main content

Explore Research Products in the NSF-PAR

Title: 3D Visual Tracking to Quantify Physical Contact Interactions in Human-to-Human Touch
Across a plethora of social situations, we touch others in natural and intuitive ways to share thoughts and emotions, such as tapping to get one’s attention or caressing to soothe one’s anxiety. A deeper understanding of these human-to-human interactions will require, in part, the precise measurement of skin-to-skin physical contact. Among prior efforts, each measurement approach exhibits certain constraints, e.g., motion trackers do not capture the precise shape of skin surfaces, while pressure sensors impede skin-to-skin contact. In contrast, this work develops an interference-free 3D visual tracking system using a depth camera to measure the contact attributes between the bare hand of a toucher and the forearm of a receiver. The toucher’s hand is tracked as a posed and positioned mesh by fitting a hand model to detected 3D hand joints, whereas a receiver’s forearm is extracted as a 3D surface updated upon repeated skin contact. Based on a contact model involving point clouds, the spatiotemporal changes of hand-to-forearm contact are decomposed as six, high-resolution, time-series contact attributes, i.e., contact area, indentation depth, absolute velocity, and three orthogonal velocity components, together with contact duration. To examine the system’s capabilities and limitations, two types of experiments were performed. First, to evaluate its ability to discern human touches, one person delivered cued social messages, e.g., happiness, anger, sympathy, to another person using their preferred gestures. The results indicated that messages and gestures, as well as the identities of the touchers, were readily discerned from their contact attributes. Second, the system’s spatiotemporal accuracy was validated against measurements from independent devices, including an electromagnetic motion tracker, sensorized pressure mat, and laser displacement sensor. While validated here in the context of social communication, this system is extendable to human touch interactions such as maternal care of infants and massage therapy.  more » « less
Award ID(s):
1908115
NSF-PAR ID:
10341465
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Frontiers in Physiology
Volume:
13
ISSN:
1664-042X
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ( , Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies)
    null (Ed.)
    This paper proposes and evaluates the use of image classification for detailed, full-body human-robot tactile interaction. A camera positioned below a translucent robot skin captures shadows generated from human touch and infers social gestures from the captured images. This approach enables rich tactile interaction with robots without the need for the sensor arrays used in traditional social robot tactile skins. It also supports the use of touch interaction with non-rigid robots, achieves high-resolution sensing for robots with different sizes and shape of surfaces, and removes the requirement of direct contact with the robot. We demonstrate the idea with an inflatable robot and a standing-alone testing device, an algorithm for recognizing touch gestures from shadows that uses Densely Connected Convolutional Networks, and an algorithm for tracking positions of touch and hovering shadows. Our experiments show that the system can distinguish between six touch gestures under three lighting conditions with 87.5 - 96.0% accuracy, depending on the lighting, and can accurately track touch positions as well as infer motion activities in realistic interaction conditions. Additional applications for this method include interactive screens on inflatable robots and privacy-maintaining robots for the home. 
    more » « less
  2. ; ; ( , IEEE Transactions on Haptics)
    Our daily observations tell us that the delivery of social sentiments and emotions differs between strangers and romantic partners. This work explores how relationship status influences our delivery and perception of social touches and emotions, by evaluating the physics of contact interactions. In a study with human participants, strangers and romantically involved touchers delivered emotional messages to receivers’ forearms. Physical contact interactions were measured using a customized 3D tracking system. The results indicate that strangers and romantic receivers recognize emotional messages with similar accuracy, but with higher levels of valence and arousal between romantic partners. Further investigation into the contact interactions which underlie the higher levels of valence and arousal reveals that a toucher tunes their strategy with their romantic partner. For example, when stroking, romantic touchers use velocities preferential to C-tactile afferents, and maintain contact for longer durations with larger contact areas. Notwithstanding, while we show that relationship intimacy influences the deployment of touch strategies, such impact is relatively subtle compared to distinctions between gestures, emotional messages, and individual preferences. 
    more » « less
  3. ; ; ( , IEEE International Conference on Robotics and Automation)
    null (Ed.)
    An important problem in designing human-robot systems is the integration of human intent and performance in the robotic control loop, especially in complex tasks. Bimanual coordination is a complex human behavior that is critical in many fine motor tasks, including robot-assisted surgery. To fully leverage the capabilities of the robot as an intelligent and assistive agent, online recognition of bimanual coordination could be important. Robotic assistance for a suturing task, for example, will be fundamentally different during phases when the suture is wrapped around the instrument (i.e., making a c- loop), than when the ends of the suture are pulled apart. In this study, we develop an online recognition method of bimanual coordination modes (i.e., the directions and symmetries of right and left hand movements) using geometric descriptors of hand motion. We (1) develop this framework based on ideal trajectories obtained during virtual 2D bimanual path following tasks performed by human subjects operating Geomagic Touch haptic devices, (2) test the offline recognition accuracy of bi- manual direction and symmetry from human subject movement trials, and (3) evalaute how the framework can be used to characterize 3D trajectories of the da Vinci Surgical System’s surgeon-side manipulators during bimanual surgical training tasks. In the human subject trials, our geometric bimanual movement classification accuracy was 92.3% for movement direction (i.e., hands moving together, parallel, or away) and 86.0% for symmetry (e.g., mirror or point symmetry). We also show that this approach can be used for online classification of different bimanual coordination modes during needle transfer, making a C loop, and suture pulling gestures on the da Vinci system, with results matching the expected modes. Finally, we discuss how these online estimates are sensitive to task environment factors and surgeon expertise, and thus inspire future work that could leverage adaptive control strategies to enhance user skill during robot-assisted surgery. 
    more » « less
  4. ; ; ( , IEEE Transactions on Haptics)
    To discriminate the compliance of soft objects, we rely upon spatiotemporal cues in the mechanical deformation of the skin. However, we have few direct observations of skin deformation over time, in particular how its response differs with indentation velocities and depths, and thereby helps inform our perceptual judgments. To help fill this gap, we develop a 3D stereo imaging method to observe contact of the skin’s surface with transparent, compliant stimuli. Experiments with human-subjects, in passive touch, are conducted with stimuli varying in compliance, indentation depth, velocity, and time duration. The results indicate that contact durations greater than 0.4 s are perceptually discriminable. Moreover, compliant pairs delivered at higher velocities are more difficult to discriminate because they induce smaller differences in deformation. In a detailed quantification of the skin’s surface deformation, we find that several, independent cues aid perception. In particular, the rate of change of gross contact area best correlates with discriminability, across indentation velocities and compliances. However, cues associated with skin surface curvature and bulk force are also predictive, for stimuli more and less compliant than skin, respectively. These findings and detailed measurements seek to inform the design of haptic interfaces. 
    more » « less
  5. ; ; ; ; ( , Frontiers in Robotics and AI)
    Principles from human-human physical interaction may be necessary to design more intuitive and seamless robotic devices to aid human movement. Previous studies have shown that light touch can aid balance and that haptic communication can improve performance of physical tasks, but the effects of touch between two humans on walking balance has not been previously characterized. This study examines physical interaction between two persons when one person aids another in performing a beam-walking task. 12 pairs of healthy young adults held a force sensor with one hand while one person walked on a narrow balance beam (2 cm wide x 3.7 m long) and the other person walked overground by their side. We compare balance performance during partnered vs. solo beam-walking to examine the effects of haptic interaction, and we compare hand interaction mechanics during partnered beam-walking vs. overground walking to examine how the interaction aided balance. While holding the hand of a partner, participants were able to walk further on the beam without falling, reduce lateral sway, and decrease angular momentum in the frontal plane. We measured small hand force magnitudes (mean of 2.2 N laterally and 3.4 N vertically) that created opposing torque components about the beam axis and calculated the interaction torque, the overlapping opposing torque that does not contribute to motion of the beam-walker’s body. We found higher interaction torque magnitudes during partnered beam-walking vs . partnered overground walking, and correlation between interaction torque magnitude and reductions in lateral sway. To gain insight into feasible controller designs to emulate human-human physical interactions for aiding walking balance, we modeled the relationship between each torque component and motion of the beam-walker’s body as a mass-spring-damper system. Our model results show opposite types of mechanical elements (active vs . passive) for the two torque components. Our results demonstrate that hand interactions aid balance during partnered beam-walking by creating opposing torques that primarily serve haptic communication, and our model of the torques suggest control parameters for implementing human-human balance aid in human-robot interactions. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email