An official website of the United States government
In today’s world, augmented reality and virtual reality (AR/VR) technologies have become more accessible to the public than ever. This brings the possibility of immersive learning to the forefront of education for future generations. However, there is still much to discover and improve in using these technologies to analyze and understand learning. This paper explores the utilization of data captured through AR/VR headsets during an immersive training program for industrial robotics. This includes data on time spent, eye gaze, and hand movement during a range of activities to track a learner’s understanding of the content and intelligently estimate learner confidence within these environments using deep learning. Leveraging a dataset that comprises responses and confidence levels from 10 individuals across 35 questions, we aim to improve the uses and applicability of confidence estimation. We explore the possibility of training a model using learners’ data to dynamically fine-tune lessons and activities for each individual, thereby improving performance. We demonstrate that a pre-trained compact LSTM classification model can be fine-tuned with relatively small data, for enhanced performance on an individual basis for better personalized learning.
more »
« less
This content will become publicly available on April 1, 2027
Hand tracking using wearable wrist imaging
The human hand is dexterous and versatile, allowing it to interact with physical and virtual environments. The ability to track hand movements during daily activities could be of use in the development of spatial computing, virtual and augmented reality, robotics and prosthetics. However, current techniques based on cameras, strain and inertial sensors, and electromyography sensors have limited view angles and hand positions, have constrained hand activities and sensations, and can track only discrete hand gestures, respectively. Here we report a fully integrated, wireless and wearable ultrasound imaging wristband that is combined with an artificial intelligence algorithm. The wristband can continuously track arbitrary hand configurations of the five fingers and the palm in real time during daily activities with a delay of less than 120 ms. We show that the wristband can be used for intuitive and versatile controls in virtual-reality and robotic-hand applications.
more »
« less
- Award ID(s):
- 2430106
- PAR ID:
- 10712179
- Publisher / Repository:
- Springer Nature
- Date Published:
- Journal Name:
- Nature Electronics
- Volume:
- 9
- Issue:
- 4
- ISSN:
- 2520-1131
- Page Range / eLocation ID:
- 426 to 440
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Numerous applications of Virtual Reality (VR) and Augmented Reality (AR) continue to emerge. However, many of the current mechanisms to provide input in those environments still require the user to perform actions (e.g., press a number of buttons, tilt a stick) that are not natural or intuitive. It would be desirable to enable users of 3D virtual environments to use natural hand gestures to interact with the environments. The implementation of a glove capable of tracking the movement and configuration of a user’s hand has been pursued by multiple groups in the past. One of the most recent approaches consists of tracking the motion of the hand and fingers using miniature sensor modules with magnetic and inertial sensors. Unfortunately, the limited quality of the signals from those sensors and the frequent deviation from the assumptions made in the design of their operations have prevented the implementation of a tracking glove able to achieve high performance and large-scale acceptance. This paper describes our development of a proof-of-concept glove that incorporates motion sensors and a signal processing algorithm designed to maintain high tracking performance even in locations that are challenging to these sensors, (e.g., where the geomagnetic field is distorted by nearby ferromagnetic objects). We describe the integration of the required components, the rationale and outline of the tracking algorithms and the virtual reality environment in which the tracking results drive the movements of the model of a hand. We also describe the protocol that will be used to evaluate the performance of the glove.more » « less
-
Chen, Jessie Y; Fragomeni, G (Ed.)Numerous applications of Virtual Reality (VR) and Augmented Reality (AR) continue to emerge. However, many of the current mechanisms to provide input in those environments still require the user to perform actions (e.g., press a number of buttons, tilt a stick) that are not natural or intuitive. It would be desirable to enable users of 3D virtual environments to use natural hand gestures to interact with the environments. The implementation of a glove capable of tracking the movement and configuration of a user’s hand has been pursued by multiple groups in the past. One of the most recent approaches consists of tracking the motion of the hand and fingers using miniature sensor modules with magnetic and inertial sensors. Unfortunately, the limited quality of the signals from those sensors and the frequent deviation from the assumptions made in the design of their operations have prevented the implementation of a tracking glove able to achieve high performance and large-scale acceptance. This paper describes our development of a proof-of-concept glove that incorporates motion sensors and a signal processing algorithm designed to maintain high tracking performance even in locations that are challenging to these sensors, (e.g., where the geomagnetic field is distorted by nearby ferromagnetic objects). We describe the integration of the required components, the rationale and outline of the tracking algorithms and the virtual reality environment in which the tracking results drive the movements of the model of a hand. We also describe the protocol that will be used to evaluate the performance of the glove.more » « less
-
In this study, we developed a new haptic–mixed reality intravenous (HMR-IV) needle insertion simulation system, providing a bimanual haptic interface integrated into a mixed reality system with programmable variabilities considering real clinical environments. The system was designed for nursing students or healthcare professionals to practice IV needle insertion into a virtual arm with unlimited attempts under various changing insertion conditions (e.g., skin: color, texture, stiffness, friction; vein: size, shape, location depth, stiffness, friction). To achieve accurate hand–eye coordination under dynamic mixed reality scenarios, two different haptic devices (Dexmo and Geomagic Touch) and a standalone mixed reality system (HoloLens 2) were integrated and synchronized through multistep calibration for different coordinate systems (real world, virtual world, mixed reality world, haptic interface world, HoloLens camera). In addition, force-profile-based haptic rendering proposed in this study was able to successfully mimic the real tactile feeling of IV needle insertion. Further, a global hand-tracking method, combining two depth sensors (HoloLens and Leap Motion), was developed to accurately track a haptic glove and simulate grasping a virtual hand with force feedback. We conducted an evaluation study with 20 participants (9 experts and 11 novices) to measure the usability of the HMR-IV simulation system with user performance under various insertion conditions. The quantitative results from our own metric and qualitative results from the NASA Task Load Index demonstrate the usability of our system.more » « less
-
While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience. By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.more » « less
