skip to main content

Explore Research Products in the NSF-PAR

Title: Mixed Reality Communication System for Procedural Tasks
We design a volumetric communication system for remote assistance of procedural medical tasks. The system allows a remote expert to spatially guide a local operator using a real-time volumetric representation of the patient. Guidance is provided by voice, virtual hand metaphor, and annotations performed in situ. We include the feedback we received from the medical professionals and early NASA TLX [5] data on the cognitive load of the system.  more » « less
Award ID(s):
2026505 2117656
NSF-PAR ID:
10351677
Author(s) / Creator(s):
; ; ; ; ; ; ; ;
Date Published:
Journal Name:
AVI 2022: Proceedings of the 2022 International Conference on Advanced Visual Interfaces
Page Range / eLocation ID:
1 to 3
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ; ( , 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR))
    ABSTRACT Medical procedures are an essential part of healthcare delivery, and the acquisition of procedural skills is a critical component of medical education. Unfortunately, procedural skill is not evenly distributed among medical providers. Skills may vary within departments or institutions, and across geographic regions, depending on the provider’s training and ongoing experience. We present a mixed reality real-time communication system to increase access to procedural skill training and to improve remote emergency assistance. Our system allows a remote expert to guide a local operator through a medical procedure. RGBD cameras capture a volumetric view of the local scene including the patient, the operator, and the medical equipment. The volumetric capture is augmented onto the remote expert’s view to allow the expert to spatially guide the local operator using visual and verbal instructions. We evaluated our mixed reality communication system in a study in which experts teach the ultrasound-guided placement of a central venous catheter (CVC) to students in a simulation setting. The study compares state-of-theart video communication against our system. The results indicate that our system enhances and offers new possibilities for visual communication compared to video teleconference-based training. 
    more » « less
  2. ; ; ; ; ; ; ; ( , 2022 8th International Conference of the Immersive Learning Research Network (iLRN))
    We present our work in progress, a real-time mixed reality communication system for remote assistance in medical emergency situations. 3D cameras capture the emergency situation and send volumetric data to a remote expert. The remote expert sees the volumetric scene through mixed reality glasses and guides an operator to the patient. The local operator receives audio and visual guidance augmented onto the mixed reality headset. We compare the mixed reality system against traditional video communication in a user study on a CPR emergency simulation. We evaluate task performance, cognitive load, and user interaction. The results will help to better understand the benefits of using augmented and volumetric information in medical emergency procedures. 
    more » « less
  3. ; ; ; ; ; ; ; ; ( , 15th International Conference on Education and New Learning Technologies)
    Over the last two years, we built, evaluated, and published a volumetric communication system for volumetric-based training and assistantship of medical personnel. Technical aspects of the system have been published at HICCS conference. In this paper, we discuss a follow-up work: the design and evaluation of a mixed reality real-time communication system for remote assistance during CPR emergencies. Our system allows an expert to guide a first responder, remotely, on how to give first aid. RGBD cameras capture a volumetric view of the local scene including the patient, the first responder, and the environment. The volumetric capture is augmented onto the remote expert's view to spatially guide the first responder using visual and verbal instructions. We evaluate the mixed reality communication system in a research study in which participants face a simulated emergency. The first responder moves the patient to the recovery position and performs chest compressions as well as mouth-to-mask ventilation. Our study compares mixed reality against videoconferencing-based assistance using CPR performance measures, cognitive workload surveys, and semi-structured interviews. We find that more visual communication including gestures and objects is used by the remote expert when assisting in mixed reality compared to videoconferencing. Moreover, the performance and the workload of the first responder during simulation do not differ significantly between the two technologies. 
    more » « less
  4. ( , Proceedings of the Annual Hawaii International Conference on System Sciences)
    We present the design of a mixed reality (MR) telehealth training system that aims to close the gap between in-person and distance training and re-training for medical procedures. Our system uses real-time volumetric capture as a means for communicating and relating spatial information between the non-colocated trainee and instructor. The system's design is based on a requirements elicitation study performed in situ, at a medical school simulation training center. The focus is on the lightweight real-time transmission of volumetric data - meaning the use of consumer hardware, easy and quick deployment, and low-demand computations. We evaluate the MR system design by analyzing the workload for the users during medical training. We compare in-person, video, and MR training workloads. The results indicate that the overall workload for central line placement training with MR does not increase significantly compared to video communication. Our work shows that, when designed strategically together with domain experts, an MR communication system can be used effectively for complex medical procedural training without increasing the overall workload for users significantly. Moreover, MR systems offer new opportunities for teaching due to spatial information, hand tracking, and augmented communication. 
    more » « less
  5. ( , Proceedings of the 56th Hawaii International Conference on System Sciences)
    Abstract We present the design of a mixed reality (MR) telehealth training system that aims to close the gap between in-person and distance training and re-training for medical procedures. Our system uses real-time volumetric capture as a means for communicating and relating spatial information between the non-colocated trainee and instructor. The system's design is based on a requirements elicitation study performed in situ, at a medical school simulation training center. The focus is on the lightweight real-time transmission of volumetric data - meaning the use of consumer hardware, easy and quick deployment, and low-demand computations. We evaluate the MR system design by analyzing the workload for the users during medical training. We compare in-person, video, and MR training workloads. The results indicate that the overall workload for central line placement training with MR does not increase significantly compared to video communication. Our work shows that, when designed strategically together with domain experts, an MR communication system can be used effectively for complex medical procedural training without increasing the overall workload for users significantly. Moreover, MR systems offer new opportunities for teaching due to spatial information, hand tracking, and augmented communication. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email