The emerging potential of augmented reality (AR)
to improve 3D medical image visualization for diagnosis, by
immersing the user into 3D morphology is further enhanced
with the advent of wireless head-mounted displays (HMD).
Such information-immersive capabilities may also enhance planning
and visualization of interventional procedures. To this
end, we introduce a computational platform to generate an
augmented reality holographic scene that fuses pre-operative
magnetic resonance imaging (MRI) sets, segmented anatomical
structures, and an actuated model of an interventional robot
for performing MRI-guided and robot-assisted interventions. The
interface enables the operator to manipulate the presented images
and rendered structures using voice and gestures, as well as to
robot control. The software uses forbidden-region virtual fixtures
that alerts the operator of collisions with vital structures. The
platform was tested with a HoloLens HMD in silico. To address
the limited computational power of the HMD, we deployed the
platform on a desktop PC with two-way communication to the
HMD. Operation studies demonstrated the functionality and
underscored the importance of interface customization to fit a
particular operator and/or procedure, as well as the need for
on-site studies to assess its merit in the clinical realm.
Index Terms—augmented reality, robot-assistance, imageguided
interventions.
more »
« less
Evaluation of how users interface with holographic augmented reality surgical scenes: Interactive planning MR‐Guided prostate biopsies
User interfaces play a vital role in the planning and execution of an interventional procedure. The objective of this study is to investigate the effect of using different user interfaces for planning transrectal robot‐assisted MR‐guided prostate biopsy (MRgPBx) in an augmented reality (AR) environment.
End‐user studies were conducted by simulating an MRgPBx system with end‐ and side‐firing modes. The information from the system to the operator was rendered on HoloLens as an output interface. Joystick, mouse/keyboard, and holographic menus were used as input interfaces to the system.
The studies indicated that using a joystick improved the interactive capacity and enabled operator to plan MRgPBx in less time. It efficiently captures the operator's commands to manipulate the augmented environment representing the state of MRgPBx system.
The study demonstrates an alternative to conventional input interfaces to interact and manipulate an AR environment within the context of MRgPBx planning.
- NSF-PAR ID:
- 10450166
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- The International Journal of Medical Robotics and Computer Assisted Surgery
- Volume:
- 17
- Issue:
- 5
- ISSN:
- 1478-5951
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The emerging potential of augmented reality (AR) to improve 3D medical image visualization for diagnosis, by immersing the user into 3D morphology is further enhanced with the advent of wireless head-mounted displays (HMD). Such information-immersive capabilities may also enhance planning and visualization of interventional procedures. To this end, we introduce a computational platform to generate an augmented reality holographic scene that fuses pre-operative magnetic resonance imaging (MRI) sets, segmented anatomical structures, and an actuated model of an interventional robot for performing MRI-guided and robot-assisted interventions. The interface enables the operator to manipulate the presented images and rendered structures using voice and gestures, as well as to robot control. The software uses forbidden-region virtual fixtures that alerts the operator of collisions with vital structures. The platform was tested with a HoloLens HMD in silico. To address the limited computational power of the HMD, we deployed the platform on a desktop PC with two-way communication to the HMD. Operation studies demonstrated the functionality and underscored the importance of interface customization to fit a particular operator and/or procedure, as well as the need for on-site studies to assess its merit in the clinical realm.more » « less
-
more » « less
Abstract Successful surgical operations are characterized by preplanning routines to be executed during actual surgical operations. To achieve this, surgeons rely on the experience acquired from the use of cadavers, enabling technologies like virtual reality (VR) and clinical years of practice. However, cadavers, having no dynamism and realism as they lack blood, can exhibit limited tissue degradation and shrinkage, while current VR systems do not provide amplified haptic feedback. This can impact surgical training increasing the likelihood of medical errors. This work proposes a novel Mixed Reality Combination System (MRCS) that pairs Augmented Reality (AR) technology and an inertial measurement unit (IMU) sensor with 3D printed, collagen-based specimens that can enhance task performance like planning and execution. To achieve this, the MRCS charts out a path prior to a user task execution based on a visual, physical, and dynamic environment on the state of a target object by utilizing surgeon-created virtual imagery that, when projected onto a 3D printed biospecimen as AR, reacts visually to user input on its actual physical state. This allows a real-time user reaction of the MRCS by displaying new multi-sensory virtual states of an object prior to performing on the actual physical state of that same object enabling effective task planning. Tracked user actions using an integrated 9-Degree of Freedom IMU demonstrate task execution This demonstrates that a user, with limited knowledge of specific anatomy, can, under guidance, execute a preplanned task. In addition, to surgical planning, this system can be generally applied in areas such as construction, maintenance, and education.
-
In many complex tasks, a remote expert may need to assist a local user or to guide his or her actions in the local user's environment. Existing solutions also allow multiple users to collaborate remotely using high-end Augmented Reality (AR) and Virtual Reality (VR) head-mounted displays (HMD). In this paper, we propose a portable remote collaboration approach, with the integration of AR and VR devices, both running on mobile platforms, to tackle the challenges of existing approaches. The AR mobile platform processes the live video and measures the 3D geometry of the local environment of a local user. The 3D scene is then transited and rendered in the remote side on a mobile VR device, along with a simple and effective user interface, which allows a remote expert to easily manipulate the 3D scene on the VR platform and to guide the local user to complete tasks in the local environment.more » « less
-
more » « less
Abstract Background This study presents user evaluation studies to assess the effect of information rendered by an interventional planning software on the operator's ability to plan transrectal magnetic resonance (MR)‐guided prostate biopsies using actuated robotic manipulators.
Methods An intervention planning software was developed based on the clinical workflow followed for MR‐guided transrectal prostate biopsies. The software was designed to interface with a generic virtual manipulator and simulate an intervention environment using 2D and 3D scenes. User studies were conducted with urologists using the developed software to plan virtual biopsies.
Results User studies demonstrated that urologists with prior experience in using 3D software completed the planning less time. 3D scenes were required to control all degrees‐of‐freedom of the manipulator, while 2D scenes were sufficient for planar motion of the manipulator.
Conclusions The study provides insights on using 2D versus 3D environment from a urologist's perspective for different operational modes of MR‐guided prostate biopsy systems.
