The emerging potential of augmented reality (AR)
to improve 3D medical image visualization for diagnosis, by
immersing the user into 3D morphology is further enhanced
with the advent of wireless head-mounted displays (HMD).
Such information-immersive capabilities may also enhance planning
and visualization of interventional procedures. To this
end, we introduce a computational platform to generate an
augmented reality holographic scene that fuses pre-operative
magnetic resonance imaging (MRI) sets, segmented anatomical
structures, and an actuated model of an interventional robot
for performing MRI-guided and robot-assisted interventions. The
interface enables the operator to manipulate the presented images
and rendered structures using voice and gestures, as well as to
robot control. The software uses forbidden-region virtual fixtures
that alerts the operator of collisions with vital structures. The
platform was tested with a HoloLens HMD in silico. To address
the limited computational power of the HMD, we deployed the
platform on a desktop PC with two-way communication to the
HMD. Operation studies demonstrated the functionality and
underscored the importance of interface customization to fit a
particular operator and/or procedure, as well as the need for
on-site studies to assess its merit in the clinical realm.
Index Terms—augmented reality, robot-assistance, imageguided
interventions.
more »
« less
Towards MRI-guided and actuated tetherless milli-robots: Preoperative planning and modeling of control
Image-guided and robot-assisted surgical procedures are rapidly evolving due to their potential to improve patient management and cost effectiveness. Magnetic Resonance Imaging (MRI) is used for pre-operative planning and is also investigated for real-time intra-operative guidance. A new type of technology is emerging that uses the magnetic field gradients of the MR scanner to maneuver ferromagnetic agents for local delivery of therapeutics. With this approach, MRI is both a sensor and forms a closed-loop controlled entity that behaves as a robot (we refer to them as MRbots). The objective of this paper is to introduce a computational framework for preoperative planning using MRI and modeling of MRbot maneuvering inside tortuous blood vessels. This platform generates a virtual corridor that represents a safety zone inside the vessel that is then used to access the safety of the MRbot maneuvering. In addition, to improve safety we introduce a control that sets speed based on the local curvature of the vessel. The functionality of the framework was then tested on a realistic operational scenario of accessing a neurological lesion, a meningioma. This virtual case study demonstrated the functionality and potential of MRbots as well as revealed two primary challenges: real-time MRI (during propulsion) and the need of very strong gradients for maneuvering small MRbots inside narrow cerebral vessels. Our ongoing research focuses on further developing the computational core, MR tracking methods, and on-line interfacing to the MR scanner.
more »
« less
- Award ID(s):
- 1646566
- NSF-PAR ID:
- 10048943
- Date Published:
- Journal Name:
- Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on
- Page Range / eLocation ID:
- 6440 to 6447
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The emerging potential of augmented reality (AR) to improve 3D medical image visualization for diagnosis, by immersing the user into 3D morphology is further enhanced with the advent of wireless head-mounted displays (HMD). Such information-immersive capabilities may also enhance planning and visualization of interventional procedures. To this end, we introduce a computational platform to generate an augmented reality holographic scene that fuses pre-operative magnetic resonance imaging (MRI) sets, segmented anatomical structures, and an actuated model of an interventional robot for performing MRI-guided and robot-assisted interventions. The interface enables the operator to manipulate the presented images and rendered structures using voice and gestures, as well as to robot control. The software uses forbidden-region virtual fixtures that alerts the operator of collisions with vital structures. The platform was tested with a HoloLens HMD in silico. To address the limited computational power of the HMD, we deployed the platform on a desktop PC with two-way communication to the HMD. Operation studies demonstrated the functionality and underscored the importance of interface customization to fit a particular operator and/or procedure, as well as the need for on-site studies to assess its merit in the clinical realm.more » « less
-
Puig Puig, Anna and (Ed.)Motivated by the potential of holographic augmented reality (AR) to offer an immersive 3D appreciation of morphology and anatomy, the purpose of this work is to develop and assess an interface for image-based planning of prostate interventions with a head-mounted display (HMD). The computational system is a data and command pipeline that links a magnetic resonance imaging (MRI) scanner/data and the operator, that includes modules dedicated to image processing and segmentation, structure rendering, trajectory planning and spatial co-registration. The interface was developed with the Unity3D Engine (C#) and deployed and tested on a HoloLens HMD. For ergonomics in the surgical suite, the system was endowed with hands-free interactive manipulation of images and the holographic scene via hand gestures and voice commands. The system was tested in silico using MRI and ultrasound datasets of prostate phantoms. The holographic AR scene rendered by the HoloLens HMD was subjectively found superior to desktop-based volume or 3D rendering with regard to structure detection and appreciation of spatial relationships, planning access paths and manual co-registration of MRI and Ultrasound. By inspecting the virtual trajectory superimposed to rendered structures and MR images, the operator observes collisions of the needle path with vital structures (e.g. urethra) and adjusts accordingly. Holographic AR interfacing with wireless HMD endowed with hands-free gesture and voice control is a promising technology. Studies need to systematically assess the clinical merit of such systems and needed functionalities.more » « less
-
This work presents a platform that integrates a customized MRI data acquisition scheme with reconstruction and three-dimensional (3D) visualization modules along with a module for controlling an MRI-compatible robotic device to facilitate the performance of robot-assisted, MRI-guided interventional procedures. Using dynamically-acquired MRI data, the computational framework of the platform generates and updates a 3D model representing the area of the procedure (AoP). To image structures of interest in the AoP that do not reside inside the same or parallel slices, the MRI acquisition scheme was modified to collect a multi-slice set of intraoblique to each other slices; which are termed composing slices. Moreover, this approach interleaves the collection of the composing slices so the same k-space segments of all slices are collected during similar time instances. This time matching of the k-space segments results in spatial matching of the imaged objects in the individual composing slices. The composing slices were used to generate and update the 3D model of the AoP. The MRI acquisition scheme was evaluated with computer simulations and experimental studies. Computer simulations demonstrated that k-space segmentation and time-matched interleaved acquisition of these segments provide spatial matching of the structures imaged with composing slices. Experimental studies used the platform to image the maneuvering of an MRI-compatible manipulator that carried tubing filled with MRI contrast agent. In vivo experimental studies to image the abdomen and contrast enhanced heart on free-breathing subjects without cardiac triggering demonstrated spatial matching of imaged anatomies in the composing planes. The described interventional MRI framework could assist in performing real-time MRI-guided interventions.more » « less
-
Magnetic resonance imaging (MRI) is a routinely used imaging technique in medical diagnostics. To enhance the quality of MR images, contrast agents (CAs) are used, which account for nearly 40% of MRI exams in the clinic globally. The most used CAs are gadolinium-based CAs (GBCAs) but the use of GBCAs has been linked with metal-deposition in vital organs. Gadolinium deposition has been shown to be correlated with nephrogenic systemic fibrosis, a fibrosis of the skin and internal organs. Therefore, there is an unmet need for a new CA alternative to GBCAs for T 1 -weighted Ce-MRI. Herein, we designed paramagnetic ferric iron( iii ) ion-chelated poly(lactic- co -glycolic)acid nanoparticle formulation and routinely examined their application in Ce-MRI using clinical and ultra-high-field MRI scanners. Nanoparticles were monodispersed and highly stable at physiological pH over time with the hydrodynamic size of 130 ± 12 nm and polydispersity index of 0.231 ± 0.026. The T 1 -contrast efficacy of the nanoparticles was compared with commercial agent gadopentetate dimeglumine, called Magnevist®, in aqueous phantoms in vitro and then validated in vivo by visualizing an angiographic map in a clinical MRI scanner. Relaxivities of the nanoparticles in an aqueous environment were r 1 = 10.59 ± 0.32 mmol −1 s −1 and r 1 = 3.02 ± 0.14 mmol −1 s −1 at 3.0 T and 14.1 T measured at room temperature and pH 7.4, respectively. The clinically relevant magnetic field relaxivity is three times higher compared to the Magnevist®, a clinical GBCA, signifying its potential applicability in clinical settings. Moreover, iron is an endogenous metal with known metabolic safety, and the polymer and phospholipids used in the nanoconstruct are biodegradable and biocompatible components. These properties further put the proposed T 1 agent in a promising position in contrast-enhanced MRI of patients with any disease conditions.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/IROS.2017.8206550
