Accurate semantic image segmentation from medical imaging can enable intelligent vision-based assistance in robot-assisted minimally invasive surgery. The human body and surgical procedures are highly dynamic. While machine-vision presents a promising approach, sufficiently large training image sets for robust performance are either costly or unavailable. This work examines three novel generative adversarial network (GAN) methods of providing usable synthetic tool images using only surgical background images and a few real tool images. The best of these three novel approaches generates realistic tool textures while preserving local background content by incorporating both a style preservation and a content loss component into the proposed multi-level loss function. The approach is quantitatively evaluated, and results suggest that the synthetically generated training tool images enhance UNet tool segmentation performance. More specifically, with a random set of 100 cadaver and live endoscopic images from the University of Washington Sinus Dataset, the UNet trained with synthetically generated images using the presented method resulted in 35.7% and 30.6% improvement over using purely real images in mean Dice coefficient and Intersection over Union scores, respectively. This study is promising towards the use of more widely available and routine screening endoscopy to preoperatively generate synthetic training tool images for intraoperative UNet tool segmentation.
more »
« less
Real-time vision-based surgical tool segmentation with robot kinematics prior
Robot-assisted minimally invasive surgery com-
bines the skills and techniques of highly-trained surgeons
with the robustness and precision of machines. Several advantages include precision beyond human dexterity alone, greater
kinematic degrees of freedom at the surgical tool tip, and
possibilities in remote surgical practices through teleoperation.
Nevertheless, obtaining accurate force feedback during surgical
operations remains a challenging hurdle. Though direct force
sensing using tool tip mounted sensors is theoretically possible, it is not amenable to required sterilization procedures.
Vision-based force estimation according to real-time analysis
of tissue deformation serves as a promising alternative. In this
application, along with numerous related research in robot-
assisted minimally invasive surgery, segmentation of surgical
instruments in endoscopic images is a prerequisite. Thus, a
surgical tool segmentation algorithm robust to partial occlusion
is proposed using DFT shape matching of robot kinematics
shape prior (u) fused with log likelihood mask (Q) in the
Opponent color space to generate final mask (U). Implemented
on the Raven II surgical robot system, a real-time performance
robust to tool tip orientation and up to 6 fps without GPU
acceleration is achieved.
more »
« less
- Award ID(s):
- 1637444
- NSF-PAR ID:
- 10209007
- Date Published:
- Journal Name:
- 2018 International Symposium on Medical Robotics (ISMR), Atlanta, GA, 2018
- Page Range / eLocation ID:
- 1 to 6
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Dynamic 3D reconstruction of surgical cavities is essential in a wide range of computer-assisted surgical intervention applications, including but not limited to surgical guidance, pre-operative image registration and vision-based force estimation. According to a survey on vision based 3D reconstruction for abdominal minimally invasive surgery (MIS) [1], real-time 3D reconstruction and tissue deformation recovery remain open challenges to researchers. The main challenges include specular reflections from the wet tissue surface and the highly dynamic nature of abdominal surgical scenes. This work aims to overcome these obstacles by using multiple viewpoint and independently moving RGB cameras to generate an accurate measurement of tissue deformation at the volume of interest (VOI), and proposes a novel efficient camera pairing algorithm. Experimental results validate the proposed camera grouping and pair sequencing, and were evaluated with the Raven-II [2] surgical robot system for tool navigation, the Medtronic Stealth Station s7 surgical navigation system for real- time camera pose monitoring, and the Space Spider white light scanner to derive the ground truth 3D model.more » « less
-
Haptic feedback can render real-time force interactions with computer simulated objects. In several telerobotic applications, it is desired that a haptic simulation reflects a physical task space or interaction accurately. This is particularly true when excessive applied force can result in disastrous consequences, as with the case of robot-assisted minimally invasive surgery (RMIS) and tissue damage. Since force cannot be directly measured in RMIS, non-contact methods are desired. A promising direction of non-contact force estimation involves the primary use of vision sensors to estimate deformation. However, the required fidelity of non-contact force rendering of deformable interaction to maintain surgical operator performance is not well established. This work attempts to empirically evaluate the degree to which haptic feedback may deviate from ground truth yet result in acceptable teleoperated performance in a simulated RMIS-based palpation task. A preliminary user-study is conducted to verify the utility of the simulation platform, and the results of this work have implications in haptic feedback for RMIS and inform guidelines for vision-based tool-tissue force estimation. An adaptive thresholding method is used to collect the minimum and maximum tolerable errors in force orientation and magnitude of presented haptic feedback to maintain sufficient performance.more » « less
-
While robot-assisted minimally invasive surgery (RMIS) procedures afford a variety of benefits over open surgery and manual laparoscopic operations (including increased tool dexterity, reduced patient pain, incision size, trauma and recovery time, and lower infection rates [ 1 ], lack of spatial awareness remains an issue. Typical laparoscopic imaging can lack sufficient depth cues and haptic feedback, if provided, rarely reflects realistic tissue–tool interactions. This work is part of a larger ongoing research effort to reconstruct 3D surfaces using multiple viewpoints in RMIS to increase visual perception. The manual placement and adjustment of multicamera systems in RMIS are nonideal and prone to error [ 2 ], and other autonomous approaches focus on tool tracking and do not consider reconstruction of the surgical scene [ 3 , 4 , 5 ]. The group’s previous work investigated a novel, context-aware autonomous camera positioning method [ 6 ], which incorporated both tool location and scene coverage for multiple camera viewpoint adjustments. In this paper, the authors expand upon this prior work by implementing a streamlined deep reinforcement learning approach between optimal viewpoints calculated using the prior method [ 6 ] which encourages discovery of otherwise unobserved and additional camera viewpoints. Combining the framework and robustness of the previous work with the efficiency and additional viewpoints of the augmentations presented here results in improved performance and scene coverage promising towards real-time implementation.more » « less
-
more » « less
Combining laser technology with robotic precision and accuracy promises to introduce significant advances in minimally invasive surgical interventions. Lasers have already become a widespread tool in numerous surgical applications. They are proposed as a replacement for traditional tools (i.e., scalpels and electrocautery devices) to minimize surgical trauma, decrease healing times, and reduce the risk of postoperative complications. Compared to other energy sources, laser energy is wavelength‐dependent, allowing for preferential energy absorption in specific tissue types. This potentially leads to minimizing damage to healthy tissue and increasing surgical outcomes control and quality. Merging robotic control with laser techniques can help physicians achieve more accurate laser aiming and pave the way to automatic control of laser–tissue interactions in closed loop. Herein, a review of the state‐of‐the‐art robotic systems for laser surgery is presented. The goals of this paper are to present recent contributions in advanced intelligent systems for robot‐assisted laser surgery, provide readers with a better understanding of laser optics and the physics of laser–tissue interactions, discuss clinical applications of lasers in surgery, and provide guidance for future systems design.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/ISMR.2018.8333305
