Accurate semantic image segmentation from medical imaging can enable intelligent vision-based assistance in robot-assisted minimally invasive surgery. The human body and surgical procedures are highly dynamic. While machine-vision presents a promising approach, sufficiently large training image sets for robust performance are either costly or unavailable. This work examines three novel generative adversarial network (GAN) methods of providing usable synthetic tool images using only surgical background images and a few real tool images. The best of these three novel approaches generates realistic tool textures while preserving local background content by incorporating both a style preservation and a content loss component into the proposed multi-level loss function. The approach is quantitatively evaluated, and results suggest that the synthetically generated training tool images enhance UNet tool segmentation performance. More specifically, with a random set of 100 cadaver and live endoscopic images from the University of Washington Sinus Dataset, the UNet trained with synthetically generated images using the presented method resulted in 35.7% and 30.6% improvement over using purely real images in mean Dice coefficient and Intersection over Union scores, respectively. This study is promising towards the use of more widely available and routine screening endoscopy to preoperatively generate synthetic training tool images for intraoperative UNet tool segmentation.
more »
« less
Real-time vision-based surgical tool segmentation with robot kinematics prior
Robot-assisted minimally invasive surgery com-
bines the skills and techniques of highly-trained surgeons
with the robustness and precision of machines. Several advantages include precision beyond human dexterity alone, greater
kinematic degrees of freedom at the surgical tool tip, and
possibilities in remote surgical practices through teleoperation.
Nevertheless, obtaining accurate force feedback during surgical
operations remains a challenging hurdle. Though direct force
sensing using tool tip mounted sensors is theoretically possible, it is not amenable to required sterilization procedures.
Vision-based force estimation according to real-time analysis
of tissue deformation serves as a promising alternative. In this
application, along with numerous related research in robot-
assisted minimally invasive surgery, segmentation of surgical
instruments in endoscopic images is a prerequisite. Thus, a
surgical tool segmentation algorithm robust to partial occlusion
is proposed using DFT shape matching of robot kinematics
shape prior (u) fused with log likelihood mask (Q) in the
Opponent color space to generate final mask (U). Implemented
on the Raven II surgical robot system, a real-time performance
robust to tool tip orientation and up to 6 fps without GPU
acceleration is achieved.
more »
« less
- Award ID(s):
- 1637444
- NSF-PAR ID:
- 10209007
- Date Published:
- Journal Name:
- 2018 International Symposium on Medical Robotics (ISMR), Atlanta, GA, 2018
- Page Range / eLocation ID:
- 1 to 6
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Dynamic 3D reconstruction of surgical cavities is essential in a wide range of computer-assisted surgical intervention applications, including but not limited to surgical guidance, pre-operative image registration and vision-based force estimation. According to a survey on vision based 3D reconstruction for abdominal minimally invasive surgery (MIS) [1], real-time 3D reconstruction and tissue deformation recovery remain open challenges to researchers. The main challenges include specular reflections from the wet tissue surface and the highly dynamic nature of abdominal surgical scenes. This work aims to overcome these obstacles by using multiple viewpoint and independently moving RGB cameras to generate an accurate measurement of tissue deformation at the volume of interest (VOI), and proposes a novel efficient camera pairing algorithm. Experimental results validate the proposed camera grouping and pair sequencing, and were evaluated with the Raven-II [2] surgical robot system for tool navigation, the Medtronic Stealth Station s7 surgical navigation system for real- time camera pose monitoring, and the Space Spider white light scanner to derive the ground truth 3D model.more » « less
-
more » « less
Abstract Surgical resection is the primary and most effective treatment for most patients with solid tumors. However, patients suffer from postoperative recurrence and metastasis. In the past years, emerging nanotechnology has led the way to minimally invasive, precision and intelligent oncological surgery after the rapid development of minimally invasive surgical technology. Advanced nanotechnology in the construction of nanomaterials (NMs) for precision imaging‐guided surgery (IGS) as well as surgery‐assisted synergistic therapy is summarized, thereby unlocking the advantages of nanotechnology in multimodal IGS‐assisted precision synergistic cancer therapy. First, mechanisms and principles of NMs to surgical targets are briefly introduced. Multimodal imaging based on molecular imaging technologies provides a practical method to achieve intraoperative visualization with high resolution and deep tissue penetration. Moreover, multifunctional NMs synergize surgery with adjuvant therapy (e.g., chemotherapy, immunotherapy, phototherapy) to eliminate residual lesions. Finally, key issues in the development of ideal theranostic NMs associated with surgical applications and challenges of clinical transformation are discussed to push forward further development of NMs for multimodal IGS‐assisted precision synergistic cancer therapy.
-
null (Ed.)Robot-assisted minimally invasive surgery has made a substantial impact in operating rooms over the past few decades with their high dexterity, small tool size, and impact on adoption of minimally invasive techniques. In recent years, intelligence and different levels of surgical robot autonomy have emerged thanks to the medical robotics endeavors at numerous academic institutions and leading surgical robot companies. To accelerate interaction within the research community and prevent repeated development, we propose the Collaborative Robotics Toolkit (CRTK), a common API for the RAVEN-II and da Vinci Research Kit (dVRK) - two open surgical robot platforms installed at more than 40 institutions worldwide. CRTK has broadened to include other robots and devices, including simulated robotic systems and industrial robots. This common API is a community software infrastructure for research and education in cutting edge human-robot collaborative areas such as semi-autonomous teleoperation and medical robotics. This paper presents the concepts, design details and the integration of CRTK with physical robot systems and simulation platforms.more » « less
-
null (Ed.)Robot-assisted minimally invasive surgery has made a substantial impact in operating rooms over the past few decades with their high dexterity, small tool size, and impact on adoption of minimally invasive techniques. In recent years, intelligence and different levels of surgical robot autonomy have emerged thanks to the medical robotics endeavors at numerous academic institutions and leading surgical robot companies. To accelerate interaction within the research community and prevent repeated development, we propose the Collaborative Robotics Toolkit (CRTK), a common API for the RAVEN-II and da Vinci Research Kit (dVRK) - two open surgical robot platforms installed at more than 40 institutions worldwide. CRTK has broadened to include other robots and devices, including simulated robotic systems and industrial robots. This common API is a community software infrastructure for research and education in cutting edge human-robot collaborative areas such as semi-autonomous teleoperation and medical robotics. This paper presents the concepts, design details and the integration of CRTK with physical robot systems and simulation platforms.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/ISMR.2018.8333305
