Search for: All records

Accurate instrument segmentation in the endoscopic vision of minimally invasive surgery is challenging due to complex instruments and environments. Deep learning techniques have shown competitive performance in recent years. However, deep learning usually requires a large amount of labeled data to achieve accurate prediction, which poses a significant workload. To alleviate this workload, we propose an active learning-based framework to generate synthetic images for efficient neural network training. In each active learning iteration, a small number of informative unlabeled images are first queried by active learning and manually labeled. Next, synthetic images are generated based on these selected images. The instruments and backgrounds are cropped out and randomly combined with blending and fusion near the boundary. The proposed method leverages the advantage of both active learning and synthetic images. The effectiveness of the proposed method is validated on two sinus surgery datasets and one intraabdominal surgery dataset. The results indicate a considerable performance improvement, especially when the size of the annotated dataset is small. All the code is open-sourced at: https://github.com/HaonanPeng/active_syn_generator 

This paper proposes a low-cost interface and refined digital twin for the Raven-II surgical robot. Previous simulations of the Raven-II, e.g. via the Asynchronous Multibody Framework (AMBF), presented salient drawbacks, including control inputs inconsistent with Raven-II software, and lack of stable, high-fidelity physical contact simulations. This work bridges both of these gaps, both (1) enabling robust, simulated contact mechanics for dynamic physical interactions with the Raven-II, and (2) developing a universal input format for both simulated and physical platforms. The method furthermore proposes a low cost, commodity game-controller interface for controlling both virtual and real realizations of Raven-II, thus greatly reducing the barrier to access for Raven-II research and collaboration. Overall, this work aims to eliminate the inconsistencies between simulated and real representations of the Raven-II. Such a development can expand the reach of surgical robotics research. Namely, providing end-to-end transparency between the simulated AMBF and physical Raven-II platforms enables a software testbed previously unavailable, e.g. for training real surgeons, for creating digital synthetic datasets, or for prototyping novel architectures like shared control strategies. Experiments validate this transparency by comparing joint trajectories between digital twin and physical testbed given identical inputs. This work may be extended and incorporated into recent efforts in developing modular or common software infrastructures for both simulation and control of real robotic devices, such as the Collaborative Robotics Toolkit (CRTK). 

Accurate semantic image segmentation from medical imaging can enable intelligent vision-based assistance in robot-assisted minimally invasive surgery. The human body and surgical procedures are highly dynamic. While machine-vision presents a promising approach, sufficiently large training image sets for robust performance are either costly or unavailable. This work examines three novel generative adversarial network (GAN) methods of providing usable synthetic tool images using only surgical background images and a few real tool images. The best of these three novel approaches generates realistic tool textures while preserving local background content by incorporating both a style preservation and a content loss component into the proposed multi-level loss function. The approach is quantitatively evaluated, and results suggest that the synthetically generated training tool images enhance UNet tool segmentation performance. More specifically, with a random set of 100 cadaver and live endoscopic images from the University of Washington Sinus Dataset, the UNet trained with synthetically generated images using the presented method resulted in 35.7% and 30.6% improvement over using purely real images in mean Dice coefficient and Intersection over Union scores, respectively. This study is promising towards the use of more widely available and routine screening endoscopy to preoperatively generate synthetic training tool images for intraoperative UNet tool segmentation. 

Surgical robots have been introduced to operating rooms over the past few decades due to their high sensitivity, small size, and remote controllability. The cable-driven nature of many surgical robots allows the systems to be dexterous and lightweight, with diameters as low as 5mm. However, due to the slack and stretch of the cables and the backlash of the gears, inevitable uncertainties are brought into the kinematics calcu- lation [1]. Since the reported end effector position of surgical robots like RAVEN-II [2] is directly calculated using the motor encoder measurements and forward kinematics, it may contain relatively large error up to 10mm, whereas semi-autonomous functions being introduced into abdominal surgeries require position inaccuracy of at most 1mm. To resolve the problem, a cost-effective, real-time and data-driven pipeline for robot end effector position precision estimation is proposed and tested on RAVEN-II. Analysis shows an improved end effector position error of around 1mm RMS traversing through the entire robot workspace without high-resolution motion tracker. The open source code, data sets, videos, and user guide can be found at //github.com/HaonanPeng/RAVEN Neural Network Estimator.