More Like this

1. Real-time vision-based surgical tool segmentation with robot kinematics prior

   https://doi.org/10.1109/ISMR.2018.8333305  

   Su, Yun-Hsuan; Huang, Kevin; Hannaford, Blake (March 2018, 2018 International Symposium on Medical Robotics (ISMR), Atlanta, GA, 2018)

   null (Ed.)

   Robot-assisted minimally invasive surgery com- bines the skills and techniques of highly-trained surgeons with the robustness and precision of machines. Several advantages include precision beyond human dexterity alone, greater kinematic degrees of freedom at the surgical tool tip, and possibilities in remote surgical practices through teleoperation. Nevertheless, obtaining accurate force feedback during surgical operations remains a challenging hurdle. Though direct force sensing using tool tip mounted sensors is theoretically possible, it is not amenable to required sterilization procedures. Vision-based force estimation according to real-time analysis of tissue deformation serves as a promising alternative. In this application, along with numerous related research in robot- assisted minimally invasive surgery, segmentation of surgical instruments in endoscopic images is a prerequisite. Thus, a surgical tool segmentation algorithm robust to partial occlusion is proposed using DFT shape matching of robot kinematics shape prior (u) fused with log likelihood mask (Q) in the Opponent color space to generate final mask (U). Implemented on the Raven II surgical robot system, a real-time performance robust to tool tip orientation and up to 6 fps without GPU acceleration is achieved. 

   more » « less
2. Surgical Tool Segmentation with Pose-Informed Morphological Polar Transform of Endoscopic Images

   https://doi.org/10.1142/S2424905X22410033  

   Huang, Kevin; Chitrakar, Digesh; Jiang, Wenfan; Yung, Isabella; Su, Yun-Hsuan (June 2022, Journal of Medical Robotics Research)

   This paper presents a tool-pose-informed variable center morphological polar transform to enhance segmentation of endoscopic images. The representation, while not loss-less, transforms rigid tool shapes into morphologies consistently more rectangular that may be more amenable to image segmentation networks. The proposed method was evaluated using the U-Net convolutional neural network, and the input images from endoscopy were represented in one of the four different coordinate formats (1) the original rectangular image representation, (2) the morphological polar coordinate transform, (3) the proposed variable center transform about the tool-tip pixel and (4) the proposed variable center transform about the tool vanishing point pixel. Previous work relied on the observations that endoscopic images typically exhibit unused border regions with content in the shape of a circle (since the image sensor is designed to be larger than the image circle to maximize available visual information in the constrained environment) and that the region of interest (ROI) was most ideally near the endoscopic image center. That work sought an intelligent method for, given an input image, carefully selecting between methods (1) and (2) for best image segmentation prediction. In this extension, the image center reference constraint for polar transformation in method (2) is relaxed via the development of a variable center morphological transformation. Transform center selection leads to different spatial distributions of image loss, and the transform-center location can be informed by robot kinematic model and endoscopic image data. In particular, this work is examined using the tool-tip and tool vanishing point on the image plane as candidate centers. The experiments were conducted for each of the four image representations using a data set of 8360 endoscopic images from real sinus surgery. The segmentation performance was evaluated with standard metrics, and some insight about loss and tool location effects on performance are provided. Overall, the results are promising, showing that selecting a transform center based on tool shape features using the proposed method can improve segmentation performance. 

   more » « less
3. Autonomous Neurosurgical Instrument Segmentation Using End-to-End Learning

   Kalavakonda, Niveditha; Hannaford, Blake; Qazi, Zeeshan; Sekhar, Laligam (January 2019, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops)

   Monitoring surgical instruments is an essential task in computer-assisted interventions and surgical robotics. It is also important for navigation, data analysis, skill as- sessment and surgical workflow analysis in conventional surgery. However, there are no standard datasets and benchmarks for tool identification in neurosurgery. To this end, we are releasing a novel neurosurgical instrument seg- mentation dataset called NeuroID for advancing research in the field. Delineating surgical tools from the background requires accurate pixel-wise instrument segmentation. In this paper, we present a comparison between three encoder- decoder approaches to binary segmentation of neurosurgi- cal instruments, where we classify each pixel in the image to be either tool or background. A baseline performance was obtained by using heuristics to combine extracted features. We also extend the analysis to a publicly available robotic instrument segmentation dataset and include its results. The source code for our methods and the neurosurgical instru- ment dataset will be made publicly available1 to facilitate reproducibility. 

   more » « less
4. Experience with Using Synthetic Training Images for Wearable Cognitive Assistance

   Iyengar, R.; Zhang, E.; Satyanarayanan, M. (March 2022, Proceedings of the AAAI Spring Symposium on AI Engineering, 2022)

   Wearable Cognitive Assistance (WCA) applications use computer vision models that require thousands of labeled training images. Capturing and labeling these images requires a substantial amount of work. By using synthetically generated images for training, we avoided this labor-intensive step. The performance of these models was comparable to that of models that were trained on real images. 

   more » « less
5. Enhanced U-Net Tool Segmentation using Hybrid Coordinate Representations of Endoscopic Images

   https://doi.org/10.1109/ISMR48346.2021.9661519  

   Huang, Kevin; Chitrakar, Digesh; Jiang, Wenfan; Su, Yun-Hsuan (November 2021, 2021 International Symposium on Medical Robotics (ISMR))

   This paper presents an approach to enhanced endoscopic tool segmentation combining separate pathways utilizing input images in two different coordinate representations. The proposed method examines U-Net convolutional neural networks with input endoscopic images represented via (1) the original rectangular coordinate format alongside (2) a morphological polar coordinate transformation. To maximize information and the breadth of the endoscope frustrum, imaging sensors are oftentimes larger than the image circle. This results in unused border regions. Ideally, the region of interest is proximal to the image center. The above two observations formed the basis for the morphological polar transformation pathway as an augmentation to typical rectangular input image representations. Results indicate that neither of the two investigated coordinate representations consistently yielded better segmentation performance as compared to the other. Improved segmentation can be achieved with a hybrid approach that carefully selects which of the two pathways to be used for individual input images. Towards that end, two binary classifiers were trained to identify, given an input endoscopic image, which of the two coordinate representation segmentation pathways (rectangular or polar), would result in better segmentation performance. Results are promising and suggest marked improvements using a hybrid pathway selection approach compared to either alone. The experiment used to evaluate the proposed hybrid method utilized a dataset consisting of 8360 endoscopic images from real surgery and evaluated segmentation performance with Dice coefficient and Intersection over Union. The results suggest that on-the-fly polar transformation for tool segmentation is useful when paired with the proposed hybrid tool-segmentation approach. 

   more » « less