An official website of the United States government
Abstract Laparoscopic surgery has a notably high learning curve, hindering typical approaches to training. Due to unique challenges that are not present in open surgery (the hinge effect, small field of view (FoV), lack of depth perception, and small workspace), a surgical resident may be delayed in participating in laparoscopic surgery until later in residency. Having a narrow window to complete highly specialized training can lead to graduates feeling under-prepared for solo practice. Additionally, delayed introduction may expose trainees to fewer than 200 laparoscopic cases. Therefore, there is a need for surgical residents to increase both their caseload and training window without compromising patient safety. This project aims to develop and test a proof-of-concept prototype that uses granular jamming technology to controllably vary the force required to move a laparoscopic tool. By increasing tool resistance, the device helps prevents accidental injury to important nearby anatomical structures such as urinary tract, vasculature, and/or bowel. Increasing the safety of laparoscopic surgery would allow residents to begin their training earlier, gaining exposure and confidence. A device to adjust tool resistance has benefits to the experienced surgeon as well – surgeries require continuous tool adjustment and tension, resulting in fatigue. Increasing tool resistance can assist surgeons in situations requiring continuous tension and can also provide safety against sudden movements. This investigational device was prototyped using SolidWorks CAD software, then 3D printed and assessed with a laparoscopic box trainer.
more »
« less
Multicamera 3D Viewpoint Adjustment for Robotic Surgery via Deep Reinforcement Learning
While robot-assisted minimally invasive surgery (RMIS) procedures afford a variety of benefits over open surgery and manual laparoscopic operations (including increased tool dexterity, reduced patient pain, incision size, trauma and recovery time, and lower infection rates [ 1 ], lack of spatial awareness remains an issue. Typical laparoscopic imaging can lack sufficient depth cues and haptic feedback, if provided, rarely reflects realistic tissue–tool interactions. This work is part of a larger ongoing research effort to reconstruct 3D surfaces using multiple viewpoints in RMIS to increase visual perception. The manual placement and adjustment of multicamera systems in RMIS are nonideal and prone to error [ 2 ], and other autonomous approaches focus on tool tracking and do not consider reconstruction of the surgical scene [ 3 , 4 , 5 ]. The group’s previous work investigated a novel, context-aware autonomous camera positioning method [ 6 ], which incorporated both tool location and scene coverage for multiple camera viewpoint adjustments. In this paper, the authors expand upon this prior work by implementing a streamlined deep reinforcement learning approach between optimal viewpoints calculated using the prior method [ 6 ] which encourages discovery of otherwise unobserved and additional camera viewpoints. Combining the framework and robustness of the previous work with the efficiency and additional viewpoints of the augmentations presented here results in improved performance and scene coverage promising towards real-time implementation.
more »
« less
- Award ID(s):
- 2101107
- PAR ID:
- 10326935
- Date Published:
- Journal Name:
- Journal of Medical Robotics Research
- Volume:
- 06
- Issue:
- 01n02
- ISSN:
- 2424-905X
- Page Range / eLocation ID:
- 2140003
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Haptic feedback can render real-time force interactions with computer simulated objects. In several telerobotic applications, it is desired that a haptic simulation reflects a physical task space or interaction accurately. This is particularly true when excessive applied force can result in disastrous consequences, as with the case of robot-assisted minimally invasive surgery (RMIS) and tissue damage. Since force cannot be directly measured in RMIS, non-contact methods are desired. A promising direction of non-contact force estimation involves the primary use of vision sensors to estimate deformation. However, the required fidelity of non-contact force rendering of deformable interaction to maintain surgical operator performance is not well established. This work attempts to empirically evaluate the degree to which haptic feedback may deviate from ground truth yet result in acceptable teleoperated performance in a simulated RMIS-based palpation task. A preliminary user-study is conducted to verify the utility of the simulation platform, and the results of this work have implications in haptic feedback for RMIS and inform guidelines for vision-based tool-tissue force estimation. An adaptive thresholding method is used to collect the minimum and maximum tolerable errors in force orientation and magnitude of presented haptic feedback to maintain sufficient performance.more » « less
-
Significance: Laparoscopic surgery presents challenges in localizing oncological margins due to poor contrast between healthy and malignant tissues. Optical properties can uniquely identify various tissue types and disease states with high sensitivity and specificity, making it a promising tool for surgical guidance. Although spatial frequency domain imaging (SFDI) effectively measures quantitative optical properties, its deployment in laparoscopy is challenging due to the constrained imaging environment. Thus, there is a need for compact structured illumination techniques to enable accurate, quantitative endogenous contrast in minimally invasive surgery. Aim: We introduce a compact, two-camera laparoscope that incorporates both active stereo depth estimation and speckle-illumination SFDI (si-SFDI) to map profile-corrected, pixel-level absorption (μa), and reduced scattering (μ′s) optical properties in images of tissues with complex geometries. Approach: We used a multimode fiber-coupled 639-nm laser illumination to generate high-contrast speckle patterns on the object. These patterns were imaged through a modified commercial stereo laparoscope for optical property estimation via si-SFDI. Compared with the original si-SFDI work, which required ≥10 images of randomized speckle patterns for accurate optical property estimations, our approach approximates the DC response using a laser speckle reducer (LSR) and consequently requires only two images. In addition, we demonstrate 3D profilometry using active stereo from low-coherence RGB laser flood illumination. Sample topography was then used to correct for measured intensity variations caused by object height and surface angle differences with respect to a calibration phantom. The low-contrast RGB speckle pattern was blurred using an LSR to approximate incoherent white light illumination. We validated profile-corrected si-SFDI against conventional SFDI in phantoms with simple and complex geometries, as well as in a human finger in vivo time-series constriction study. Results: Laparoscopic si-SFDI optical property measurements agreed with conventional SFDI measurements when measuring flat tissue phantoms, exhibiting an error of 6.4% for absorption and 5.8% for reduced scattering. Profile-correction improved the accuracy for measurements of phantoms with complex geometries, particularly for absorption, where it reduced the error by 23.7%. An in vivo finger constriction study further validated laparoscopic si-SFDI, demonstrating an error of 8.2% for absorption and 5.8% for reduced scattering compared with conventional SFDI. Moreover, the observed trends in optical properties due to physiological changes were consistent with previous studies. Conclusions: Our stereo-laparoscopic implementation of si-SFDI provides a simple method to obtain accurate optical property maps through a laparoscope for flat and complex geometries. This has the potential to provide quantitative endogenous contrast for minimally invasive surgical guidance.more » « less
-
In modern industrial manufacturing processes, robotic manipulators are routinely used in the assembly, packaging, and material handling operations. During production, changing end-of-arm tooling is frequently necessary for process flexibility and reuse of robotic resources. In conventional operation, a tool changer is sometimes employed to load and unload end-effectors, however, the robot must be manually taught to locate the tool changers by operators via a teach pendant. During tool change teaching, the operator takes considerable effort and time to align the master and tool side of the coupler by adjusting the motion speed of the robotic arm and observing the alignment from different viewpoints. In this paper, a custom robotic system, the NeXus, was programmed to locate and change tools automatically via an RGB-D camera. The NeXus was configured as a multi-robot system for multiple tasks including assembly, bonding, and 3D printing of sensor arrays, solar cells, and microrobot prototypes. Thus, different tools are employed by an industrial robotic arm to position grippers, printers, and other types of end-effectors in the workspace. To improve the precision and cycle-time of the robotic tool change, we mounted an eye-in-hand RGB-D camera and employed visual servoing to automate the tool change process. We then compared the teaching time of the tool location using this system and compared the cycle time with those of 6 human operators in the manual mode. We concluded that the tool location time in automated mode, on average, more than two times lower than the expert human operators.more » « less
-
Current commercially available robotic minimally invasive surgery (RMIS) platforms provide no haptic feedback of tool interactions with the surgical environment. As a consequence, novice robotic surgeons must rely exclusively on visual feedback to sense their physical interactions with the surgical environment. This technical limitation can make it challenging and time-consuming to train novice surgeons to proficiency in RMIS. Extensive prior research has demonstrated that incorporating haptic feedback is effective at improving surgical training task performance. However, few studies have investigated the utility of providing feedback of multiple modalities of haptic feedback simultaneously (multi-modality haptic feedback) in this context, and these studies have presented mixed results regarding its efficacy. Furthermore, the inability to generalize and compare these mixed results has limited our ability to understand why they can vary significantly between studies. Therefore, we have developed a generalized, modular multi-modality haptic feedback and data acquisition framework leveraging the real-time data acquisition and streaming capabilities of the Robot Operating System (ROS). In our preliminary study using this system, participants complete a peg transfer task using a da Vinci robot while receiving haptic feedback of applied forces, contact accelerations, or both via custom wrist-worn haptic devices. Results highlight the capability of our system in running systematic comparisons between various single and dual-modality haptic feedback approaches.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1142/S2424905X21400031
