skip to main content
Explore Scholarly Publications and Datasets in the NSF-PAR

Title: Autonomously Navigating a Surgical Tool Inside the Eye by Learning from Demonstration
A fundamental challenge in retinal surgery is safely navigating a surgical tool to a desired goal position on the retinal surface while avoiding damage to surrounding tissues, a procedure that typically requires tens-of-microns accuracy. In practice, the surgeon relies on depth-estimation skills to localize the tool-tip with respect to the retina and perform the tool-navigation task, which can be prone to human error. To alleviate such uncertainty, prior work has introduced ways to assist the surgeon by estimating the tool-tip distance to the retina and providing haptic or auditory feedback. However, automating the tool-navigation task itself remains unsolved and largely un-explored. Such a capability, if reliably automated, could serve as a building block to streamline complex procedures and reduce the chance for tissue damage. Towards this end, we propose to automate the tool-navigation task by mimicking the perception-action feedback loop of an expert surgeon. Specifically, a deep network is trained to imitate expert trajectories toward various locations on the retina based on recorded visual servoing to a given goal specified by the user. The proposed autonomous navigation system is evaluated in simulation and in real-life experiments using a silicone eye phantom. We show that the network can reliably navigate a more » surgical tool to various desired locations within 137 µm accuracy in phantom experiments and 94 µm in simulation, and generalizes well to unseen situations such as in the presence of auxiliary surgical tools, variable eye backgrounds, and brightness conditions. « less
Authors:
; ; ; ; ; ;
Award ID(s):
1637949
Publication Date:
NSF-PAR ID:
10136845
Journal Name:
ICRA 2019
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ( , 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC))
    Haptic feedback can render real-time force interactions with computer simulated objects. In several telerobotic applications, it is desired that a haptic simulation reflects a physical task space or interaction accurately. This is particularly true when excessive applied force can result in disastrous consequences, as with the case of robot-assisted minimally invasive surgery (RMIS) and tissue damage. Since force cannot be directly measured in RMIS, non-contact methods are desired. A promising direction of non-contact force estimation involves the primary use of vision sensors to estimate deformation. However, the required fidelity of non-contact force rendering of deformable interaction to maintain surgical operator performance is not well established. This work attempts to empirically evaluate the degree to which haptic feedback may deviate from ground truth yet result in acceptable teleoperated performance in a simulated RMIS-based palpation task. A preliminary user-study is conducted to verify the utility of the simulation platform, and the results of this work have implications in haptic feedback for RMIS and inform guidelines for vision-based tool-tissue force estimation. An adaptive thresholding method is used to collect the minimum and maximum tolerable errors in force orientation and magnitude of presented haptic feedback to maintain sufficient performance.
  2. ; ; ; ( , 2020 Fourth IEEE International Conference on Robotic Computing (IRC))
    Laparoscopic surgery presents practical benefits over traditional open surgery, including reduced risk of infection, discomfort and recovery time for patients. Introducing robot systems into surgical tasks provides additional enhancements, including improved precision, remote operation, and an intelligent software layer capable of filtering aberrant motion and scaling surgical maneuvers. However, the software interface in telesurgery also lends itself to potential adversarial cyber attacks. Such attacks can negatively effect both surgeon motion commands and sensory information relayed to the operator. To combat cyber attacks on the latter, one method to enhance surgeon feedback through multiple sensory pathways is to incorporate reliable, complementary forms of information across different sensory modes. Built-in partial redundancies or inferences between perceptual channels, or perception complementarities, can be used both to detect and recover from compromised operator feedback. In surgery, haptic sensations are extremely useful for surgeons to prevent undue and unwanted tissue damage from excessive tool-tissue force. Direct force sensing is not yet deployable due to sterilization requirements of the operating room. Instead, combinations of other sensing methods may be relied upon, such as noncontact model-based force estimation. This paper presents the design of a surgical simulator software that can be used for vision-based non-contact force sensingmore »to inform the perception complementarity of vision and force feedback for telesurgery. A brief user study is conducted to verify the efficacy of graphical force feedback from vision-based force estimation, and suggests that vision may effectively complement direct force sensing.« less
  3. ; ; ; ; ; ; ( , British Journal of Ophthalmology)
    Purpose To investigate relationships between blood pressure and the thickness of single retinal layers in the macula. Methods Participants of the population-based Beijing Eye Study, free of retinal or optic nerve disease, underwent medical and ophthalmological examinations including optical coherence tomographic examination of the macula. Applying a multiple-surface segmentation solution, we automatically segmented the retina into its various layers. Results The study included 2237 participants (mean age 61.8±8.4 years, range 50–93 years). Mean thicknesses of the retinal nerve fibre layer (RNFL), ganglion cell layer (GCL), inner plexiform layer, inner nuclear layer (INL), outer plexiform layer, outer nuclear layer/external limiting membrane, ellipsoid zone, photoreceptor outer segments (POS) and retinal pigment epithelium–Bruch membrane were 31.1±2.3 µm, 39.7±3.5 µm, 38.4±3.3 µm, 34.8±2.0 µm, 28.1±3.0 µm, 79.2±7.3 µm, 22.9±0.6 µm, 19.2±3.3 µm and 20.7±1.4 µm, respectively. In multivariable analysis, higher systolic blood pressure (SBP) and diastolic blood pressure (DBP) were associated with thinner GCL and thicker INL, after adjusting for age, sex and axial length (all p<0.0056). Higher SBP was additionally associated with thinner POS and higher DBP with thinner RNFL. For an elevation of SBP/DBP by 10 mm Hg, the RNFL, GCL, INL and POS changed by 2.0, 3.0, 1.5 and 2.0 µm, respectively. Conclusions Thickness of RNFL, GCL and POS was inversely andmore »INL thickness was positively associated with higher blood pressure, while the thickness of the other retinal layers was not significantly correlated with blood pressure. The findings may be helpful for refinement of the morphometric detection of retinal diseases.« less
  4. ; ; ; ; ; ( , Journal of Medical Devices)
    Abstract Endovascular navigation proficiency requires a significant amount of manual dexterity from surgeons. Objective performance measures derived from endovascular tool tip kinematics have been shown to correlate with expertise; however, such metrics have not yet been used during training as a basis for real-time performance feedback. This paper evaluates a set of velocity-based performance measures derived from guidewire motion to determine their suitability for online performance evaluation and feedback. We evaluated the endovascular navigation skill of 75 participants using three metrics (spectral arc length, average velocity, and idle time) as they steered tools to anatomical targets using a virtual reality simulator. First, we examined the effect of navigation task and experience level on performance and found that novice performance was significantly different from intermediate and expert performance. Then we computed correlations between measures calculated online and spectral arc length, our "gold standard" metric, calculated offline (at the end of the trial, using data from the entire trial). Our results suggest that average velocity and idle time calculated online are strongly and consistently correlated with spectral arc length computed offline, which was not the case when comparing spectral arc length computed online and offline. Average velocity and idle time, both time-domainmore »based performance measures, are therefore more suitable measures than spectral arc length, a frequency-domain based metric, to use as the basis of online performance feedback. Future work is needed to determine how to best provide real-time performance feedback to endovascular surgery trainees based on these metrics.« less
  5. ; ; ; ( , Frontiers in Physiology)
    In many mechanistic medical, biological, physical, and engineered spatiotemporal dynamic models the numerical solution of partial differential equations (PDEs), especially for diffusion, fluid flow and mechanical relaxation, can make simulations impractically slow. Biological models of tissues and organs often require the simultaneous calculation of the spatial variation of concentration of dozens of diffusing chemical species. One clinical example where rapid calculation of a diffusing field is of use is the estimation of oxygen gradients in the retina, based on imaging of the retinal vasculature, to guide surgical interventions in diabetic retinopathy. Furthermore, the ability to predict blood perfusion and oxygenation may one day guide clinical interventions in diverse settings, i.e., from stent placement in treating heart disease to BOLD fMRI interpretation in evaluating cognitive function (Xie et al., 2019 ; Lee et al., 2020 ). Since the quasi-steady-state solutions required for fast-diffusing chemical species like oxygen are particularly computationally costly, we consider the use of a neural network to provide an approximate solution to the steady-state diffusion equation. Machine learning surrogates, neural networks trained to provide approximate solutions to such complicated numerical problems, can often provide speed-ups of several orders of magnitude compared to direct calculation. Surrogates of PDEs couldmore »enable use of larger and more detailed models than are possible with direct calculation and can make including such simulations in real-time or near-real time workflows practical. Creating a surrogate requires running the direct calculation tens of thousands of times to generate training data and then training the neural network, both of which are computationally expensive. Often the practical applications of such models require thousands to millions of replica simulations, for example for parameter identification and uncertainty quantification, each of which gains speed from surrogate use and rapidly recovers the up-front costs of surrogate generation. We use a Convolutional Neural Network to approximate the stationary solution to the diffusion equation in the case of two equal-diameter, circular, constant-value sources located at random positions in a two-dimensional square domain with absorbing boundary conditions. Such a configuration caricatures the chemical concentration field of a fast-diffusing species like oxygen in a tissue with two parallel blood vessels in a cross section perpendicular to the two blood vessels. To improve convergence during training, we apply a training approach that uses roll-back to reject stochastic changes to the network that increase the loss function. The trained neural network approximation is about 1000 times faster than the direct calculation for individual replicas. Because different applications will have different criteria for acceptable approximation accuracy, we discuss a variety of loss functions and accuracy estimators that can help select the best network for a particular application. We briefly discuss some of the issues we encountered with overfitting, mismapping of the field values and the geometrical conditions that lead to large absolute and relative errors in the approximate solution.« less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email