Successful surgical operations are characterized by preplanning routines to be executed during actual surgical operations. To achieve this, surgeons rely on the experience acquired from the use of cadavers, enabling technologies like virtual reality (VR) and clinical years of practice. However, cadavers, having no dynamism and realism as they lack blood, can exhibit limited tissue degradation and shrinkage, while current VR systems do not provide amplified haptic feedback. This can impact surgical training increasing the likelihood of medical errors. This work proposes a novel Mixed Reality Combination System (MRCS) that pairs Augmented Reality (AR) technology and an inertial measurement unit (IMU) sensor with 3D printed, collagen-based specimens that can enhance task performance like planning and execution. To achieve this, the MRCS charts out a path prior to a user task execution based on a visual, physical, and dynamic environment on the state of a target object by utilizing surgeon-created virtual imagery that, when projected onto a 3D printed biospecimen as AR, reacts visually to user input on its actual physical state. This allows a real-time user reaction of the MRCS by displaying new multi-sensory virtual states of an object prior to performing on the actual physical state of that same object enabling effective task planning. Tracked user actions using an integrated 9-Degree of Freedom IMU demonstrate task execution This demonstrates that a user, with limited knowledge of specific anatomy, can, under guidance, execute a preplanned task. In addition, to surgical planning, this system can be generally applied in areas such as construction, maintenance, and education.
- NSF-PAR ID:
- 10377863
- Date Published:
- Journal Name:
- Journal of Medical Robotics Research
- Volume:
- 06
- Issue:
- 03n04
- ISSN:
- 2424-905X
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
more » « less
ABSTRACT Introduction Remote military operations require rapid response times for effective relief and critical care. Yet, the military theater is under austere conditions, so communication links are unreliable and subject to physical and virtual attacks and degradation at unpredictable times. Immediate medical care at these austere locations requires semi-autonomous teleoperated systems, which enable the completion of medical procedures even under interrupted networks while isolating the medics from the dangers of the battlefield. However, to achieve autonomy for complex surgical and critical care procedures, robots require extensive programming or massive libraries of surgical skill demonstrations to learn effective policies using machine learning algorithms. Although such datasets are achievable for simple tasks, providing a large number of demonstrations for surgical maneuvers is not practical. This article presents a method for learning from demonstration, combining knowledge from demonstrations to eliminate reward shaping in reinforcement learning (RL). In addition to reducing the data required for training, the self-supervised nature of RL, in conjunction with expert knowledge-driven rewards, produces more generalizable policies tolerant to dynamic environment changes. A multimodal representation for interaction enables learning complex contact-rich surgical maneuvers. The effectiveness of the approach is shown using the cricothyroidotomy task, as it is a standard procedure seen in critical care to open the airway. In addition, we also provide a method for segmenting the teleoperator’s demonstration into subtasks and classifying the subtasks using sequence modeling.
Materials and Methods A database of demonstrations for the cricothyroidotomy task was collected, comprising six fundamental maneuvers referred to as surgemes. The dataset was collected by teleoperating a collaborative robotic platform—SuperBaxter, with modified surgical grippers. Then, two learning models are developed for processing the dataset—one for automatic segmentation of the task demonstrations into a sequence of surgemes and the second for classifying each segment into labeled surgemes. Finally, a multimodal off-policy RL with rewards learned from demonstrations was developed to learn the surgeme execution from these demonstrations.
Results The task segmentation model has an accuracy of 98.2%. The surgeme classification model using the proposed interaction features achieved a classification accuracy of 96.25% averaged across all surgemes compared to 87.08% without these features and 85.4% using a support vector machine classifier. Finally, the robot execution achieved a task success rate of 93.5% compared to baselines of behavioral cloning (78.3%) and a twin-delayed deep deterministic policy gradient with shaped rewards (82.6%).
Conclusions Results indicate that the proposed interaction features for the segmentation and classification of surgical tasks improve classification accuracy. The proposed method for learning surgemes from demonstrations exceeds popular methods for skill learning. The effectiveness of the proposed approach demonstrates the potential for future remote telemedicine on battlefields.
-
Spatial reasoning skills have been linked to success in STEM and are considered an important part of geoscience problem solving. Most agree that these are a group of skills rather than a single ability, though there is no agreement on the full list of constituent skills. Few studies have attempted to isolate specific spatial skills for deliberate training. We conducted an experiment to isolate and train the skill of recognizing horizontal (a crucial component in measuring the orientation of planes) using a dedicated Virtual Reality (VR) module. We recruited 21 undergraduate students from natural science and social science majors for the study, which consisted of a pretest, 15-minute training, and posttest. The pre- and posttests consisted of a short multiple choice vocabulary quiz, 5 hand-drawn and 5 multiple choice Water Level Task (WLT) questions, and the Vandenberg and Kuse Mental Rotation Task (MRT). Participants were sorted based on pre-test Water Level Task scores, only those with scores <80% were placed in an intervention group and randomly assigned to training, either in VR (experimental) or on paper (standard), of about 15 minutes. The high-scoring participants received no training (comparison). All three groups of participants completed a posttest after the training (if any). After removing three participants who did not return for the posttest session, we had 18 participants in total: 6 in VR, 7 in the comparison group, and 5 in the standard group. Repeated measures ANOVA of the pre to post hand-drawn WLT scores shows at least one group is different (p=.002) and Tukey’s Post-Hoc analysis indicates that the VR group improved significantly more that the high-scoring comparison group (Mean Difference = -1.857, p = .001) and the standard group (Mean Difference = -1.200, p = .049). While any significant result is encouraging, a major limitation of this study is the small sample size and unequal variances on both the pretest (Levene’s HOV test, F = 7.50, p = .006) and posttest (F = 13.53, p < .001), despite random assignment. More trials are needed to demonstrate reproducibility. While more tests are needed, this preliminary study shows the potential benefit of VR in training spatial reasoning skills.more » « less
-
null (Ed.)Advancements in Artificial Intelligence (AI), Information Technology, Augmented Reality (AR) and Virtual Reality (VR), and Robotic Automation is transforming jobs in the Architecture, Engineering and Construction (AEC) industries. However, it is also expected that these technologies will lead to job displacement, alter skill profiles for existing jobs, and change how people work. Therefore, preparing the workforce for an economy defined by these technologies is imperative. This ongoing research focuses on developing an immersive learning training curriculum to prepare the future workforce of the building industry. In this paper we are demonstrating a prototype of a mobile AR application to deliver lessons for training in robotic automation for construction industry workers. The application allows a user to interact with a virtual robot manipulator to learn its basic operations. The goal is to evaluate the effectiveness of the AR application by gauging participants' performance using pre and post surveys.more » « less
-
null (Ed.)An important problem in designing human-robot systems is the integration of human intent and performance in the robotic control loop, especially in complex tasks. Bimanual coordination is a complex human behavior that is critical in many fine motor tasks, including robot-assisted surgery. To fully leverage the capabilities of the robot as an intelligent and assistive agent, online recognition of bimanual coordination could be important. Robotic assistance for a suturing task, for example, will be fundamentally different during phases when the suture is wrapped around the instrument (i.e., making a c- loop), than when the ends of the suture are pulled apart. In this study, we develop an online recognition method of bimanual coordination modes (i.e., the directions and symmetries of right and left hand movements) using geometric descriptors of hand motion. We (1) develop this framework based on ideal trajectories obtained during virtual 2D bimanual path following tasks performed by human subjects operating Geomagic Touch haptic devices, (2) test the offline recognition accuracy of bi- manual direction and symmetry from human subject movement trials, and (3) evalaute how the framework can be used to characterize 3D trajectories of the da Vinci Surgical System’s surgeon-side manipulators during bimanual surgical training tasks. In the human subject trials, our geometric bimanual movement classification accuracy was 92.3% for movement direction (i.e., hands moving together, parallel, or away) and 86.0% for symmetry (e.g., mirror or point symmetry). We also show that this approach can be used for online classification of different bimanual coordination modes during needle transfer, making a C loop, and suture pulling gestures on the da Vinci system, with results matching the expected modes. Finally, we discuss how these online estimates are sensitive to task environment factors and surgeon expertise, and thus inspire future work that could leverage adaptive control strategies to enhance user skill during robot-assisted surgery.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1142/S2424905X21500069
