An official website of the United States government
Abstract Digital twins represent a key technology for precision health. Medical digital twins consist of computational models that represent the health state of individual patients over time, enabling optimal therapeutics and forecasting patient prognosis. Many health conditions involve the immune system, so it is crucial to include its key features when designing medical digital twins. The immune response is complex and varies across diseases and patients, and its modelling requires the collective expertise of the clinical, immunology, and computational modelling communities. This review outlines the initial progress on immune digital twins and the various initiatives to facilitate communication between interdisciplinary communities. We also outline the crucial aspects of an immune digital twin design and the prerequisites for its implementation in the clinic. We propose some initial use cases that could serve as “proof of concept” regarding the utility of immune digital technology, focusing on diseases with a very different immune response across spatial and temporal scales (minutes, days, months, years). Lastly, we discuss the use of digital twins in drug discovery and point out emerging challenges that the scientific community needs to collectively overcome to make immune digital twins a reality.
more »
« less
Neural digital twins: reconstructing complex medical environments for spatial planning in virtual reality
Abstract PurposeSpecialized robotic and surgical tools are increasing the complexity of operating rooms (ORs), requiring elaborate preparation especially when techniques or devices are to be used for the first time. Spatial planning can improve efficiency and identify procedural obstacles ahead of time, but real ORs offer little availability to optimize space utilization. Methods for creating reconstructions of physical setups, i.e., digital twins, are needed to enable immersive spatial planning of such complex environments in virtual reality. MethodsWe present a neural rendering-based method to create immersive digital twins of complex medical environments and devices from casual video capture that enables spatial planning of surgical scenarios. To evaluate our approach we recreate two operating rooms and ten objects through neural reconstruction, then conduct a user study with 21 graduate students carrying out planning tasks in the resulting virtual environment. We analyze task load, presence, perceived utility, plus exploration and interaction behavior compared to low visual complexity versions of the same environments. ResultsResults show significantly increased perceived utility and presence using the neural reconstruction-based environments, combined with higher perceived workload and exploratory behavior. There’s no significant difference in interactivity. ConclusionWe explore the feasibility of using modern reconstruction techniques to create digital twins of complex medical environments and objects. Without requiring expert knowledge or specialized hardware, users can create, explore and interact with objects in virtual environments. Results indicate benefits like high perceived utility while being technically approachable, which may indicate promise of this approach for spatial planning and beyond.
more »
« less
- Award ID(s):
- 2239077
- PAR ID:
- 10505190
- Publisher / Repository:
- Springer Science + Business Media
- Date Published:
- Journal Name:
- International Journal of Computer Assisted Radiology and Surgery
- Volume:
- 19
- Issue:
- 7
- ISSN:
- 1861-6429
- Format(s):
- Medium: X Size: p. 1301-1312
- Size(s):
- p. 1301-1312
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
IntroductionDigital twins of patients are virtual models that can create a digital patient replica to test clinical interventionsin silicowithout exposing real patients to risk. With the increasing availability of electronic health records and sensor-derived patient data, digital twins offer significant potential for applications in the healthcare sector. MethodsThis article presents a scalable full-stack architecture for a patient simulation application driven by graph-based models. This patient simulation application enables medical practitioners and trainees to simulate the trajectory of critically ill patients with sepsis. Directed acyclic graphs are utilized to model the complex underlying causal pathways that focus on the physiological interactions and medication effects relevant to the first 6 h of critical illness. To realize the sepsis patient simulation at scale, we propose an application architecture with three core components, a cross-platform frontend application that clinicians and trainees use to run the simulation, a simulation engine hosted in the cloud on a serverless function that performs all of the computations, and a graph database that hosts the graph model utilized by the simulation engine to determine the progression of each simulation. ResultsA short case study is presented to demonstrate the viability of the proposed simulation architecture. DiscussionThe proposed patient simulation application could help train future generations of healthcare professionals and could be used to facilitate clinicians’ bedside decision-making.more » « less
-
ImportanceAssessing nontechnical skills in operating rooms (ORs) is crucial for enhancing surgical performance and patient safety. However, automated and real-time evaluation of these skills remains challenging. ObjectiveTo explore the feasibility of using motion features extracted from surgical video recordings to automatically assess nontechnical skills during cardiac surgical procedures. Design, Setting, and ParticipantsThis cross-sectional study used video recordings of cardiac surgical procedures at a tertiary academic US hospital collected from January 2021 through May 2022. The OpenPose library was used to analyze videos to extract body pose estimations of team members and compute various team motion features. The Non-Technical Skills for Surgeons (NOTSS) assessment tool was employed for rating the OR team’s nontechnical skills by 3 expert raters. Main Outcomes and MeasuresNOTSS overall score, with motion features extracted from surgical videos as measures. ResultsA total of 30 complete cardiac surgery procedures were included: 26 (86.6%) were on-pump coronary artery bypass graft procedures and 4 (13.4%) were aortic valve replacement or repair procedures. All patients were male, and the mean (SD) age was 72 (6.3) years. All surgical teams were composed of 4 key roles (attending surgeon, attending anesthesiologist, primary perfusionist, and scrub nurse) with additional supporting roles. NOTSS scores correlated significantly with trajectory (r = 0.51,P = .005), acceleration (r = 0.48,P = .008), and entropy (r = −0.52,P = .004) of team displacement. Multiple linear regression, adjusted for patient factors, showed average team trajectory (adjustedR2 = 0.335; coefficient, 10.51 [95% CI, 8.81-12.21];P = .004) and team displacement entropy (adjustedR2 = 0.304; coefficient, −12.64 [95% CI, −20.54 to −4.74];P = .003) were associated with NOTSS scores. Conclusions and RelevanceThis study suggests a significant link between OR team movements and nontechnical skills ratings by NOTSS during cardiac surgical procedures, suggesting automated surgical video analysis could enhance nontechnical skills assessment. Further investigation across different hospitals and specialties is necessary to validate these findings.more » « less
-
This work, situated at Rensselaer's Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVELab), uses panoramic image datasets for spatial audio display. A system is developed for the room-centered immersive virtual reality facility to analyze panoramic images on a segment-by-segment basis, using pre-trained neural network models for semantic segmentation and object detection, thereby generating audio objects with respective spatial locations. These audio objects are then mapped with a series of synthetic and recorded audio datasets and populated within a spatial audio environment as virtual sound sources. The resulting audiovisual outcomes are then displayed using the facility's human-scale panoramic display, as well as the 128-channel loudspeaker array for wave field synthesis (WFS). Performance evaluation indicates effectiveness for real-time enhancements, with potentials for large-scale expansion and rapid deployment in dynamic immersive virtual environments.more » « less
-
Extreme environments, such as search and rescue missions, defusing bombs, or exploring extraterrestrial planets, are unsafe environments for humans to be in. Robots enable humans to explore and interact in these environments through remote presence and teleoperation and virtual reality provides a medium to create immersive and easy-to-use teleoperation interfaces. However, current virtual reality interfaces are still very limited in their capabilities. In this work, we aim to advance robot teleoperation virtual reality interfaces by developing an environment reconstruction methodology capable of recognizing objects in a robot’s environment and rendering high fidelity models inside a virtual reality headset. We compare our proposed environment reconstruction method against traditional point cloud streaming by having operators plan waypoint trajectories to accomplish a pick-and-place task. Overall, our results show that users find our environment reconstruction method more usable and less cognitive work compared to raw point cloud streaming.more » « less
