An official website of the United States government
Digitalization shapes the ways of learning, working, and entertainment. The Internet, which enables us to connect and socialize is evolving to become the metaverse, a post-reality universe, enabling virtual life parallel to reality. In addition to gaming and entertainment, industry and academia have noticed the metaverse’s benefits and possibilities. For industry, the metaverse is the enabler of the future digital workplace, and for academia, digital learning spaces enable realistic virtual training environments. A connection bridging the virtual world with physical production systems is required to enable digital workplaces and digital learning spaces. In this publication, extended reality–digital twin to real use cases are presented. The presented use cases utilize extended reality as high-level user interfaces and digital twins to create a bridge between virtual environments and robotic systems in industry, academia, and underwater exploration.
more »
« less
Digital Twins Utilizing XR-Technology as Robotic Training Tools
Digital technology has evolved towards a new way of processing information: web searches, social platforms, internet forums, and video games have substituted reading books and writing essays. Trainers and educators currently face the challenge of providing natural training and learning environments for digital-natives. In addition to physical spaces, effective training and education require virtual spaces. Digital twins enable trainees to interact with real hardware in virtual training environments. Interactive real-world elements are essential in the training of robot operators. A natural environment for the trainee supports an interesting learning experience while including enough professional substances. We present examples of how virtual environments utilizing digital twins and extended reality can be applied to enable natural and effective robotic training scenarios. Scenarios are validated using cross-platform client devices for extended reality implementations and safety training applications.
more »
« less
- PAR ID:
- 10471227
- Publisher / Repository:
- Machines
- Date Published:
- Journal Name:
- Machines
- Volume:
- 11
- Issue:
- 1
- ISSN:
- 2075-1702
- Page Range / eLocation ID:
- 13
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract PurposeSpecialized robotic and surgical tools are increasing the complexity of operating rooms (ORs), requiring elaborate preparation especially when techniques or devices are to be used for the first time. Spatial planning can improve efficiency and identify procedural obstacles ahead of time, but real ORs offer little availability to optimize space utilization. Methods for creating reconstructions of physical setups, i.e., digital twins, are needed to enable immersive spatial planning of such complex environments in virtual reality. MethodsWe present a neural rendering-based method to create immersive digital twins of complex medical environments and devices from casual video capture that enables spatial planning of surgical scenarios. To evaluate our approach we recreate two operating rooms and ten objects through neural reconstruction, then conduct a user study with 21 graduate students carrying out planning tasks in the resulting virtual environment. We analyze task load, presence, perceived utility, plus exploration and interaction behavior compared to low visual complexity versions of the same environments. ResultsResults show significantly increased perceived utility and presence using the neural reconstruction-based environments, combined with higher perceived workload and exploratory behavior. There’s no significant difference in interactivity. ConclusionWe explore the feasibility of using modern reconstruction techniques to create digital twins of complex medical environments and objects. Without requiring expert knowledge or specialized hardware, users can create, explore and interact with objects in virtual environments. Results indicate benefits like high perceived utility while being technically approachable, which may indicate promise of this approach for spatial planning and beyond.more » « less
-
Abstract Fluid power systems can be expensive and difficult to access, making it challenging to provide hands-on training. This work discusses the incorporation of Mixed Reality (MR) technology in Fluid Power applications for providing a virtual training environment that simulates the behavior of fluid power systems, allowing users to receive immediate feedback on the system’s performance. Mixed reality is a digitized-based technology that integrates a virtual environment with our real world by utilizing real-world sensor data and computer models. This technology allows running simulations that examine the complexity of highly-coupled systems, producing new digital environments where physical and digital elements can interact in real-time. With all these features, MR technology can be a practical training tool for running virtual simulations that mimic real-life industry settings. It can extend the user with a virtual training environment, thus preparing the next generation of fluid power engineers and specialists. Throughout this work, we present the development and capabilities of a digitized virtual copy of a hydraulic excavator’s arm in an MR environment as a proof of concept. The MR arm module is developed and deployed using Microsoft’s Mixed Reality Tool Kit (MRTK) for Unity through HoloLens 2 MR headset. The MR development involves generating virtual copies of the mechanical and hydraulic subsystems, conducting the virtual assembly, and creating a user interface in the MR environment to visualize and interact with the model. The developed MR module enables visualizing the excavator’s internal structure, conducting the virtual assembly, and running virtual simulations, all of which assist in training future fluid power operators. It is an effective training tool that helps train junior engineers/technicians, cutting down on cost and time.more » « less
-
This paper proposes a conceptual architecture of digital twin with human-in-the-loop-based smart manufacturing (DH-SM). Our proposed architecture integrates cyber-physical systems with human spaces, where artificial intelligence and human cognition are employed jointly to make informed decisions. This will enable real-time, collaborative decision-making between humans, software, and machines. For example, when evaluating a new product design, information about the product’s physical features, manufacturing requirements, and customer demands must be processed concurrently. Moreover, the DH-SM architecture enables the creation of an immersive environment that allows customers to be effectively involved in the manufacturing process. The DH-SM architecture is well fitted to those relatively new manufacturing processes, such as metal additive manufacturing, since they can benefit from using digital twins, data analytics, and artificial intelligence for monitoring and controlling those processes to support non-contact manufacturing. The proposed DH-SM will enable manufacturers to leverage the existing cyber-physical system and extended reality technologies to generate immersive experiences for end users, operators, managers, and stakeholders. A use case of wire + arc additive manufacturing is discussed to demonstrate the applicability of the proposed architecture. Relevant development and implementation challenges are also discussed.more » « less
-
As next-generation scientific instruments and simulations generate ever larger datasets, there is a growing need for high-performance computing (HPC) techniques that can provide timely and accurate analysis. With artificial intelligence (AI) and hardware breakthroughs at the forefront in recent years, interest in using this technology to perform decision-making tasks with continuously evolving real-world datasets has increased. Digital twinning is one method in which virtual replicas of real-world objects are modeled, updated, and interpreted to perform such tasks. However, the interface between AI techniques, digital twins (DT), and HPC technologies has yet to be thoroughly investigated despite the natural synergies between them. This paper explores the interface between digital twins, scientific computing, and machine learning (ML) by presenting a consistent definition for the digital twin, performing a systematic analysis of the literature to build a taxonomy of ML-enhanced digital twins, and discussing case studies from various scientific domains. We identify several promising future research directions, including hybrid assimilation frameworks and physics-informed techniques for improved accuracy. Through this comprehensive analysis, we aim to highlight both the current state-of-the-art and critical paths forward in this rapidly evolving field.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/machines11010013
