An official website of the United States government
Digitalization shapes the ways of learning, working, and entertainment. The Internet, which enables us to connect and socialize is evolving to become the metaverse, a post-reality universe, enabling virtual life parallel to reality. In addition to gaming and entertainment, industry and academia have noticed the metaverse’s benefits and possibilities. For industry, the metaverse is the enabler of the future digital workplace, and for academia, digital learning spaces enable realistic virtual training environments. A connection bridging the virtual world with physical production systems is required to enable digital workplaces and digital learning spaces. In this publication, extended reality–digital twin to real use cases are presented. The presented use cases utilize extended reality as high-level user interfaces and digital twins to create a bridge between virtual environments and robotic systems in industry, academia, and underwater exploration.
more »
« less
Digital Twins Utilizing XR-Technology as Robotic Training Tools
Digital technology has evolved towards a new way of processing information: web searches, social platforms, internet forums, and video games have substituted reading books and writing essays. Trainers and educators currently face the challenge of providing natural training and learning environments for digital-natives. In addition to physical spaces, effective training and education require virtual spaces. Digital twins enable trainees to interact with real hardware in virtual training environments. Interactive real-world elements are essential in the training of robot operators. A natural environment for the trainee supports an interesting learning experience while including enough professional substances. We present examples of how virtual environments utilizing digital twins and extended reality can be applied to enable natural and effective robotic training scenarios. Scenarios are validated using cross-platform client devices for extended reality implementations and safety training applications.
more »
« less
- PAR ID:
- 10471227
- Publisher / Repository:
- Machines
- Date Published:
- Journal Name:
- Machines
- Volume:
- 11
- Issue:
- 1
- ISSN:
- 2075-1702
- Page Range / eLocation ID:
- 13
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract PurposeSpecialized robotic and surgical tools are increasing the complexity of operating rooms (ORs), requiring elaborate preparation especially when techniques or devices are to be used for the first time. Spatial planning can improve efficiency and identify procedural obstacles ahead of time, but real ORs offer little availability to optimize space utilization. Methods for creating reconstructions of physical setups, i.e., digital twins, are needed to enable immersive spatial planning of such complex environments in virtual reality. MethodsWe present a neural rendering-based method to create immersive digital twins of complex medical environments and devices from casual video capture that enables spatial planning of surgical scenarios. To evaluate our approach we recreate two operating rooms and ten objects through neural reconstruction, then conduct a user study with 21 graduate students carrying out planning tasks in the resulting virtual environment. We analyze task load, presence, perceived utility, plus exploration and interaction behavior compared to low visual complexity versions of the same environments. ResultsResults show significantly increased perceived utility and presence using the neural reconstruction-based environments, combined with higher perceived workload and exploratory behavior. There’s no significant difference in interactivity. ConclusionWe explore the feasibility of using modern reconstruction techniques to create digital twins of complex medical environments and objects. Without requiring expert knowledge or specialized hardware, users can create, explore and interact with objects in virtual environments. Results indicate benefits like high perceived utility while being technically approachable, which may indicate promise of this approach for spatial planning and beyond.more » « less
-
Abstract Fluid power systems can be expensive and difficult to access, making it challenging to provide hands-on training. This work discusses the incorporation of Mixed Reality (MR) technology in Fluid Power applications for providing a virtual training environment that simulates the behavior of fluid power systems, allowing users to receive immediate feedback on the system’s performance. Mixed reality is a digitized-based technology that integrates a virtual environment with our real world by utilizing real-world sensor data and computer models. This technology allows running simulations that examine the complexity of highly-coupled systems, producing new digital environments where physical and digital elements can interact in real-time. With all these features, MR technology can be a practical training tool for running virtual simulations that mimic real-life industry settings. It can extend the user with a virtual training environment, thus preparing the next generation of fluid power engineers and specialists. Throughout this work, we present the development and capabilities of a digitized virtual copy of a hydraulic excavator’s arm in an MR environment as a proof of concept. The MR arm module is developed and deployed using Microsoft’s Mixed Reality Tool Kit (MRTK) for Unity through HoloLens 2 MR headset. The MR development involves generating virtual copies of the mechanical and hydraulic subsystems, conducting the virtual assembly, and creating a user interface in the MR environment to visualize and interact with the model. The developed MR module enables visualizing the excavator’s internal structure, conducting the virtual assembly, and running virtual simulations, all of which assist in training future fluid power operators. It is an effective training tool that helps train junior engineers/technicians, cutting down on cost and time.more » « less
-
This paper proposes a conceptual architecture of digital twin with human-in-the-loop-based smart manufacturing (DH-SM). Our proposed architecture integrates cyber-physical systems with human spaces, where artificial intelligence and human cognition are employed jointly to make informed decisions. This will enable real-time, collaborative decision-making between humans, software, and machines. For example, when evaluating a new product design, information about the product’s physical features, manufacturing requirements, and customer demands must be processed concurrently. Moreover, the DH-SM architecture enables the creation of an immersive environment that allows customers to be effectively involved in the manufacturing process. The DH-SM architecture is well fitted to those relatively new manufacturing processes, such as metal additive manufacturing, since they can benefit from using digital twins, data analytics, and artificial intelligence for monitoring and controlling those processes to support non-contact manufacturing. The proposed DH-SM will enable manufacturers to leverage the existing cyber-physical system and extended reality technologies to generate immersive experiences for end users, operators, managers, and stakeholders. A use case of wire + arc additive manufacturing is discussed to demonstrate the applicability of the proposed architecture. Relevant development and implementation challenges are also discussed.more » « less
-
Virtual reality, a well-established educational technology, offers unique affordances such as immersion, interactivity, visualization, and co-presence, with significant potential to enhance learning experiences and outcomes. Computational thinking, a vital skill in science, technology, engineering, and mathematics, is essential for effective human-robot collaboration, enabling efficient problem-solving and decision-making in future work environments. Virtual reality provides a cost-effective, safe alternative to physical interaction with robots, reducing equipment risks and addressing the limitations of physical training. This study examines how virtual reality's affordances support computational thinking development, presenting a forward-looking training scenario and an assessment rubric for evaluation. The proposed framework and design strategies offer technological and pedagogical guidance for creating virtual reality environments that foster computational thinking in human-robot collaboration contexts.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/machines11010013
