The construction industry is currently facing significant challenges, such as shortage of skilled workers and aging workforce [1], compounded by the perception of unsafe working conditions that deter young individuals from entering the field [2]. This situation is critical, as the U.S. Associated Builders And Contractors (ABC) estimated a pressing need for over 650,000 new workers in 2022, underscoring the urgent state of the labor market in this sector [3,4]. In response, the industry is increasingly turning to automation and integration of digital technologies, such as robotics, to address labor shortages, enhance safety, and improve efficiency [5,6]. This technological shift is transforming traditional roles, introducing innovations such as self-driving vehicles, drones, and robotics, which mark a significant evolution in construction practices [7].
Simultaneously, the manufacturing industry is navigating its own set of unique challenges by embracing automation and robotics [8]. This adaptation is crucial for maintaining competitiveness and safety, requiring workers across both sectors to learn to collaborate with robotic technologies, thereby reshaping the labor and technology landscape [9]. The integration of Human-Robot Collaboration (HRC) with Virtual Reality (VR) has seen widespread applications across industries From healthcare, where they facilitate surgical training and rehabilitation, to manufacturing, where they improve precision tasks and safety, the versatility and effectiveness of these technologies are clear [10,11].
Against the backdrop of technological innovation, HRC emerges as a significant development [12,13]. It involves direct interaction between human workers and robots, aiming to combine human flexibility and intelligence with robotic precision and endurance to enhance productivity and safety. HRC is revolutionizing the field, moving from simple task automation to intelligent systems that enable synergistic human-robot partnerships, powered by advanced AI, sensors, and machine-learning capabilities [14]. Recent advances in deep-distributed learning and robotic wireless energy transfer further underscore the potential of these intelligent systems. For instance, Guo et al. explored deep-distributed learning-based Point-of-Interest (POI) recommendations under mobile-edge networks, showcasing how advanced learning frameworks can significantly improve system performance and efficiency [15]. Similarly, Wang et al. investigated robotic wireless energy transfer in dynamic environments, highlighting innovative approaches to maintaining energy efficiency in robotic systems [16]. These advances illustrate the diverse technological innovations contributing to the development of more efficient and effective HRC systems. HRC encompasses various levels, each defined by the degree of interaction and integration between humans and robots. According to Mukherjee et al. and De Pace et al., these levels range from fully programmed robots to fully autonomous systems [17,18]. At the fully programmed (L0) level, robots carry out predefined repetitive tasks within safety-fenced areas, with no human interaction. Co-existence (L1) involves robots and humans operating in close proximity but not sharing the same workspace, using safety measures like laser fencing. At the assistance (L2) level, robots assist human workers by reducing physical effort, such as in the use of wearable robotic devices. Cooperation (L3) occurs when humans and robots work in a shared space, performing sequential tasks towards a common goal. In collaboration (L4), humans and robots work simultaneously on shared tasks, requiring high levels of interaction and communication. Finally, at the fully autonomous (L5) level, robots operate independently, performing a wide variety of tasks without human intervention. Understanding these levels is crucial for designing effective HRC systems. This review explores how VR technology facilitates these varying levels of collaboration in manufacturing and construction industries, providing insights into recent advances and practical implementations.
In parallel to these HRC developments, VR training has emerged as a pivotal tool, offering numerous advantages over conventional training methods and other interactive technologies such as Augmented Reality (AR), Extended Reality (XR), and Mixed Reality (MR). While AR and MR augment the real world and XR encompasses a spectrum of combined real and virtual environments, VR is unique in its ability to create a fully immersive simulation environment. This total immersion is crucial for scenarios where understanding spatial awareness, complex machinery interaction, and precise execution of tasks are essential. VR provides a realistic environment for workers to practice their skills without the risk of injury or damage to equipment [19]. This realistic environment is particularly invaluable in industries where physical safety is a constant concern. Additionally, VR allows for skill development and task training, where workers can repeatedly practice and master complex interactions with robots, avoiding the use of costly and limited-supply machinery. Furthermore, VR significantly enhances user experience by providing more engaging and intuitive ways for workers to interact with robotic systems. This improves both the speed and quality of the learning process. Through the integration of HRC and VR, an immersive environment is established that replicates real-world scenarios [20]. Such experiences are crucial for preparing workers to navigate the complexities of real-life collaboration with robots, including understanding their movements, behaviors, and safety protocols [21]. Additionally, VR training can be tailored to individual learning preferences, enabling a personalized approach to skill development and improvement [22,23].
Although considerable research has been done in the field of HRC using immersive technologies, such as the work by Walker et al., there are still gaps in literature. Walker et al. present a taxonomy for Virtual, Augmented, and Mixed reality (VAM) interfaces for Human-Robot Interaction (HRI), emphasizing the need for a shared terminology and framework to characterize these interfaces [24]. Their work categorizes virtual design elements and discusses their development and applications over the past 30 years. This paper addresses these gaps by focusing specifically on the application of VR in HRC within the construction and manufacturing industries. Key research questions related to application domains, types of robots, VR setups, and software solutions in these sectors are addressed, providing a more targeted review of these areas. By concentrating on the construction and manufacturing industries, this study offers insights into how VR and HRC technologies are being used to address real-world challenges in these sectors. This approach highlights practical implementations and outcomes of integrating HRC with VR, offering a fresh perspective on the current advances in the field. This review provides a comprehensive analysis of recent studies on HRC integrated with VR. The primary focus is on addressing three research questions (RQs) to deepen the understanding of the role of VR and HRC in addressing industry challenges:
•RQ1: What are the key application domains in the construction and manufacturing industries where VR and HRC have been integrated?
•RQ2: Which types of robots are predominantly used in HRC studies within VR environments in the construction and manufacturing industries?
•RQ3: What are the most utilized VR setups (e.g., Desktop VR, CAVE VR, HMDs) and software solutions in VR environments for HRC within these sectors?
The review considers the use of avatars, whether the robots are real or virtual, and levels of collaboration to provide a well-rounded understanding of the current state of HRC-VR integration. This comprehensive literature review provides valuable insights for educators, industry professionals, and policymakers focused on the construction and manufacturing industries. The study elucidates the roles and types of robots prevalent in VR-enhanced HRC settings, highlighting how these technologies meet current industry needs and the challenges they face. This review not only showcases the applications and benefits of VR-enhanced HRC but also pinpoints limitations and research gaps that can guide future studies in enhancing HRC through VR technologies.
Answering these questions through a comprehensive literature review provides insights for educators, industry professionals, and policymakers in the construction and manufacturing industries. It also highlights the current limitations and research gaps in using VR for HRC. The remainder of the paper proceeds as follows. The methods section outlines the methodology of the study, including a detailed description of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) systematic review analysis.
The results section presents trends in the annual publication on HRC-VR, research frequency by industry type, avatar usage in HRC-VR studies, and a comparison between virtual and real robots in HRC-VR studies.
The discussion section addresses the RQs posed earlier. The future research section highlights challenges and limitations of current HRC-VR research. Finally, the conclusion summarizes the main findings and limitations of the study.
In this study, a systematic literature review was conducted using a detailed and methodical approach to select the most relevant papers for the research questions. The approach is based on the PRISMA methodology, renowned for its comprehensive keyword-based search strategy, facilitating effective navigation through scientific databases [25].
The search strategy was carried out using combinations of keywords together with Boolean operators in Web of Science: (("human-robot collaboration" OR "collaborative robotics" OR "human-robot interaction" OR "robot-assisted construction" OR "robotics in construction" OR "robotic automation" OR "robotic systems" OR "robotic applications" OR "robotic technology" OR "robotic solutions") AND ("virtual reality" OR "VR" OR "immersive virtual reality") AND ("construction" OR "building" OR "civil engineering" OR "architecture" OR "manufacturing" OR "industrial robotics" OR "construction industry" OR "building design" OR "civil infrastructure" OR "smart manufacturing")).
To complement the search in the Web of Science, Google Scholar was also employed with a tailored query specifically focused on the construction aspect. The Google Scholar query was formulated as follows.
"human-robot collaboration" AND "virtual reality" AND "Construction" -"Augmented" -"Digital Twin" -"BIM" -"mixed reality" -"AR" -"MR". It was strategically designed to delve deeper into the nuanced applications of VR in construction-related HRC scenarios. This more focused approach in Google Scholar was intended to complement the broader search in Web of Science, enabling us to capture detailed insights and specific case studies within the construction sector that are highly relevant to the research questions. The dual-strategy approach, which used both Web of Science and Google Scholar, was implemented to ensure comprehensive literature coverage. This approach was necessary as some technical papers meeting the criteria were exclusively available in Google Scholar, while others were only accessible through Web of Science. By employing this dual strategy, this study thoroughly explored the integration of HRC with VR both in the wider context of manufacturing and more specifically within construction. The search was carried out on 28 December 2023, and inclusion and exclusion criteria were applied, as listed in Table 1. These criteria were used to select the articles suitable for the study, considering the keywords, abstracts, and titles of papers published over the years. The selection criteria of the literature review targeted English language articles, focusing on HRC using VR technology, related to specific research questions. To ensure consistency and clarity in analysis, articles written in other languages and those lacking complete data were excluded. The review precedes peerreviewed journal publications and conferences, valued for their peer-reviewed rigor, over book chapters and brief articles. Employing the PRISMA approach ensured a detailed and transparent analysis of the studies selected. To address conflicting conclusions, the quality and context of each study were thoroughly evaluated, considering differences in study design, experimental setup, and analytical methods. This critical appraisal allowed for a comprehensive understanding and documentation of any potential discrepancies. By following PRISMA guidelines, the review accurately reflects the most reliable conclusions based on the overall evidence.
In this reviewing process, the PRISMA methodology was executed, using a combination of keywords and Boolean operators. Initially, 383 research papers were identified through Web of Science and Google Scholar.
After removing duplicates, 380 articles remained. The exclusion criteria then filtered out 113 articles for reasons such as language differences, missing data, and document type, leaving 267 articles for full-text assessment. Of these, only 53 met the criteria for inclusion in the literature review, as the rest did not respond to the research questions. The result obtained from the PRISMA methodology is illustrated in Figure 1.
Applying the PRISMA methodology to the analysis showed that a significant majority (approximately 74%) of the studies were published journal articles. A significant trend in the distribution of research across the construction and manufacturing industries was also identified. The results indicate that the manufacturing sector received the most substantial focus. The term 'General' in the context of the analysis refers to research that is neither specific to the construction industry nor the manufacturing industry but rather applies broadly across multiple sectors. In contrast, the construction industry appears to have attracted less attention from researchers, as evidenced by the fewer studies undertaken in this area. This disparity suggests a potential gap in literature and highlights an opportunity for future investigations to balance the research efforts more evenly, especially to deepen the understanding of technology applications in the construction industry. 0 2 4 6 8 10 12 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 Frequency Year Published Annual publication trends in HRC-VR for construction and manufacturing industries 0 5 10 15 20 25 30 35 40 Manufacturing Construction General Frequency Type of industry
The presence of avatars in VR settings was evaluated using the PRISMA framework. The frequency of avatar use within HRC-VR studies is depicted in Figure 4. Additionally, it was identified whether the research used virtual robots, actual robots, or a blend of both, with these distinctions presented in Figure 5. The review identified the levels of collaboration between humans and robots in VR settings, as previously outlined in the introduction. The distribution of levels of collaboration in HRC-VR studies is shown in Figure 6. Each article was reviewed and classified based on the degree of interaction and integration described in the study. As Figure 6 shows, 28% of the studies focused on the assistance (L2) level, 23% on the cooperation (L3) level, and 49% on the collaboration (L4) level. Notably, no studies were found at the fully programmed (L0), co-existence (L1), or fully autonomous (L5) levels, suggesting that research in HRC-VR is more focused on interactive and collaborative environments rather than fully independent or basic operational The wide range of applications of HRC and VR interactions, as outlined in these articles, is systematically summarized. The insights derived from all 53 articles are thoroughly detailed in Table 2.
In this section, a critical review and analysis of a carefully selected collection of 53 research papers is conducted, offering a thorough response to the research questions posed.
In response to RQ1-which asks about the key application domains and use cases in the construction and manufacturing industries where VR and HRC are integrated-the analysis of selected papers has led to the identification of six primary categories: safety, task efficiency and design, human-interaction improvement, training and education, collaborative and automated technologies, and innovative applications and prototyping. Each category reflects a different dimension of integration, showing the diverse potential of VR and HRC in improving industry practices:
In the domain of HRC integrated with VR technology, the impact on safety and efficiency is a paramount concern, addressed through several innovative approaches. It has been demonstrated in the aerospace industry that even a modest reduction in the speed of robots (specifically by 15%) can significantly enhance safety without detrimentally affecting operational efficiency [21]. This finding underscores the potential of VR to harmonize safety with productivity. VR training has been shown to substantially improve user experience and behavior within shared workspace collaborations, offering a compelling case for the capability of VR to augment safety and efficiency in HRC scenarios [68]. These advances illustrate the foundational role of VR in elevating safety protocols alongside maintaining high productivity standards.
Building upon these foundational insights, further advances in the integration of VR with actual robotic systems have optimized interactions within HRC frameworks [69]. This integration merges virtual simulations with real-world robotic operations, enhancing the safety and efficacy of these interactions. Such advances not only optimize the practical aspects of HRC but also introduce a novel approach to safety, blending the predictive capabilities of VR with the tangible dynamics of real-world operations. Regulatory guidelines are necessary in the integration of robots in construction to ensure the safe operation of mobile robots within such dynamic environments [77]. This underscores the essential role of structured frameworks in maintaining safety, emphasizing the intricate balance between innovation and regulatory compliance.
Parallel to technological and regulatory advances, predictive safety measures have garnered attention for their potential to proactively address safety concerns. Studies focusing on real-time human hand motion prediction and the assessment of drone impacts on worker safety in construction sectors [33,42] illustrated the critical role of advanced technologies in enhancing safety and monitoring within high-risk sectors. These predictive approaches signify a shift towards a more anticipatory stance on safety, leveraging technology to foresee and mitigate potential hazards before they manifest into incidents, thereby strengthening the safety infrastructure within HRC-VR applications.
Lastly, the effectiveness of safety training and monitoring in HRC-VR applications has been significantly
bolstered by technological integration. The use of speed and separation monitoring with lightweight industrial robots, as evidenced in automotive brake disc assembly showcases the practical benefits of such technologies in enhancing safety while minimizing operator fatigue [71]. Furthermore, the strong preference for VR-based safety training in HRC underscores its transformative potential for safety training practices [78]. The focus on proactive and adaptive safety techniques in VR environments further emphasizes the critical role of forward-thinking safety strategies [67]. These cumulative efforts in advancing safety measures through VR not only enhance the operational efficiency of HRC but also ensure a safer working environment, marking a significant leap forward in the application domain of HRC-VR.
In the realm of task efficiency and design in HRC, significant advances have been catalyzed by the integration of VR. The design of collaborative assembly and warehousing workstations has been revolutionized through the application of VR, which has facilitated early prototyping and scenario simulation. This application has effectively reduced design iterations and led to more efficient task execution, underscoring the value of VR in enhancing design processes [46]. Subsequently, insights into ergonomic indices and operator workload have been provided by real-time ergonomic assessments in automotive HRC using VR. These assessments have highlighted the potential of VR to significantly impact ergonomic design processes to enhance operator comfort and productivity [35]. The optimization of robot end-effector positioning through Particle Swarm
Optimization in VR has been demonstrated, marking a substantial improvement in ergonomic outcomes. This indicates the potential of combining optimization algorithms with VR for task efficiency and ergonomic benefits in HRC [27].
Efficiency in HRC assembly has also been explored through the integration of eye-hand coordination and a finite-state machine within a virtual environment. This exploration has demonstrated a novel approach to enhancing task efficiency and collaborative performance in HRC, showcasing the innovative application of VR in advancing assembly processes [49]. Furthermore, the design and simulation of cooperative robotic cells focused on integrating safety and ergonomic considerations into design processes have been investigated.
This focus aims to foster safer and more productive work environments, illustrating the importance of incorporating such considerations from the design phase [41].
Deeper insights into understanding human intent and movement in HRC settings have been provided by the prediction of human arm movement direction based on eye tracking and comparison of different neural network models for gaze-based intention prediction. These insights potentially improve interaction dynamics, emphasizing the utility of VR in enhancing the intuitive understanding of human actions [29,31]. VR-based task modeling and interaction design have also proven valuable for advancing human-robot collaboration in industrial contexts. Immersive VR environments support early evaluation of collaborative tasks, ergonomic planning, and safety assessment, ultimately leading to more efficient and intuitive task execution [47,56,76].
A VR-based architecture ensuring safety and efficiency in HRC has focused on the importance of prototyping and evaluating communication methods within VR environments, demonstrating the role of VR in facilitating safer and more efficient collaborative workspaces [64]. Together, these studies not only extend the knowledge in task efficiency and design in HRC but also demonstrate the critical role of VR in enhancing and optimizing HRC.
A wide array of research has been undertaken to improve the dynamics of interaction in HRC. It has been discovered that leveraging human gaze as an indicator of attention significantly enhances remote manufacturing collaboration in dual reality setups [75]. The utility of VR in training drone operators has been demonstrated, offering valuable insights into performance prediction and stress management. This underscores the critical role of VR in readying operators for real-world applications [53].
The variability in human responses to changes in robotic speed and behavior has been uncovered, suggesting the necessity for personalized control strategies to better accommodate individual differences, thereby potentially improving interaction efficiency and comfort [54]. The impact of robot speed and predictability on human operators has been scrutinized, with findings indicating that unpredictability and faster movements can exacerbate stress and hinder task performance. These insights highlight the necessity for designing robot behaviors that are both balanced and predictable, ensuring conducive work environments [72].
Integrating VR with neuroimaging presents a novel approach to tailoring HRI, optimizing interfaces based on neurobehavioral data. This innovative method hints at the potential of such technologies to forge more intuitive and effective HRC systems, personalizing user experience based on individual cognitive and emotional responses [55]. The enhancement of collaboration and comfort in industrial HRC settings through VR further illustrates the versatility of technology. Studies focusing on task sharing and efficiency emphasize the capability of VR to improve the design and assessment of HRC tasks, facilitating a deeper understanding of the collaborative process [66,73].
The exploration of HRC applications in diverse domains, such as space exploration, underscores the broad potential of this technology. Using rescue robots in virtual environments for collaborative space exploration and mapping showcases the adaptability of HRC systems to various operational contexts [58]. The investigation into the role of robot apologies as a trust-recovery strategy in industrial HRC reveals the critical importance of nuanced communication strategies. This study emphasizes the need for mechanisms that can effectively maintain trust and collaboration, particularly in scenarios involving unexpected robot movements [50].
The development of a wearable teleoperation controller equipped with haptic feedback for use in VR environments has been a notable advancement. This technology focuses on the manipulation of virtual objects,
showcasing the potential of tactile feedback to significantly enrich user experience within virtual environments, thereby enhancing the realism and intuitiveness of interactions [30]. Further enhancing communication within HRC, a virtual sandbox has been used to study the effects of augmented communication on manufacturing tasks. This study emphasizes the role of AR and VR in improving communication and task efficiency, illustrating how augmented interfaces can streamline and optimize the collaborative process between humans and robots [51]. Additionally, the use of eye tracking for usability evaluations of an Emotional Display Agent in VR has highlighted the significance of intuitive user interface design. Such designs facilitate effective navigation and object selection, underlining the critical importance of user-centered design principles in HRC to ensure seamless and efficient HRI [65].
The assessment of neurobehavioral responses to force feedback in robotic teleoperation, particularly focusing on remote operation tasks, underscores the importance of sensory feedback. This feedback enhances control and adds a layer of realism to HRC, indicating that sensory inputs are crucial for creating more immersive and responsive interaction environments [52]. Investigations into the psychophysiological impacts of working near drones on construction sites introduce a new perspective on drone-human interactions. These studies highlight the necessity of considering human factors in drone deployment strategies, suggesting that understanding the human response is essential for the successful integration of drones into collaborative workspaces [33].
Lastly, the exploration of VR to make collaborative robot (cobot) signals more interpretable and predictable has been instrumental in enhancing trust and success in HRC. This investigation emphasizes the need for clear and effective communication mechanisms, indicating that predictability and understandability of robot actions are key to fostering trust and collaboration in shared work environments [48]. Collectively, these studies underscore the ongoing need for innovative approaches to improve HRI. By advocating for the development of technologies and design principles that create more flexible, efficient, and user-friendly collaborative environments, these research efforts highlight the strategic integration of technology and design as pivotal to advancing the state of HRC.
Advances in training and education within HRC have been markedly propelled by VR and other immersive technologies, showcasing their capacity to revolutionize learning and operational efficiency across diverse industrial settings. This evolution towards more interactive, realistic, and engaging training methodologies caters to a broad spectrum of applications, ranging from industrial robotics to unmanned aerial vehicle (UAV) operation.
The role of VR in enhancing industrial robotics training by simulating real-world scenarios has been emphasized as a critical means to ensure safety and efficiency. This demonstrates the significant value of immersive technologies in accurately replicating complex operational environments, facilitating deeper understanding and proficiency in handling industrial robotics [59]. Similarly, the application of VR in UAV training has been recognized for its notable improvements in training efficiency and as a cost-effective method for gathering deep-learning data. The comprehensive and practical learning experiences of VR offer an unparalleled approach to UAV operation training [70].
Further innovations include the development of DroneVR, a UAV training tool designed to mitigate drone crash risks through lightweight object detection and user-friendly interface. This underscores the critical importance of specialized tools in reducing operational risks and enhancing user competence, particularly in UAV operations [61]. VR training for drone-assisted bridge inspections, such as the TASBID system, addresses specific training needs and boosts confidence in drone bridge inspection tasks. This illustrates the effectiveness of VR in preparing operators for intricate inspection duties [37].
In exoskeleton training, VR-based training methods have shown superiority in enhancing performance and safety, especially when integrated with haptic feedback. Applied specifically to back exoskeletons for Direct Ground Lift (DGL) motion in the construction industry, this approach underlines the practical benefits of immersive training methods in reducing work-related musculoskeletal disorders, highlighting the health and safety advantages of VR training in physically demanding operations [40,44].
A novel VR platform for collaborative robot design and ergonomics was introduced as a pioneering approach to engineering education, relevant to Industry 4.0. This platform serves as a comprehensive tool for understanding and applying ergonomic design principles in robotic system development, showcasing the educational benefits of immersive technologies in engineering [36].
The adoption of 360-degree VR technology for training construction workers about the safety challenges associated with drones emphasizes the enhancement of safety awareness in complex construction environments. This technology stresses the importance of situational awareness and safety training in mitigating drone operation risks [32]. Furthermore, investigations into the cognitive benefits of using cobots in industrial operations have illustrated that cobots not only improve task performance and reduce errors but also bolster cognitive processes, enhancing efficiency in complex tasks. This reveals the multifaceted advantages of integrating cobots into industrial workflows [11]. Collectively, these studies underscore the transformative impact of VR and related immersive technologies in the realm of training and education within HRC. They present a variety of innovative applications that significantly improve safety, efficiency, and cognitive engagement across different industrial sectors, heralding a new era of enhanced learning and operational proficiency.
The realm of HRC has witnessed significant advances through the integration of collaborative and automated technologies, with VR playing a crucial role in propelling these developments forward. These innovations are pivotal in enhancing the efficiency and safety of HRC systems, contributing substantially to the refinement of HRI across various industrial sectors.
The capacity of VR to predict outcomes in robotic construction, especially with the integration of drones, has been demonstrated, indicating a move towards more efficient and innovative construction methodologies.
This approach highlights the ability of VR to accurately model complex construction scenarios, enabling the exploration of new construction techniques that could potentially set new industry benchmarks [39]. In terms of strategic development, the use of VR digital twins to safely test and refine HRC strategies has emerged as a groundbreaking method. This technique deepens the understanding of HRI, leading to safer and more efficient collaborative endeavors. By mirroring real-world scenarios within a risk-free environment, VR digital twins allow for the thorough testing of interactions, strategies, and workflows, playing a significant role in enhancing HRC practices [62].
Investigations into the effects of robot motion on human collaborators within VR environments have uncovered its considerable impact on performance and stress levels. These insights underscore the critical importance of incorporating human factors into the design and implementation of robotic systems, advocating for ergonomic and user-friendly robot motion designs to bolster collaboration without adversely affecting human operators [63].
Further studies on the influence of cobots in product assembly through VR have illustrated how cobots can improve both efficiency and ergonomic comfort. These assessments provide tangible evidence on the advantages of employing VR to simulate and evaluate HRC systems, affirming the value of integrating VR in the assessment process [57]. Additionally, the creation of a robot obstacle avoidance system using deep reinforcement learning marks a substantial leap in autonomous robot capabilities, enabling robots to navigate safely and effectively in intricate settings. This development showcases the potential of artificial intelligence and machine-learning technologies to augment the autonomy and safety of robots within collaborative frameworks [45]. Together, these advances underscore a growing trend towards incorporating VR into HRC,
where virtual simulations and feedback mechanisms play a crucial role in enhancing the interactions between humans and robots. This synergy fosters optimized operations, improved ergonomic practices, and deeper comprehension of the dynamic exchanges within collaborative environments, signifying a notable progression in HRC.
The fusion of VR with HRC has catalyzed a wave of innovative applications and significant advances in prototyping across a spectrum of industries, showcasing the transformative capacity of VR to boost task performance, streamline processes, and enhance HRC.
In the manufacturing sector, the deployment of a VR system in Gas Tungsten Arc Welding marks a notable advancement. This system facilitates precise tracking and adjustment of the welding torch, leading to substantial improvements in welding. Such an innovation exemplifies the ability of VR to refine traditional manufacturing techniques, presenting a leap forward in operational efficiency [60]. Moving to the aviation industry, the application of VR in drone collision avoidance testing has proven to be both cost-effective and superior in controlling the testing environment. While not a replacement for real-world trials, the capability of VR to emulate intricate scenarios underscores its potential to markedly reduce development time and expenses, particularly in aerospace applications [38].
In the textile industry, the creation of a novel VR framework for automating cloth folding represents a significant stride. Using skeletal data for garment classification, this innovation highlights the ability of VR to streamline manufacturing processes, minimizing manual labor dependency and boosting operational efficiency [28]. Venturing into the intersection of gaming technology with industrial applications, the adoption of a game engine-driven VR system for HRC in assembly tasks has not only heightened efficiency but also tapped into the creative reservoir of the open-source community. This method demonstrates the broad and versatile applications of VR in improving industrial processes [74].
A pivotal study in emergency structural inspection has benchmarked human versus robot performance in detecting structural damages, providing essential insights into the complementary strengths and limitations of humans and robots in executing critical inspection tasks. This analysis elucidates the potential of VR as an invaluable tool in the training and evaluation of both human and robotic inspectors, underscoring the complementarity of human intuition and robotic precision [43].
These advances illuminate the vast array of innovative applications of VR within HRC. From refining traditional manufacturing practices to automating intricate tasks and elevating efficiency in essential inspection scenarios, the integration of VR and HRC is reshaping the landscape of industrial operations, heralding a new era of enhanced performance and collaboration across various sectors. The analysis examines contributions and explores potential developments in safety, task efficiency, interaction improvements, training, collaborative technologies, and innovative prototyping within these domains. Table 3 summarizes the frequency of each domain of interest. [21], [68], [69], [77], [42], [34], [71], [78], and [67].
Task efficiency and design 11 [46], [35], [27], [49], [41], [29], [31], [76], [56], [47], and [64].
Human-interaction improvement 15 [75], [53], [54], [72], [55], [73], [66], [58], [50], [30], [51], [65], [52], [33], and [48].
Training and education 9 [59], [70], [61], [37], [40], [44], [36], [32], and [11].
Collaborative and automated technologies 5 [39], [62], [63], [57], and [45].
Innovative applications and prototyping 5 [60], [38], [28], [74], and [43].
This review identified six domains of interest in HRC-VR studies as illustrated in Figure 7.
In addressing RQ2-which inquiries about the types of robots predominantly employed in HRC studies within VR environments across diverse industries-the investigation identified and categorized various robots. These categorizations are based on the applications of the robots and their frequency of appearance in the studies reviewed. The detailed discussion that follows outlines these categories and delves into the specific types of robots that have been used within these innovative environments, particularly focusing on the construction and manufacturing industries. This segmentation clarifies the role of different robots in HRC
In the field of VR-enhanced HRC, particularly in automation and manufacturing, the analysis of 53 papers shows a significant focus on industrial robotic arms. Out of these, 23 studies emphasize their crucial role.
KUKA robotic arms are notably prevalent, featured in 7 studies for their robustness and high performance in heavy industrial settings. Stäubli RX90L, recognized in 6 studies, stands out for its precision and versatility, illustrating its capacity to support a wide range of tasks, from delicate assembly to precise manipulation. The ABB IRB series, cited in 4 studies, is lauded for its automation efficiency, demonstrating the contribution of the series to streamlining processes and enhancing productivity in HRC-VR environments. NACHI MZ07, mentioned in 2 studies, is distinguished for its precision, showcasing its suitability for tasks that demand accuracy and detail.
Articulated, joint-linked robot designs conducive to collaborative tasks and ergonomics are scrutinized for their potential to improve HRC. General robotic arms are also investigated for their role in virtual collaborative assembly, highlighting the significance of eye-hand coordination and control mechanisms in optimizing collaborative tasks within VR settings.
In this category, 11 papers addressed cobots, recognized for their safety features and capacity for effective interaction in environments where they collaborate closely with human workers. Baxter robots featured in
0 5 10 15 20 25 30 Industrial Robatic Arms Mobile Robots Drones Exoeskeletons AGVs Collaborative Robots Novint Falcon ROS-Simulated Ground Robot
Robot types in HRC-VR research studies three studies. For example, research focuses on analyzing and controlling robot motion in response to human hand movements. The UR-5 and UR-10 models of Universal Robots are recognized for their flexibility and ease of use. The UR3 model was examined for its ability to avoid obstacles through deep reinforcement learning. Furthermore, the Franka robot model was explored, optimizing its ergonomic aspects using Particle Swarm Optimization in VR settings. This research reflects the various roles of cobots in improving HRI within the construction and manufacturing industries, covering areas from motion control and ergonomic improvements to learning algorithms for navigation and avoiding obstacles.
UAVs or drones have been included in the study owing to their unique interaction model, which primarily involves remote control or piloting. Advances in construction and manufacturing have expanded this interaction to collaborative tasks. UAVs interact with humans through various modes, including remote control, gestures, and even haptic feedback, as highlighted in the field of human-drone interaction. Through a review of nine papers, the multifaceted roles of drones in these VR environments have been highlighted,
showcasing their increasing value in tasks ranging from operational training to safety surveillance [69].
Drone-VR, a web VR simulator aimed at minimizing drone crash risks while providing comprehensive
UAV training, exemplifies the integration of VR in enhancing drone operator proficiency and safety, emphasizing the importance of specialized training environments in reducing operational errors [60]. A VRbased training system specifically designed for bridge inspectors using assistant drones was introduced, underscoring the potential of VR in specialized training. This caters to the unique requirements of bridge inspection, highlighting the adaptability of drones to support intricate inspection tasks [37]. The effectiveness of drone operation training in VR environments has been evaluated using physiological data models, pointing to innovative approaches to assessing and enhancing drone operator skills. This review emphasizes the potential of VR to create realistic training scenarios that accurately measure operator readiness and performance [53].
Product testing in VR, focusing on the development of drone collision avoidance systems, has been explored, reflecting the economic and operational benefits of employing CAVE VR technology for drone development and highlighting the role of VR in advancing drone safety technologies [38]. The application of drones in robotic construction analysis has demonstrated the use of VR in simulating construction processes, facilitating a deeper understanding of the potential of drones to streamline and enhance construction methodologies [39].
Construction workers were trained on drone safety using 360-degree VR technology, emphasizing the critical need for comprehensive safety education in leveraging drones within construction environments. This
showcases the effectiveness of VR in promoting awareness and understanding drone safety protocols [32].
Lastly, the psychological and safety impacts of drones on construction professionals have been studied, assessing how drones affect worker safety and productivity. This highlights the broader implications of drone use in construction, considering both the operational benefits and potential challenges posed to worker wellbeing [33,34]. Collectively, these studies illuminate the diverse and innovative roles of drones in VR environments, from enhancing operational skills and safety training to improving efficiency in construction processes, indicating the pivotal role of drones in advancing the capabilities and understanding of HRC in construction and manufacturing scenarios.
In the scope of VR-enhanced HRC, the employment of mobile robots, such as the MiR100, was explored in three papers, highlighting the significant advantages of robots in mobility and autonomous operation within VR environments. These mobile robots are noted for their ability to navigate and perform tasks autonomously, offering innovative solutions to enhance operational efficiency and safety in various industrial settings.
The use of mobile robots in construction to enhance safety on construction sites was examined, providing insights into the potential of mobile robots to improve safety protocols and practices within construction environments [77]. This study underscores the importance of mobile robots in contributing to safer work environments through autonomous monitoring and intervention capabilities.
CoBot Studio VR, a VR platform designed for transdisciplinary research into HRC, introduces an innovative approach to studying the interaction between humans and robots [48]. The study explores trust and signal interpretability in HRI, employing the MiR100, an autonomous mobile robot, alongside the UR10, an industrial robotic arm. This investigation, though focused on VR environments, offers valuable insights for enhancing collaborative dynamics in real-world industrial contexts, including construction and manufacturing.
A study on mobile robots used for collaborative space exploration and mapping presents the potential of these technologies in fields beyond their traditional applications [58]. While primarily aimed at space exploration, the outcomes and technological advances discussed have significant implications for industrial sectors, particularly in facilitating collaborative operations and managing complex tasks in environments like construction and manufacturing. The diverse capabilities and potential applications of mobile robots in enhancing operational efficiency, safety, and collaborative dynamics within VR-enhanced HRC environments are demonstrated through these papers. This highlights the relevance and adaptability of mobile robots to meet the evolving needs of industries such as construction and manufacturing, underscoring their role in advancing the integration of robotics and VR technologies.
In the exploration of specialized robotics and assistive systems within VR-enhanced HRC, the innovative employment of Automated Guided Vehicles (AGVs) and cobots with specialized end-effectors, back exoskeletons, and robotic teleoperation devices across various studies has been highlighted. These technologies significantly contribute to the advancement of HRC in industrial environments, showcasing their potential in optimizing operations, enhancing safety, and improving HRI dynamics. AGVs and a cobot with a specialized end-effector were used in studies showcasing two separate case studies. The AGVs were crucial in material transport, while the cobot was pivotal in precision assembly tasks, demonstrating the versatility and effectiveness of integrating specialized robotics in industrial settings for various operational needs [46]. The application of a back exoskeleton in VR training designed for construction workers employed haptic sensation transfer aimed at reducing musculoskeletal disorders, indicating the potential of assistive systems in enhancing worker health and safety through immersive training environments [40,44]. The Novint
Falcon, a robotic teleoperation device, was the focus of studies aimed at enhancing HRC in industrial settings.
Adaptive HRI within VR for tasks using the Novint Falcon integrated VR with neurofunctional analysis to In response to RQ3-which explores the types of VR setups and software solutions most employed in VR environments for HRC within the construction and manufacturing industries-the comprehensive review and analysis of the selected papers have highlighted a significant trend. The findings indicate that Head-Mounted Displays (HMDs) are the predominant hardware tools used in HRC research, with their application being documented in 39 out of the 53 papers reviewed. This suggests a strong preference for HMDs in facilitating immersive and effective HRC within these industrial VR applications. This popularity is likely due to their affordability and increasing accessibility in recent years. The HTC VIVE and Oculus Rift DK2 are particularly notable for being used in nine and seven studies, respectively. Unique HMD models like the Pimax 5k+ VR headset [54], eMagin Z800 3DVisorTM HMD [68], and HTC VIVE Pro Eye [48] are highlighted in individual studies.
Desktop VR setups, which involve interaction with a virtual environment through a standard computer screen, are identified in nine papers. This type of VR setup provides a practical and less immersive option for specific HRC applications [37,77]. CAVE (Cave Automatic Virtual Environment) systems, which offer a semi-immersive VR experience with room-sized projections, are employed in three papers [38,69,72]. In three papers, the specific VR hardware setup was not identified. In addition, an unspecified HMD model was used [46]. Table 5 shows the articles, which are categorized and listed based on the VR setup. Among the VR setups used for HRC, the HTC VIVE has the most used VR setup in HRC-VR, as presented in Figure 9. 0 2 4 6 8 10 12 14 16 18 HTC VIVE Desktop Oculus Rift DK2 HTC Vive Pro Oculus Rift CAVE N/A Oculus Quest 2 HTC VIVE Pro Eye Oculus Rift S HMDs ( Not Specified) eMagin Z800 3DVisor Pimax 5k+ Frequency Type of VR VR setup in HRC-VR [75], [39], [76], [36], [73], [57], and [67] 8
HTC VIVE [59], [35], [60], [40], [62], [56], [74], [11], [58], [41], [50], [29], [42], [43],
Oculus Rift [59], [47], and [63] 3
Oculus Rift S [51] 1
Oculus Quest 2 [28] and [30] 2 HTC VIVE PRO [27], [55], [78], [33], and [34] 5
HTC VIVE PRO EYE [48] 1 eMagin Z800 3DVisor [68] 1
Pimax 5k+ [54] 1 (Not Specified) [46] 1 Total 40
2D screen (Desktop) [77], [70], [61], [37], [53], [49], [45], [65], and [32] 9 CAVE [69], [38], and [72] 3
The information not available [71], [66], and [64] 3
Regarding software, the analysis of the selected papers has shown that a wide variety of software tools are employed in the development of VR interfaces for HRC. Figure 7 illustrates the core software used and
showcases their respective occurrences across the articles. The Unity 3D game engine is the most prominently featured, used in 36 studies. This prevalence draws attention to the capability of Unity in crafting immersive and interactive VR environments, making it a preferred choice for HRC applications. The second most frequently used tool in these studies is the Microsoft Kinect II sensor, which features in eight papers. Its application is primarily for motion tracking and gesture recognition, which are key elements in increasing the interactivity of VR environments [36,76]. It is followed by Rhinoceros, which is used in seven studies for its precision in 3D modeling [39,73]. This functionality is essential for creating detailed and accurate VR simulations, highlighting its importance in HRC-VR interfaces. Additionally, 3ds Max and Evolver Avatar Builder, each used in four studies, showcase their strengths in 3D modeling and animation. For example, 3ds
Max is used for its modeling capabilities [76]. Evolver Avatar Builder is employed in creating detailed avatars in VR environments [36]. Leap Motion, found in three papers, enhances user interaction in VR with its advanced hand-tracking technology. This is exemplified in a study where the capabilities of Leap Motion are leveraged for more natural user interactions [48]. Novint Falcon and Visual Components, although mentioned in only two studies each, offer unique functionalities to the VR experiences. Novint Falcon is known for its haptic feedback, and is used to enhance tactile sensations in VR [11]. Visual Components, valuable for industrial simulation, is used to simulate complex industrial processes [78].
The analysis also reveals that specific software tools were chosen for their particular features to meet the unique requirements of individual studies. For instance, Unreal Engine 4 for UAV simulation [70], JavaScript libraries and mapping tools for VR UAV navigation [61], and a combination of Unity 3D, Microsoft Kinect, and other 3D tools for comprehensive VR setups [76]. Table 6 shows the articles categorized and listed based on the software run. Among identified software used for HRC, Unity 3D TM has the most used VR setup in HRC-VR as presented in Figure 10. Unity 3D™ [21], [68], [46], [59], [77], [60], [37], [39], [28], [76], [62], [54], [36], [47], [55], [63],
[73], [78], [74], [11], [57], [48], [67], [58], [49], [50], [29], [30], [51], [43], [31], [52], [32], [33], [34] and [44].
Microsoft Kinect II sensor [21], [68], [69], [27], [76], [36], [56], [73], [66], [67] and [45]. 11
Tobii eye tracker [49], [29], [42], [43], [31], [65], [33] and [34] 8
Rhinoceros™ [21], [68], [39], [76], [36], [73] and [57] 7
3ds Max™ [21], [68], [76] and [73] 4
Evolver avatar builder 4
A systematic literature review spanning 2016 to 2023 was conducted, focusing on HRC integrated with VR within the construction and manufacturing industries. The review examined the application domains, types of robots employed, and various VR setups and software solutions used in these scenarios. Notably, within the application domains and use cases, the human-interaction improvement category was predominant compared to others. This trend highlights a significant focus on enhancing the dynamics of HRI, with research directed toward optimizing interfaces and understanding the effects of robot behavior on operators. The findings showed widespread use of industrial robotic arms and cobots in HRC-VR studies, along with a notable presence of drones. In contrast, exoskeletons were rare, with only one paper in the review addressing their use in HRC with VR. HMDs, especially the HTC VIVE, were the main VR setup, showing a trend towards more immersive VR. Unity, a versatile game engine for interactive 3D content, was the most commonly used software in HRC-VR studies. Meanwhile, Unreal Engine, known for its high-quality graphics and real-time rendering capabilities, appeared in only two studies. Despite this, its potential for future HRC-VR work should not be underestimated, as it could offer enhanced realism and detail in VR simulations, potentially increasing its use in more complex or visually demanding HRC applications.
Microsoft Kinect II sensors were also used frequently for their accurate motion tracking in VR, showing the need for precise interaction in these simulations.
A significant preference for virtual robots in HRC-VR studies was noted, reflecting the trend toward virtual simulations. Real robots were less commonly used, with some studies integrating both virtual and real robots.
A few papers did not use robots at all and used VR eye tracking for arm movement prediction, indicating varied approaches in this field. It was found that nearly half of the papers used digital humans, known as avatars, while the other half did not, indicating a balanced exploration of avatar-based and non-avatar approaches in HRC-VR research. Furthermore, it was identified that assistance (L2), cooperation (L3), and collaboration (L4) were commonly used, while fully programmed (L0), co-existence (L1), and fully autonomous level (L5) were not used in HRC-VR. This suggests that levels involving minimal human-robot