More Like this

1. RealTHASC—a cyber-physical XR testbed for AI-supported real-time human autonomous systems collaborations

   https://doi.org/10.3389/frvir.2023.1210211  

   Paradise, Andre; Surve, Sushrut; Menezes, Jovan C.; Gupta, Madhav; Bisht, Vaibhav; Jang, Kyung Rak; Liu, Cong; Qiu, Suming; Dong, Junyi; Shin, Jane; et al (December 2023, Frontiers in Virtual Reality)

   Gonzalez, D. (Ed.)

   Today’s research on human-robot teaming requires the ability to test artificial intelligence (AI) algorithms for perception and decision-making in complex real-world environments. Field experiments, also referred to as experiments “in the wild,” do not provide the level of detailed ground truth necessary for thorough performance comparisons and validation. Experiments on pre-recorded real-world data sets are also significantly limited in their usefulness because they do not allow researchers to test the effectiveness of active robot perception and control or decision strategies in the loop. Additionally, research on large human-robot teams requires tests and experiments that are too costly even for the industry and may result in considerable time losses when experiments go awry. The novel Real-Time Human Autonomous Systems Collaborations (RealTHASC) facility at Cornell University interfaces real and virtual robots and humans with photorealistic simulated environments by implementing new concepts for the seamless integration of wearable sensors, motion capture, physics-based simulations, robot hardware and virtual reality (VR). The result is an extended reality (XR) testbed by which real robots and humans in the laboratory are able to experience virtual worlds, inclusive of virtual agents, through real-time visual feedback and interaction. VR body tracking by DeepMotion is employed in conjunction with the OptiTrack motion capture system to transfer every human subject and robot in the real physical laboratory space into a synthetic virtual environment, thereby constructing corresponding human/robot avatars that not only mimic the behaviors of the real agents but also experience the virtual world through virtual sensors and transmit the sensor data back to the real human/robot agent, all in real time. New cross-domain synthetic environments are created in RealTHASC using Unreal Engine™, bridging the simulation-to-reality gap and allowing for the inclusion of underwater/ground/aerial autonomous vehicles, each equipped with a multi-modal sensor suite. The experimental capabilities offered by RealTHASC are demonstrated through three case studies showcasing mixed real/virtual human/robot interactions in diverse domains, leveraging and complementing the benefits of experimentation in simulation and in the real world. 

   more » « less
2. Integrated Design of Augmented Reality Spaces Using Virtual Environments

   https://doi.org/10.1109/ISMAR55827.2022.00045  

   Scargill, Tim; Chen, Ying; Marzen, Nathan; Gorlatova, Maria (October 2022, 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR))

   Demand is growing for markerless augmented reality (AR) experiences, but designers of the real-world spaces that host them still have to rely on inexact, qualitative guidelines on the visual environment to try and facilitate accurate pose tracking. Furthermore, the need for visual texture to support markerless AR is often at odds with human aesthetic preferences, and understanding how to balance these competing requirements is challenging due to the siloed nature of the relevant research areas. To address this, we present an integrated design methodology for AR spaces, that incorporates both tracking and human factors into the design process. On the tracking side, we develop the first VI-SLAM evaluation technique that combines the flexibility and control of virtual environments with real inertial data. We use it to perform systematic, quantitative experiments on the effect of visual texture on pose estimation accuracy; through 2000 trials in 20 environments, we reveal the impact of both texture complexity and edge strength. On the human side, we show how virtual reality (VR) can be used to evaluate user satisfaction with environments, and highlight how this can be tailored to AR research and use cases. Finally, we demonstrate our integrated design methodology with a case study on AR museum design, in which we conduct both VI-SLAM evaluations and a VR-based user study of four different museum environments. 

   more » « less
3. Studies in spatial aural perception: establishing foundations for immersive sonification

   https://doi.org/https://doi.org/10.21785/icad2019.017  

   Bukvic, Ivica Ico; Earle, Gregory; Sardana, Disha; Joo, Woohun. (January 2019, International Conference on Auditory Display)

   The Spatial Audio Data Immersive Experience (SADIE) project aims to identify new foundational relationships pertaining to human spatial aural perception, and to validate existing relationships. Our infrastructure consists of an intuitive interaction interface, an immersive exocentric sonification environment, and a layer-based amplitude-panning algorithm. Here we highlight the systemﾒs unique capabilities and provide findings from an initial externally funded study that focuses on the assessment of human aural spatial perception capacity. When compared to the existing body of literature focusing on egocentric spatial perception, our data show that an immersive exocentric environment enhances spatial perception, and that the physical implementation using high density loudspeaker arrays enables significantly improved spatial perception accuracy relative to the egocentric and virtual binaural approaches. The preliminary observations suggest that human spatial aural perception capacity in real-world-like immersive exocentric environments that allow for head and body movement is significantly greater than in egocentric scenarios where head and body movement is restricted. Therefore, in the design of immersive auditory displays, the use of immersive exocentric environments is advised. Further, our data identify a significant gap between physical and virtual human spatial aural perception accuracy, which suggests that further development of virtual aural immersion may be necessary before such an approach may be seen as a viable alternative. 

   more » « less
4. Head Pose for Object Deixis in VR-Based Human-Robot Interaction

   https://doi.org/10.1109/RO-MAN53752.2022.9900631  

   Higgins, Padraig; Barron, Ryan; Matuszek, Cynthia (August 2022, International Conference on Robot & Human Interactive Communication (Ro-Man))

   Modern robotics heavily relies on machine learning and has a growing need for training data. Advances and commercialization of virtual reality (VR) present an opportunity to use VR as a tool to gather such data for human-robot interactions. We present the Robot Interaction in VR simulator, which allows human participants to interact with simulated robots and environments in real-time. We are particularly interested in spoken interactions between the human and robot, which can be combined with the robot's sensory data for language grounding. To demonstrate the utility of the simulator, we describe a study which investigates whether a user's head pose can serve as a proxy for gaze in a VR object selection task. Participants were asked to describe a series of known objects, providing approximate labels for the focus of attention. We demonstrate that using a concept of gaze derived from head pose can be used to effectively narrow the set of objects that are the target of participants' attention and linguistic descriptions. 

   more » « less
5. On the relationship between network connectivity and group performance in small teams of humans: experiments in virtual reality

   https://doi.org/10.1088/2632-072X/ab916f  

   Barak-Ventura, Roni; Richmond, Samuel; Hasanyan, Jalil; Porfiri, Maurizio (July 2020, Journal of Physics: Complexity)

   Abstract Optimizing group performance is one of the principal objectives that underlie human collaboration and prompts humans to share resources with each other. Connectivity between individuals determines how resources can be accessed and shared by the group members, yet, empirical knowledge on the relationship between the topology of the interconnecting network and group performance is scarce. To improve our understanding of this relationship, we created a game in virtual reality where small teams collaborated toward a shared goal. We conducted a series of experiments on 30 groups of three players, who played three rounds of the game, with different network topologies in each round. We hypothesized that higher network connectivity would enhance group performance due to two main factors: individuals’ ability to share resources and their arousal. We found that group performance was positively associated with the overall network connectivity, although registering a plateau effect that might be associated with topological features at the node level. Deeper analysis of the group dynamics revealed that group performance was modulated by the connectivity of high and low performers in the group. Our findings provide insight into the intricacies of group structures, toward the design of effective human teams. 

   more » « less