More Like this

1. ROV Teleoperation based on Sensory Augmentation and Digital Twins

   https://doi.org/10.4043/32376-MS  

   Xia, Pengxiang ; McSweeney, Kevin P. ; Song, Zhuoyuan ; Du, Eric ( April 2023 , OTC)

   ROV operations are mainly performed via a traditional control kiosk and limited data feedback methods, such as the use of joysticks and camera view displays equipped on a surface vessel. This traditional setup requires significant personnel on board (POB) time and imposes high requirements for personnel training. This paper proposes a virtual reality (VR) based haptic-visual ROV teleoperation system that can substantially simplify ROV teleoperation and enhance the remote operator's situational awareness.

   This study leverages the recent development in Mixed Reality (MR) technologies, sensory augmentation, sensing technologies, and closed-loop control, to visualize and render complex underwater environmental data in an intuitive and immersive way. The raw sensor data will be processed with physics engine systems and rendered as a high-fidelity digital twin model in game engines. Certain features will be visualized and displayed via the VR headset, whereas others will be manifested as haptic and tactile cues via our haptic feedback systems. We applied a simulation approach to test the developed system.

   With our developed system, a high-fidelity subsea environment is reconstructed based on the sensor data collected from an ROV including the bathymetric, hydrodynamic, visual, and vehicle navigational measurements. Specifically, the vehicle is equipped with a navigation sensor system for real-time state estimation, an acoustic Doppler current profiler for far-field flow measurement, and a bio-inspired artificial literal-line hydrodynamic sensor system for near-field small-scale hydrodynamics. Optimized game engine rendering algorithms then visualize key environmental features as augmented user interface elements in a VR headset, such as color-coded vectors, to indicate the environmental impact on the performance and function of the ROV. In addition, augmenting environmental feedback such as hydrodynamic forces are translated into patterned haptic stimuli via a haptic suit for indicating drift-inducing flows in the near field. A pilot case study was performed to verify the feasibility and effectiveness of the system design in a series of simulated ROV operation tasks.

   ROVs are widely used in subsea exploration and intervention tasks, playing a critical role in offshore inspection, installation, and maintenance activities. The innovative ROV teleoperation feedback and control system will lower the barrier for ROV pilot jobs.

    

   more » « less
2. "What Does Your Robot Do?" A Tabletop Role-Playing Game to Support Robot Design

   https://doi.org/10.1109/RO-MAN50785.2021.9515554  

   Collins, Sawyer ; Sabanovic, Selma ( August 2021 , 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN))

   null (Ed.)

   Social robot co-design requires aiding users as they imagine these novel devices within their everyday lives and enabling designers to understand and address users’ experiences. This paper presents the exploratory development and evaluation of a role-playing game aimed at identifying the desired features and uses of a social robot that can assist people diagnosed with depression. Participants (n = 16) played the game as a character with depression, designed a companion robot for that character, and chose reactions to daily challenges. Though participants initially selected robot capabilities based on their own needs, after the game they identified alternative designs that would better address daily challenges faced by individuals with depression. We discuss aspects of the game that allowed participants to understand how various robot characteristics can address the experience of depression and suggest how role-playing games can support users and designers in identifying beneficial features and uses of emerging robotic technologies. 

   more » « less
3. The role of representational competencies for students’ learning from an educational video game for astronomy

   https://doi.org/10.3389/feduc.2022.919645  

   Herder, Tiffany ; Rau, Martina A. ( November 2022 , Frontiers in Education)

   Educational video games can engage students in authentic STEM practices, which often involve visual representations. In particular, because most interactions within video games are mediated through visual representations, video games provide opportunities for students to experience disciplinary practices with visual representations. Prior research on learning with visual representations in non-game contexts suggests that visual representations may confuse students if they lack prerequisite representational-competencies. However, it is unclear how this research applies to game environments. To address this gap, we investigated the role of representational-competencies for students’ learning from video games. We first conducted a single-case study of a high-performing undergraduate student playing an astronomy game as an assignment in an astronomy course. We found that this student had difficulties making sense of the visual representations in the game. We interpret these difficulties as indicating a lack of representational-competencies. Further, these difficulties seemed to lead to the student’s inability to relate the game experiences to the content covered in his astronomy course. A second study investigated whether interventions that have proven successful in structured learning environments to support representational-competencies would enhance students’ learning from visual representations in the video game. We randomly assigned 45 students enrolled in an undergraduate course to two conditions. Students either received representational-competency support while playing the astronomy game or they did not receive this support. Results showed no effects of representational-competency supports. This suggests that instructional designs that are effective for representational-competency supports in structured learning environments may not be effective for educational video games. We discuss implications for future research, for designers of educational games, and for educators. 

   more » « less
4. Mobile augmented reality supporting families’ immersive collaborative learning: Learning-on-the-move for place-based geoscience sense-making

   https://doi.org/10.1007/s11412-023-09399-9  

   Zimmerman, Heather Toomey ; Land, Susan M. ; Faimon, Lillyanna ; Chiu, Yu-Chen ( July 2023 , International Journal of Computer-Supported Collaborative Learning)

   We investigated how families experienced immersion as they collaboratively made sense of geologic time and geoscience processes during a place-based, learning-on-the-move (LOTM) experience mediated by a mobile augmented reality (MAR) app. Our team developed an MAR app,Time Explorers, that focused on how rock-water interactions shaped Appalachia over millions of years. Data were collected at the Children’s Garden at the Arboretum at Penn State. Data sources were videos of app usage, point-of-view camera recordings with audio capturing family conversations, and interviews from 17 families (51 people). The analytical technique was interaction analysis, in which episodes of family sense-making were identified and developed into qualitative vignettes focused on how immersion did or did not support learning about geoscience and geologic time. We analyzed how design elements supported sensory, actional, narrative, and social immersion through photo-taking, discussion prompts, and augmented reality visualizations. Findings showed that sensory and social immersion supported sense-making conversations and observational inquiry, while narrative and actional immersion supported deep family engagement with the geoscience content. At many micro-sites of learning, families engaged in multiple immersive processes where conversations, observational inquiry, and deep engagement with the geoscience came together during LOTM. This analysis contributes to the CSCL literature on theory related to LOTM in outdoor informal settings, while also providing design conjectures in an immersive, family-centered, place-based LOTM framework.

    

   more » « less
5. Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence

   https://doi.org/10.1038/s41598-024-54734-1  

   Du, Jing ; Vann, William ; Zhou, Tianyu ; Ye, Yang ; Zhu, Qi ( February 2024 , Scientific Reports)

   In the realm of robotics and automation, robot teleoperation, which facilitates human–machine interaction in distant or hazardous settings, has surged in significance. A persistent issue in this domain is the delays between command issuance and action execution, causing negative repercussions on operator situational awareness, performance, and cognitive load. These delays, particularly in long-distance operations, are difficult to mitigate even with the most advanced computing advancements. Current solutions mainly revolve around machine-based adjustments to combat these delays. However, a notable lacuna remains in harnessing human perceptions for an enhanced subjective teleoperation experience. This paper introduces a novel approach of sensory manipulation for induced human adaptation in delayed teleoperation. Drawing from motor learning and rehabilitation principles, it is posited that strategic sensory manipulation, via altered sensory stimuli, can mitigate the subjective feeling of these delays. The focus is not on introducing new skills or adapting to novel conditions; rather, it leverages prior motor coordination experience in the context of delays. The objective is to reduce the need for extensive training or sophisticated automation designs. A human-centered experiment involving 41 participants was conducted to examine the effects of modified haptic cues in teleoperations with delays. These cues were generated from high-fidelity physics engines using parameters from robot-end sensors or physics engine simulations. The results underscored several benefits, notably the considerable reduction in task time and enhanced user perceptions about visual delays. Real-time haptic feedback, or the anchoring method, emerged as a significant contributor to these benefits, showcasing reduced cognitive load, bolstered self-confidence, and minimized frustration. Beyond the prevalent methods of automation design and training, this research underscores induced human adaptation as a pivotal avenue in robot teleoperation. It seeks to enhance teleoperation efficacy through rapid human adaptation, offering insights beyond just optimizing robotic systems for delay compensations.

    

   more » « less