Teleoperation in agricultural environments requires enhanced situational awareness for effective architectural scouting and decision-making for orchard management applications. The dynamic complexity of orchard structures presents challenges for remote visualization during architectural scouting operations. This study presents an adaptive streaming and rendering pipeline for real-time point cloud visualization in Virtual Reality (VR) teleoperation systems. The proposed method integrates selective streaming that localizes teleoperators within live maps, an efficient point cloud parser for Unity Engine, and an adaptive Level-of-Detail rendering system utilizing dynamically scaled and smoothed polygons. The implementation incorporates pseudo-coloring through LiDAR reflectivity fields to enhance the distinction between materials and geometry. The pipeline was evaluated using datasets containing LiDAR point cloud scans of orchard environments captured during spring and summer seasons, with testing conducted on both standalone and PC-tethered VR configurations. Performance analysis demonstrated improvements of 10.2–19.4% in runtime performance compared to existing methods, with a framerate enhancement of up to 112% achieved through selectively streamed representations. Qualitative assessment confirms the method’s capability to maintain visual continuity at close proximity while preserving the geometric features discernible for architectural scouting operations. The results establish the viability of VR-based teleoperation for precision agriculture applications, while demonstrating the critical relationship between Quality-of-Service parameters and operator Quality of Experience in remote environmental perception.
more »
« less
Telemanipulation via Virtual Reality Interfaces with Enhanced Environment Models
Extreme environments, such as search and rescue missions, defusing bombs, or exploring extraterrestrial planets, are unsafe environments for humans to be in. Robots enable humans to explore and interact in these environments through remote presence and teleoperation and virtual reality provides a medium to create immersive and easy-to-use teleoperation interfaces. However, current virtual reality interfaces are still very limited in their capabilities. In this work, we aim to advance robot teleoperation virtual reality interfaces by developing an environment reconstruction methodology capable of recognizing objects in a robot’s environment and rendering high fidelity models inside a virtual reality headset. We compare our proposed environment reconstruction method against traditional point cloud streaming by having operators plan waypoint trajectories to accomplish a pick-and-place task. Overall, our results show that users find our environment reconstruction method more usable and less cognitive work compared to raw point cloud streaming.
more »
« less
- Award ID(s):
- 1944453
- PAR ID:
- 10334715
- Date Published:
- Journal Name:
- 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Subject(s) / Keyword(s):
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Robot teleoperation has become increasingly crucial for extending human capabilities in inaccessible or hazardous environments and facilitating human–robot collaboration. While significant advancements have been made in teleoperation interfaces, the success of these systems critically depends on how effectively humans can interact with and control robotic systems across diverse manipulation tasks. However, existing research primarily evaluates interfaces within specific tasks or applications, lacking systematic assessment across different manipulation scenarios. This limitation leads to suboptimal interface selection that can compromise task efficiency in critical domains and impede the collection of high-quality demonstrations for robot learning. To address these gaps, we first introduce a novel two-axis Robotic Manipulation Task Taxonomy that systematically categorizes manipulation tasks based on their fundamental control requirements: Motion Type (translation-dominant vs. rotation-dominant) and Engagement Type (rigid vs. non-rigid object interactions). We conducted a comprehensive user study (\(n=30\)) evaluating three distinct teleoperation interfaces (Gamepad, 3D Mouse, and Virtual Reality (VR) Controller) based on this taxonomy. Our results indicate that the Gamepad and 3D mouse significantly outperformed the VR Controller in task completion time across all task categories. In contrast, the VR Controller showed higher first-attempt success rates but caused significantly greater cognitive load across all NASA Task Load Index (NASA-TLX) subscales compared to the Gamepad interface, and specifically higher mental demand, physical demand, and effort compared to the 3D mouse. Our results also indicate that although prior familiarity with an interface lowered perceived workload and enhanced perceived performance, it did not translate into improvements in actual task success rates or completion times. These insights provide valuable guidelines for optimizing teleoperation interfaces based on task requirements and highlight the importance of considering both cognitive demands and user experience in interface design.more » « less
-
By projecting into a 3-D workspace, robotic teleoperation using virtual reality allows for a more intuitive method of control for the operator than using a 2-D view from the robot's visual sensors. This paper investigates a setup that places the teleoperator in a virtual representation of the robot's environment and develops a deep learning based architecture modeling the correspondence between the operator's movements in the virtual space and joint angles for a humanoid robot using data collected from a series of demonstrations. We evaluate the correspondence model's performance in a pick-and - place teleoperation experiment.more » « less
-
With the advancement of automation and robotic technologies, the teleoperation has been leveraged as a promising solution for human workers in a hazardous construction work environment. Since human operators and construction sites are separated in a distance, teleoperation requires a seamless human-machine interface, an intermediate medium, to communicate between humans and machines in construction sites. Several types of teleoperation interfaces including conventional joysticks, haptic devices, graphic user interfaces, auditory interfaces, and tactile interfaces have been developed to control and command construction robotics remotely. The ultimate goal of human-machine interfaces for remote operations is to make intuitive sensory channels that can provide and receive enough information while reducing the associated cognitive and physical load on human operators. Previously developed interfaces have tried to achieve such goals, but each interface still has challenges that should be assessed for enhancing the future teleoperation application in construction workplaces. This paper examines different human-machine interfaces for excavator teleoperation in terms of its on-site usability and intuitiveness. The capabilities of the interfaces for excavator teleoperation are evaluated based on their limitations and requirements. The outcome is expected to provide better understanding of teleoperation interfaces for excavators and guiding future directions for addressing underlying challenges.more » « less
-
With design teams becoming more distributed, the sharing and interpreting of complex data about design concepts/prototypes and environments have become increasingly challenging. The size and quality of data that can be captured and shared directly affects the ability of receivers of that data to collaborate and provide meaningful feedback. To mitigate these challenges, the authors of this work propose the real-time translation of physical objects into an immersive virtual reality environment using readily available red, green, blue, and depth (RGB-D) sensing systems and standard networking connections. The emergence of commercial, off-the-shelf RGB-D sensing systems, such as the Microsoft Kinect, has enabled the rapid three-dimensional (3D) reconstruction of physical environments. The authors present a method that employs 3D mesh reconstruction algorithms and real-time rendering techniques to capture physical objects in the real world and represent their 3D reconstruction in an immersive virtual reality environment with which the user can then interact. Providing these features allows distributed design teams to share and interpret complex 3D data in a natural manner. The method reduces the processing requirements of the data capture system while enabling it to be portable. The method also provides an immersive environment in which designers can view and interpret the data remotely. A case study involving a commodity RGB-D sensor and multiple computers connected through standard TCP internet connections is presented to demonstrate the viability of the proposed method.more » « less
An official website of the United States government

