An official website of the United States government
IntroductionAs robot teleoperation increasingly becomes integral in executing tasks in distant, hazardous, or inaccessible environments, operational delays remain a significant obstacle. These delays, inherent in signal transmission and processing, adversely affect operator performance, particularly in tasks requiring precision and timeliness. While current research has made strides in mitigating these delays through advanced control strategies and training methods, a crucial gap persists in understanding the neurofunctional impacts of these delays and the efficacy of countermeasures from a cognitive perspective. MethodsThis study addresses the gap by leveraging functional Near-Infrared Spectroscopy (fNIRS) to examine the neurofunctional implications of simulated haptic feedback on cognitive activity and motor coordination under delayed conditions. In a human-subject experiment (N= 41), sensory feedback was manipulated to observe its influences on various brain regions of interest (ROIs) during teleoperation tasks. The fNIRS data provided a detailed assessment of cerebral activity, particularly in ROIs implicated in time perception and the execution of precise movements. ResultsOur results reveal that the anchoring condition, which provided immediate simulated haptic feedback with a delayed visual cue, significantly optimized neural functions related to time perception and motor coordination. This condition also improved motor performance compared to the asynchronous condition, where visual and haptic feedback were misaligned. DiscussionThese findings provide empirical evidence about the neurofunctional basis of the enhanced motor performance with simulated synthetic force feedback in the presence of teleoperation delays. The study highlights the potential for immediate haptic feedback to mitigate the adverse effects of operational delays, thereby improving the efficacy of teleoperation in critical applications.
more »
« less
Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence
Abstract In the realm of robotics and automation, robot teleoperation, which facilitates human–machine interaction in distant or hazardous settings, has surged in significance. A persistent issue in this domain is the delays between command issuance and action execution, causing negative repercussions on operator situational awareness, performance, and cognitive load. These delays, particularly in long-distance operations, are difficult to mitigate even with the most advanced computing advancements. Current solutions mainly revolve around machine-based adjustments to combat these delays. However, a notable lacuna remains in harnessing human perceptions for an enhanced subjective teleoperation experience. This paper introduces a novel approach of sensory manipulation for induced human adaptation in delayed teleoperation. Drawing from motor learning and rehabilitation principles, it is posited that strategic sensory manipulation, via altered sensory stimuli, can mitigate the subjective feeling of these delays. The focus is not on introducing new skills or adapting to novel conditions; rather, it leverages prior motor coordination experience in the context of delays. The objective is to reduce the need for extensive training or sophisticated automation designs. A human-centered experiment involving 41 participants was conducted to examine the effects of modified haptic cues in teleoperations with delays. These cues were generated from high-fidelity physics engines using parameters from robot-end sensors or physics engine simulations. The results underscored several benefits, notably the considerable reduction in task time and enhanced user perceptions about visual delays. Real-time haptic feedback, or the anchoring method, emerged as a significant contributor to these benefits, showcasing reduced cognitive load, bolstered self-confidence, and minimized frustration. Beyond the prevalent methods of automation design and training, this research underscores induced human adaptation as a pivotal avenue in robot teleoperation. It seeks to enhance teleoperation efficacy through rapid human adaptation, offering insights beyond just optimizing robotic systems for delay compensations.
more »
« less
- Award ID(s):
- 2024784
- PAR ID:
- 10540185
- Publisher / Repository:
- Nature
- Date Published:
- Journal Name:
- Scientific Reports
- Volume:
- 14
- Issue:
- 1
- ISSN:
- 2045-2322
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
IntroductionLong-distance robot teleoperation faces high latencies that pose cognitive challenges to human operators. Latency between command, execution, and feedback in teleoperation can impair performance and affect operators’ mental state. The neural underpinnings of these effects are not well understood. MethodsThis study aims to understand the cognitive impact of latency in teleoperation and the related mitigation methods, using functional Near-Infrared Spectroscopy (fNIRS) to analyze functional connectivity. A human subject experiment (n= 41) of a simulated remote robot manipulation task was performed. Three conditions were tested: no latency, with visual and haptic latency, with visual latency and no haptic latency. fNIRS and performance data were recorded and analyzed. ResultsThe presence of latency in teleoperation significantly increased functional connectivity within and between prefrontal and motor cortexes. Maintaining visual latency while providing real-time haptic feedback reduced the average functional connectivity in all cortical networks and showed a significantly different connectivity ratio within prefrontal and motor cortical networks. The performance results showed the worst performance in the all-delayed condition and best performance in no latency condition, which echoes the neural activity patterns. ConclusionThe study provides neurological evidence that latency in teleoperation increases cognitive load, anxiety, and challenges in motion planning and control. Real-time haptic feedback, however, positively influences neural pathways related to cognition, decision-making, and sensorimotor processes. This research can inform the design of ergonomic teleoperation systems that mitigate the effects of latency.more » « less
-
Robot teleoperation has become increasingly crucial for extending human capabilities in inaccessible or hazardous environments and facilitating human–robot collaboration. While significant advancements have been made in teleoperation interfaces, the success of these systems critically depends on how effectively humans can interact with and control robotic systems across diverse manipulation tasks. However, existing research primarily evaluates interfaces within specific tasks or applications, lacking systematic assessment across different manipulation scenarios. This limitation leads to suboptimal interface selection that can compromise task efficiency in critical domains and impede the collection of high-quality demonstrations for robot learning. To address these gaps, we first introduce a novel two-axis Robotic Manipulation Task Taxonomy that systematically categorizes manipulation tasks based on their fundamental control requirements: Motion Type (translation-dominant vs. rotation-dominant) and Engagement Type (rigid vs. non-rigid object interactions). We conducted a comprehensive user study (\(n=30\)) evaluating three distinct teleoperation interfaces (Gamepad, 3D Mouse, and Virtual Reality (VR) Controller) based on this taxonomy. Our results indicate that the Gamepad and 3D mouse significantly outperformed the VR Controller in task completion time across all task categories. In contrast, the VR Controller showed higher first-attempt success rates but caused significantly greater cognitive load across all NASA Task Load Index (NASA-TLX) subscales compared to the Gamepad interface, and specifically higher mental demand, physical demand, and effort compared to the 3D mouse. Our results also indicate that although prior familiarity with an interface lowered perceived workload and enhanced perceived performance, it did not translate into improvements in actual task success rates or completion times. These insights provide valuable guidelines for optimizing teleoperation interfaces based on task requirements and highlight the importance of considering both cognitive demands and user experience in interface design.more » « less
-
With the advancement of automation and robotic technologies, the teleoperation has been leveraged as a promising solution for human workers in a hazardous construction work environment. Since human operators and construction sites are separated in a distance, teleoperation requires a seamless human-machine interface, an intermediate medium, to communicate between humans and machines in construction sites. Several types of teleoperation interfaces including conventional joysticks, haptic devices, graphic user interfaces, auditory interfaces, and tactile interfaces have been developed to control and command construction robotics remotely. The ultimate goal of human-machine interfaces for remote operations is to make intuitive sensory channels that can provide and receive enough information while reducing the associated cognitive and physical load on human operators. Previously developed interfaces have tried to achieve such goals, but each interface still has challenges that should be assessed for enhancing the future teleoperation application in construction workplaces. This paper examines different human-machine interfaces for excavator teleoperation in terms of its on-site usability and intuitiveness. The capabilities of the interfaces for excavator teleoperation are evaluated based on their limitations and requirements. The outcome is expected to provide better understanding of teleoperation interfaces for excavators and guiding future directions for addressing underlying challenges.more » « less
-
Objective: Haptic perception is an important component of bidirectional human-machine interactions that allow users to better interact with their environment. Artificial haptic sensation along an individual’s hand can be evoked via noninvasive electrical nerve stimulation; however, continuous stimulation can result in adaptation of sensory perception over time. In this study, we sought to quantify the adaptation profile via the change in perceived sensation intensity over time. Approach: Noninvasive stimulation of the peripheral nerve bundles evoked haptic perception using a 2x5 electrode grid placed along the medial side of the upper arm near the median and ulnar nerves. An electrode pair that evoked haptic sensation along the forearm and hand was selected. During a trial of 110-s of continuous stimulation, a constant stimulus amplitude just below the motor threshold was delivered. Each subject was instructed to press on a force transducer producing a force amplitude matched with the perceived intensity of haptic sensation. Main Findings: A force decay (i.e., intensity of sensation) was observed in all 7 subjects. Variations in the rate of decay and the start of decay across subjects were also observed. Significance: The preliminary findings established the sensory adaptation profile of peripheral nerve stimulation. Accounting for these subject-specific profiles of adaptation can allow for more stable communication between a robotic device and a user. Additionally, sensory adaptation characterization can promote the development of new stimulation strategies that can mitigate these observed adaptations, allowing for a better and more stable human-machine interaction experience.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.1038/s41598-024-54734-1
