An official website of the United States government
Assistive robotics is an important subarea of robotics that focuses on the well-being of people with disabilities. A robotic guide dog is an assistive quadruped robot for assisting visually impaired people in obstacle avoidance and navigation. Enabling language capabilities on robotic guide dogs goes beyond naively adding an existing dialog system onto a mobile robot. The novel challenges include grounding language to the dynamically changing environment and improving spatial awareness for the human handler. To address those challenges, we develop a novel dialog system for robotic guide dogs that uses large language models to verbalize both navigational plans and scenes. The goal is to enable verbal communication for collaborative decision-making within the handler-robot team. In experiments, we performed a human study to evaluate different verbalization strategies, and a simulation study to evaluate the efficiency and accuracy in navigation tasks.
more »
« less
This content will become publicly available on March 16, 2027
GuideNav: User-Informed Development of a Vision-Only Robotic Navigation Assistant for Blind Travelers
While commendable progress has been made in user-centric research on mobile assistive systems for blind and low-vision (BLV) individuals, references that directly inform robot navigation design remain rare. To bridge this gap, we conducted a comprehensive human study involving interviews with 26 guide dog handlers, four white cane users, nine guide dog trainers, and one O&M trainer, along with 15+ hours of observing guide dog–assisted walking. After de-identification, we open-sourced the dataset to promote human-centered development and informed decision-making for assistive systems for BLV people. Building on insights from this formative study, we developed GuideNav, a vision-only, teach-and-repeat navigation system. Inspired by how guide dogs are trained and assist their handlers, GuideNav autonomously repeats a path demonstrated by a sighted person using a robot. Specifically, the system constructs a topological representation of the taught route, integrates visual place recognition with temporal filtering, and employs a relative pose estimator to compute navigation actions---all without relying on costly, heavy, power-hungry sensors such as LiDAR. In field tests, GuideNav consistently achieved kilometer-scale route following across five outdoor environments, maintaining reliability despite noticeable scene variations between teach and repeat runs. A user study with 3 guide dog handlers and 1 guide dog trainer further confirmed the system’s feasibility, marking, to our knowledge, the first demonstration of a quadruped mobile system guiding a route in a manner comparable to guide dogs.
more »
« less
- Award ID(s):
- 2427788
- PAR ID:
- 10699726
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9798400721281
- Page Range / eLocation ID:
- 1129 to 1139
- Format(s):
- Medium: X
- Location:
- Edinburgh Scotland UK
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Social interactions often rely on the interpretation of visual cues. The inaccessibility of these nonverbal signals can relegate blind and low vision (BLV) people to the social periphery and restrict their independence in forming connections with others. Many assistive technologies for BLV people focus on spatial navigation or object recognition, but there is a gap in supporting “social wayfinding”: the capacity to perceive and navigate human interaction dynamics. To address this, we explore the design space of a wearable system that provides BLV users with real-time details about the social environment. The system, SocialCue, serves as a technology probe to explore BLV people’s preferences for social navigation assistance. We conducted a two-phase formative study which identified four social attributes that the system should communicate: identity, social availability, facial expression, and physical descriptions including clothing and hairstyle. We describe the implementation of SocialCue and close by discussing our future evaluation plans.more » « less
-
This paper proposes an AR-based real-time mobile system for assistive indoor navigation with target segmentation (ARMSAINTS) for both sighted and blind or low-vision (BLV) users to safely explore and navigate in an indoor environment. The solution comprises four major components: graph construction, hybrid modeling, real-time navigation and target segmentation. The system utilizes an automatic graph construction method to generate a graph from a 2D floorplan and the Delaunay triangulation-based localization method to provide precise localization with negligible error. The 3D obstacle detection method integrates the existing capability of AR with a 2D object detector and a semantic target segmentation model to detect and track 3D bounding boxes of obstacles and people to increase BLV safety and understanding when traveling in the indoor environment. The entire system does not require the installation and maintenance of expensive infrastructure, run in real-time on a smartphone, and can easily adapt to environmental changes.more » « less
-
Blind and low-vision (BLV) users face significant barriers when encountering charts in digital content. While recent advances in large language models (LLMs) show promise for chart interpretation, existing accessible systems rely on pre-structured data formats, limiting applicability to real-world charts. We present GraphWhisper, a conversational system enabling BLV users to explore charts through natural language without requiring pre-structured data. GraphWhisper employs enhanced chart-specific prompting strategies that guide LLMs to systematically extract and analyze data directly from chart images, achieving 98.1% accuracy on visualization literacy benchmarks. A user study with 15 BLV participants demonstrated 94% question-answering accuracy across 185 questions and exceptional satisfaction scores (4.87-5.0 on 5-point scales), validating the system’s practical effectiveness for everyday chart accessibility.more » « less
-
Blind and low-vision (BLV) people rely on GPS-based systems for outdoor navigation. GPS's inaccuracy, however, causes them to veer off track, run into obstacles, and struggle to reach precise destinations. While prior work has made precise navigation possible indoors via hardware installations, enabling this outdoors remains a challenge. Interestingly, many outdoor environments are already instrumented with hardware such as street cameras. In this work, we explore the idea of repurposing existing street cameras for outdoor navigation. Our community-driven approach considers both technical and sociotechnical concerns through engagements with various stakeholders: BLV users, residents, business owners, and Community Board leadership. The resulting system, StreetNav, processes a camera's video feed using computer vision and gives BLV pedestrians real-time navigation assistance. Our evaluations show that StreetNav guides users more precisely than GPS, but its technical performance is sensitive to environmental occlusions and distance from the camera. We discuss future implications for deploying such systems at scalemore » « less
