An official website of the United States government
For people with visual impairments, photography is essential in identifying objects through remote sighted help and image recognition apps. This is especially the case for teachable object recognizers, where recognition models are trained on user's photos. Here, we propose real-time feedback for communicating the location of an object of interest in the camera frame. Our audio-haptic feedback is powered by a deep learning model that estimates the object center location based on its proximity to the user's hand. To evaluate our approach, we conducted a user study in the lab, where participants with visual impairments (N=9) used our feedback to train and test their object recognizer in vanilla and cluttered environments. We found that very few photos did not include the object (2% in the vanilla and 8% in the cluttered) and the recognition performance was promising even for participants with no prior camera experience. Participants tended to trust the feedback even though they know it can be wrong. Our cluster analysis indicates that better feedback is associated with photos that include the entire object. Our results provide insights into factors that can degrade feedback and recognition performance in teachable interfaces.
more »
« less
Blind Users Accessing Their Training Images in Teachable Object Recognizers
Teachable object recognizers provide a solution for a very practical need for blind people – instance level object recognition. They assume one can visually inspect the photos they provide for training, a critical and inaccessible step for those who are blind. In this work, we engineer data descriptors that address this challenge. They indicate in real time whether the object in the photo is cropped or too small, a hand is included, the photos is blurred, and how much photos vary from each other. Our descriptors are built into open source testbed iOS app, called MYCam. In a remote user study in (N = 12) blind participants’ homes, we show how descriptors, even when error-prone, support experimentation and have a positive impact in the quality of training set that can translate to model performance though this gain is not uniform. Participants found the app simple to use indicating that they could effectively train it and that the descriptors were useful. However, many found the training being tedious, opening discussions around the need for balance between information, time, and cognitive load.
more »
« less
- Award ID(s):
- 1816380
- PAR ID:
- 10344780
- Date Published:
- Journal Name:
- ACM SIGACCESS Conference on Computers and Accessibility (ASSETS)
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Chua Chin Heng, Matthew (Ed.)Early Childhood Caries (ECC) is the most common childhood disease worldwide and a health disparity among underserved children. ECC is preventable and reversible if detected early. However, many children from low-income families encounter barriers to dental care. An at-home caries detection technology could potentially improve access to dental care regardless of patients’ economic status and address the overwhelming prevalence of ECC. Our team has developed a smartphone application (app), AICaries, that uses artificial intelligence (AI)-powered technology to detect caries using children’s teeth photos. We used mixed methods to assess the acceptance, usability, and feasibility of the AICaries app among underserved parent-child dyads. We conducted moderated usability testing (Step 1) with ten parent-child dyads using "Think-aloud" methods to assess the flow and functionality of the app and analyze the data to refine the app and procedures. Next, we conducted unmoderated field testing (Step 2) with 32 parent-child dyads to test the app within their natural environment (home) over two weeks. We administered the System Usability Scale (SUS) and conducted semi-structured individual interviews with parents and conducted thematic analyses. AICaries app received a 78.4 SUS score from the participants, indicating an excellent acceptance. Notably, the majority (78.5%) of parent-taken photos of children’s teeth were satisfactory in quality for detection of caries using the AI app. Parents suggested using community health workers to provide training to parents needing assistance in taking high quality photos of their young child’s teeth. Perceived benefits from using the AICaries app include convenient at-home caries screening, informative on caries risk and education, and engaging family members. Data from this study support future clinical trial that evaluates the real-world impact of using this innovative smartphone app on early detection and prevention of ECC among low-income children.more » « less
-
BackgroundReminiscence, a therapy that uses stimulating materials such as old photos and videos to stimulate long-term memory, can improve the emotional well-being and life satisfaction of older adults, including those who are cognitively intact. However, providing personalized reminiscence therapy can be challenging for caregivers and family members. ObjectiveThis study aimed to achieve three objectives: (1) design and develop the GoodTimes app, an interactive multimodal photo album that uses artificial intelligence (AI) to engage users in personalized conversations and storytelling about their pictures, encompassing family, friends, and special moments; (2) examine the app’s functionalities in various scenarios using use-case studies and assess the app’s usability and user experience through the user study; and (3) investigate the app’s potential as a supplementary tool for reminiscence therapy among cognitively intact older adults, aiming to enhance their psychological well-being by facilitating the recollection of past experiences. MethodsWe used state-of-the-art AI technologies, including image recognition, natural language processing, knowledge graph, logic, and machine learning, to develop GoodTimes. First, we constructed a comprehensive knowledge graph that models the information required for effective communication, including photos, people, locations, time, and stories related to the photos. Next, we developed a voice assistant that interacts with users by leveraging the knowledge graph and machine learning techniques. Then, we created various use cases to examine the functions of the system in different scenarios. Finally, to evaluate GoodTimes’ usability, we conducted a study with older adults (N=13; age range 58-84, mean 65.8 years). The study period started from January to March 2023. ResultsThe use-case tests demonstrated the performance of GoodTimes in handling a variety of scenarios, highlighting its versatility and adaptability. For the user study, the feedback from our participants was highly positive, with 92% (12/13) reporting a positive experience conversing with GoodTimes. All participants mentioned that the app invoked pleasant memories and aided in recollecting loved ones, resulting in a sense of happiness for the majority (11/13, 85%). Additionally, a significant majority found GoodTimes to be helpful (11/13, 85%) and user-friendly (12/13, 92%). Most participants (9/13, 69%) expressed a desire to use the app frequently, although some (4/13, 31%) indicated a need for technical support to navigate the system effectively. ConclusionsOur AI-based interactive photo album, GoodTimes, was able to engage users in browsing their photos and conversing about them. Preliminary evidence supports GoodTimes’ usability and benefits cognitively intact older adults. Future work is needed to explore its potential positive effects among older adults with cognitive impairment.more » « less
-
Riding public transit can be confusing for everyone, especially in an unfamiliar environment. One needs to figure out which transportation lines to take to reach a destination, when and where to catch a bus or a train, when to exit, and how to negotiate transfers. For those with sensorial or cognitive disa- bilities, these problems become even more daunting. Several technological ap- proaches have been proposed to facilitate use of public transit for everyone. For any assistive technology to be successful, though, it is imperative that it is de- veloped from the ground up with a clear understanding of the intended users’ needs and requirements, and possibly with a direct participation of these users throughout the project lifecycle. In this study, we conduct a focus group with blind participants, designed to highlight the main issues, problems, and limita- tions with the current transit system in our local area as well as the perception of the participants our proposed RouteMe2 technology. We found two core categories of issues faced by blind travelers: (1) spatial/location awareness, and (2) temporal/ time awareness. Configurability and accessibility were the most desired features requested for a new transit information app.more » « less
-
People often do not receive the reactions they desire when they use social networking sites to share data collected through personal tracking tools like Fitbit, Strava, and Swarm. Although some people have found success sharing with close connections or in finding online communities, most audiences express limited interest and rarely respond. We report on findings from a human-centered design process undertaken to examine how tracking tools can better support people in telling their story using their data. 23 formative interviews contribute design goals for telling stories of accomplishment, including a need to include relevant data. We implement these goals in Yarn, a mobile app that offers structure for telling stories of accomplishment around training for running races and completing Do-It-Yourself projects. 21 participants used Yarn for 4 weeks across two studies. Although Yarn’s structure led some participants to include more data or explanation in the moments they created, many felt like the structure prevented them from telling their stories in the way they desired. In light of participant use, we discuss additional challenges to using personal data to inform and target an interested audiencemore » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3517428.3544824
