Despite the availability of numerous automatic accessibility testing solutions, web accessibility issues persist on many websites. Moreover, there is a lack of systematic evaluations of the efficacy of current accessibility testing tools. To address this gap, we present the first mutation analysis framework, called Ma11y, designed to assess web accessibility testing tools. Ma11y includes 25 mutation operators that intentionally violate various accessibility principles and an automated oracle to determine whether a mutant is detected by a testing tool. Evaluation on real-world websites demonstrates the practical applicability of the mutation operators and the framework’s capacity to assess tool performance. Our results demonstrate that the current tools cannot identify nearly 50% of the accessibility bugs injected by our framework, thus underscoring the need for the development of more effective accessibility testing tools. Finally, the framework’s accuracy and performance attest to its potential for seamless and automated application in practical settings
more »
« less
This content will become publicly available on June 19, 2026
Enhancing Web Accessibility: Automated Detection of Issues with Generative AI
Websites are integral to people’s daily lives, with billions in use today. However, due to limited awareness of accessibility and its guidelines, developers often release web apps that are inaccessible to people with disabilities, who make up around 16% of the global population. To ensure a baseline of accessibility, software engineers rely on automated checkers that assess a webpage’s compliance based on predefined rules. Unfortunately, these tools typically cover only a small subset of accessibility guidelines and often overlook violations that require a semantic understanding of the webpage. The advent of generative AI, known for its ability to comprehend textual and visual content, has created new possibilities for detecting accessibility violations. We began by studying the most widely used guideline, WCAG, to determine the testable success criteria that generative AI could address. This led to the development of an automated tool called GenA11y, which extracts elements from a page related to each success criterion and inputs them into an LLM prompted to detect accessibility issues on the web. Evaluations of GenA11y showed its effectiveness, with a precision of 94.5% and a recall of 87.61%. Additionally, when tested on real websites, GenA11y identified an average of eight more types of accessibility violations than the combination of existing tools.
more »
« less
- PAR ID:
- 10618081
- Publisher / Repository:
- ACM International Conference on the Foundations of Software Engineering (FSE 2025)
- Date Published:
- Journal Name:
- Proceedings of the ACM on Software Engineering
- Volume:
- 2
- Issue:
- FSE
- ISSN:
- 2994-970X
- Page Range / eLocation ID:
- 2264 to 2287
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The rapid adoption of generative AI in software development has impacted the industry, yet its efects on developers with visual impairments remain largely unexplored. To address this gap, we used an Activity Theory framework to examine how developers with visual impairments interact with AI coding assistants. For this purpose, we conducted a study where developers who are visually impaired completed a series of programming tasks using a generative AI coding assistant. We uncovered that, while participants found the AI assistant benefcial and reported signifcant advantages, they also highlighted accessibility challenges. Specifcally, the AI coding assistant often exacerbated existing accessibility barriers and introduced new challenges. For example, it overwhelmed users with an excessive number of suggestions, leading developers who are visually impaired to express a desire for “AI timeouts.” Additionally, the generative AI coding assistant made it more difcult for developers to switch contexts between the AI-generated content and their own code. Despite these challenges, participants were optimistic about the potential of AI coding assistants to transform the coding experience for developers with visual impairments. Our fndings emphasize the need to apply activity-centered design principles to generative AI assistants, ensuring they better align with user behaviors and address specifc accessibility needs. This approach can enable the assistants to provide more intuitive, inclusive, and efective experiences, while also contributing to the broader goal of enhancing accessibility in software developmentmore » « less
-
AGGA (Academic Guidelines for Generative AIs) is a dataset of 80 academic guidelines for the usage of generative AIs and large language models in academia, selected systematically and collected from official university websites across six continents. Comprising 181,225 words, the dataset supports natural language processing tasks such as language modeling, sentiment and semantic analysis, model synthesis, classification, and topic labeling. It can also serve as a benchmark for ambiguity detection and requirements categorization. This resource aims to facilitate research on AI governance in educational contexts, promoting a deeper understanding of the integration of AI technologies in academia.more » « less
-
Artificial intelligence (AI) leverages mathematical algorithms to emulate human cognitive abilities, leading to a transformative impact on the education sector. Educators are at the front lines of implementing AI in the classroom. Recent scientific studies demonstrate the capacity of AI, particularly generative models like ChatGPT, to reshape various aspects of education. In a recent study, we showcased that the integration of both artificial intelligence, specifically ChatGPT, and interactive learning activities significantly enhances the engagement levels of STEM students enrolled in a General Biology course. Furthermore, this combined approach not only boosts student engagement but also demonstrates an improvement in their overall performance within the course. Building on preliminary studies, the objective of this review article is to delineate the diverse applications of generative AI in education. To achieve this objective, we conducted a thorough search across scientific databases, including Google Scholar, Science Direct, government websites, and other resources, to collect relevant papers. Our findings underscore the contributions of generative AI, exemplified by ChatGPT, in enabling students to generate innovative text for written assignments, providing personalized feedback, facilitating adaptive learning, enhancing accessibility to education by eliminating barriers for individuals with disabilities, and supporting research endeavors.more » « less
-
Artificial intelligence (AI) leverages mathematical algorithms to emulate human cognitive abilities, leading to a transformative impact on the education sector. Educators are at the front lines of implementing AI in the classroom. Recent scientific studies demonstrate the capacity of AI, particularly generative models like ChatGPT, to reshape various aspects of education. In a recent study, we showcased that the integration of both artificial intelligence, specifically ChatGPT, and interactive learning activities significantly enhances the engagement levels of STEM students enrolled in a General Biology course. Furthermore, this combined approach not only boosts student engagement but also demonstrates an improvement in their overall performance within the course. Building on preliminary studies, the objective of this review article is to delineate the diverse applications of generative AI in education. To achieve this objective, we conducted a thorough search across scientific databases, including Google Scholar, Science Direct, government websites, and other resources, to collect relevant papers. Our findings underscore the contributions of generative AI, exemplified by ChatGPT, in enabling students to generate innovative text for written assignments, providing personalized feedback, facilitating adaptive learning, enhancing accessibility to education by eliminating barriers for individuals with disabilities, and supporting research endeavors.more » « less
An official website of the United States government
