The rise of complex AI systems in healthcare and other sectors has led to a growing area of research called
Explainable AI (XAI) designed to increase transparency. In this area, quantitative and qualitative studies focus on
improving user trust and task performance by providing system- and prediction-level XAI features. We analyze
stakeholder engagement events (interviews and workshops) on the use of AI for kidney transplantation. From this
we identify themes which we use to frame a scoping literature review on current XAI features. The stakeholder
engagement process lasted over nine months covering three stakeholder group's workflows, determining where
AI could intervene and assessing a mock XAI decision support system. Based on the stakeholder engagement, we identify four major themes relevant to designing XAI systems – 1) use of AI predictions, 2) information included
in AI predictions, 3) personalization of AI predictions for individual differences, and 4) customizing AI predictions
for specific cases. Using these themes, our scoping literature review finds that providing AI predictions
before, during, or after decision-making could be beneficial depending on the complexity of the stakeholder's
task. Additionally, expert stakeholders like surgeons prefer minimal to no XAI features, AI prediction, and uncertainty estimates for easy use cases. However, almost all stakeholders prefer to have optional XAI features to
review when needed, especially in hard-to-predict cases. The literature also suggests that providing both system and prediction-level information is necessary to build the user's mental model of the system appropriately.
Although XAI features improve users' trust in the system, human-AI team performance is not always enhanced.
Overall, stakeholders prefer to have agency over the XAI interface to control the level of information based on
their needs and task complexity. We conclude with suggestions for future research, especially on customizing XAI
features based on preferences and tasks.
more »
« less
TRUST XAI: Model-Agnostic Explanations for AI With a Case Study on IIoT Security
Despite AI’s significant growth, its “black box”
nature creates challenges in generating adequate trust. Thus, it
is seldom utilized as a standalone unit in high-risk applications.
Explainable AI (XAI) has emerged to help with this problem.
Designing effectively fast and accurate XAI is still challenging,
especially in numerical applications. We propose a novel XAI
model named Transparency Relying Upon Statistical Theory
(TRUST) for XAI. TRUST XAI models the statistical behavior of
the underlying AI’s outputs. Factor analysis is used to transform
the input features into a new set of latent variables. We use
mutual information to rank these parameters and pick only the
most influential ones on the AI’s outputs and call them “representatives” of the classes. Then we use multi-model Gaussian
distributions to determine the likelihood of any new sample
belonging to each class. The proposed technique is a surrogate
model that is not dependent on the type of the underlying AI.
TRUST is suitable for any numerical application. Here, we use
cybersecurity of the industrial internet of things (IIoT) as an
example application. We analyze the performance of the model
using three different cybersecurity datasets, including “WUSTLIIoT”, “NSL-KDD”, and “UNSW”. We also show how TRUST
is explained to the user. The TRUST XAI provides explanations
for new random samples with an average success rate of 98%.
Also, the advantages of our model over another popular XAI
model, LIME, including performance, speed, and the method of
explainability are evaluated.
more »
« less
- Award ID(s):
- 1718929
- NSF-PAR ID:
- 10326532
- Date Published:
- Journal Name:
- IEEE Internet of Things Journal
- ISSN:
- 2372-2541
- Page Range / eLocation ID:
- 1 to 1
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Automated Driving Systems (ADS), like many other systems people use today, depend on successful Artificial Intelligence (AI) for safe roadway operations. In ADS, an essential function completed by AI is the computer vision techniques for detecting roadway signs by vehicles. The AI, though, is not always reliable and sometimes requires the human’s intelligence to complete a task. For the human to collaborate with the AI, it is critical to understand the human’s perception of AI. In the present study, we investigated how human drivers perceive the AI’s capabilities in a driving context where a stop sign is compromised and how knowledge, experience, and trust related to AI play a role. We found that participants with more knowledge of AI tended to trust AI more, and those who reported more experience with AI had a greater understanding of AI. Participants correctly deduced that a maliciously manipulated stop sign would be more difficult for AI to identify. Nevertheless, participants still overestimated the AI’s ability to recognize the malicious stop sign. Our findings suggest that the public do not yet have a sufficiently accurate understanding of specific AI systems, which leads them to over-trust the AI in certain conditions.
-
Abstract Despite the increasingly successful application of neural networks to many problems in the geosciences, their complex and nonlinear structure makes the interpretation of their predictions difficult, which limits model trust and does not allow scientists to gain physical insights about the problem at hand. Many different methods have been introduced in the emerging field of eXplainable Artificial Intelligence (XAI), which aims at attributing the network’s prediction to specific features in the input domain. XAI methods are usually assessed by using benchmark datasets (such as MNIST or ImageNet for image classification). However, an objective, theoretically derived ground truth for the attribution is lacking for most of these datasets, making the assessment of XAI in many cases subjective. Also, benchmark datasets specifically designed for problems in geosciences are rare. Here, we provide a framework, based on the use of additively separable functions, to generate attribution benchmark datasets for regression problems for which the ground truth of the attribution is known a priori. We generate a large benchmark dataset and train a fully connected network to learn the underlying function that was used for simulation. We then compare estimated heatmaps from different XAI methods to the ground truth in order to identify examples where specific XAI methods perform well or poorly. We believe that attribution benchmarks as the ones introduced herein are of great importance for further application of neural networks in the geosciences, and for more objective assessment and accurate implementation of XAI methods, which will increase model trust and assist in discovering new science.more » « less
-
Recent years have witnessed the growing literature in empirical evaluation of explainable AI (XAI) methods. This study contributes to this ongoing conversation by presenting a comparison on the effects of a set of established XAI methods in AI-assisted decision making. Based on our review of previous literature, we highlight three desirable properties that ideal AI explanations should satisfy — improve people’s understanding of the AI model, help people recognize the model uncertainty, and support people’s calibrated trust in the model. Through three randomized controlled experiments, we evaluate whether four types of common model-agnostic explainable AI methods satisfy these properties on two types of AI models of varying levels of complexity, and in two kinds of decision making contexts where people perceive themselves as having different levels of domain expertise. Our results demonstrate that many AI explanations do not satisfy any of the desirable properties when used on decision making tasks that people have little domain expertise in. On decision making tasks that people are more knowledgeable, the feature contribution explanation is shown to satisfy more desiderata of AI explanations, even when the AI model is inherently complex. We conclude by discussing the implications of our study for improving the design of XAI methods to better support human decision making, and for advancing more rigorous empirical evaluation of XAI methods.more » « less
-
Abstract Recently artificial intelligence (AI) and machine learning (ML) models have demonstrated remarkable progress with applications developed in various domains. It is also increasingly discussed that AI and ML models and applications should be transparent, explainable, and trustworthy. Accordingly, the field of Explainable AI (XAI) is expanding rapidly. XAI holds substantial promise for improving trust and transparency in AI-based systems by explaining how complex models such as the deep neural network (DNN) produces their outcomes. Moreover, many researchers and practitioners consider that using provenance to explain these complex models will help improve transparency in AI-based systems. In this paper, we conduct a systematic literature review of provenance, XAI, and trustworthy AI (TAI) to explain the fundamental concepts and illustrate the potential of using provenance as a medium to help accomplish explainability in AI-based systems. Moreover, we also discuss the patterns of recent developments in this area and offer a vision for research in the near future. We hope this literature review will serve as a starting point for scholars and practitioners interested in learning about essential components of provenance, XAI, and TAI.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1109/JIOT.2021.3122019
