similar to how Nadeem might use it -as someone who was not an AI researcher or experienced prompting Large Language Models (LLMs). Fortunately, ChatGPT generated a coherent response. ChatGPT gave the names of the authors of the paper and details about the paper's publication at ACM FAccT 2020. Unfortunately, the names of the authors and publication details were made up! The confabulated author names were immediately obvious because we wrote the paper that was prompted to be summarized [20]. However, the rest of the details was extremely plausible -the paper very well could have appeared at that conference, but did not. The first author of this paper almost missed another detail in ChatGPT's summary. The original paper described a framework with two dimensions: social and technical. The generated summary claimed the framework described three dimensions: social, technical, and cultural, which, while wrong, was plausible enough that even the very author of the paper almost missed that crucial inaccuracy!
The true story above demonstrates the effortful process required to disentangle fact from fiction in GPT's output, even from someone knowledgeable of the source material. Even more notably, there was no way for our protagonist, an expert in Explainable AI, to "open" the black-box of ChatGPT and understand why it produced what it produced or where it might be faithful to the facts or prone to confabulation (also called hallucination). On the one hand, he lacked access to the internal details such as the parameters of the model. On the other hand, even if one did have access to the internal parameters of the model, given the scale and complexity of the neural architecture of such a large language model, interpreting it is unlikely to produce human-understandable and actionable information.
The field of Explainable AI (XAI) is concerned with developing techniques, concepts, and processes that can help stakeholders understand the reasons behind the AI system's decision-making [21,34].
For our purposes, we adopt a design lens in XAI that is sociotechnically-informed [12,19,34] and adopt the broad definition that an explanation is an answer to a why-question [11,30,35]. Given AI systems exist in sociotechnical settings [33,45], it takes more than just algorithmic transparency to make them explainable [23,35]. Thus, explaining what is happening "inside the black box" often requires us to also understand things "outside the black box" [12,16,32], requiring us to consider the entire AI lifecycle (vs. just the algorithm). For instance, why a facial recognition system disproportionately misclassified women of color [8] can be explained by looking at demographic compositions in the training data. A sociotechnically situated view of XAI expands the concept of explainability beyond the bounds of the algorithm [16] and positions it as a relational and audience-dependent construct instead of a model-inherent one [4,5,35,36]. Emerging work [27,40,41] showcases how a broader XAI perspective can potentially address criticisms of popular algorithm-centered XAI techniques, which can be ineffective [3,39,48] and potentially risky [29,44].
When we consider a service such as ChatGPT, GPT-4, Microsoft Copilot, Google Gemini, Claude, or Meta AI, what prospects are there for "opening" the black-box of AI? These models have hundreds of billions of parameters, all acting in conjunction to generate a distribution over possible words to choose from to build a response, word by word. If we had access to all the weights, could we interpret and explain the model? If we had access to the parameters of a model and the activation values for an input could we interpret and explain the model? In the case of the above large language models the point is moot. All these models run on servers behind APIs that do not allow inspection of the neuron activations and weights. However, even if we could access this information, the raw values of weights and activations are meaningless to most people without synthesizing some visualization or text summarization that provides a lay-understandable analysis of the internal operations of the system and how the results were generated by the system. Consider OpenAI's work on interpreting the patterns that cause individual neurons to activate [7]. How would knowing what causes neuron #2142 to activate have helped Nadeem, a non-AI expert, know how to better use ChatGPT to complete his homework? What actionable information from this neural activation pattern can a non-AI expert use meaningfully?
LLMs are increasingly being incorporated as components in systems that chain multiple processes together. In particular, Retrieval Augmented Generation (RAG) combines LLMs with web search such that a web retrieval module first retrieves relevant documents, which are then used to inform an LLM [31]. While opening the black-box LLMs may not yield actionable explanations, modular architectures afford the ability to inspect and explain how data is changed going in and out of black-box modules.
Despite the commendable progress in algorithm-centered approaches in XAI, there are significant deficiencies. Studies examining how people actually interact with AI explanations have found popular XAI techniques to be ineffective [3,39,48], potentially risky [29,44], and even obsolete in real-world deployed contexts [32]. XAI developers tend to design explanations as if people like them are going to use their systems, earning an infamous reputation of "inmates running the asylum" [35]. In fact, a majority of current deployments serve AI engineers instead of end-users whose needs are ignored [6]. This creates a gap between design expectations and reality-how developers envision the designed AI explanations to get interpreted and how users actually perceive those explanations in reality.
As Large Language Models (LLMs) become prominent, is Explainable AI -a research area in flux and its infancy -doomed to fail? No. There is hope. Before we throw in the towel, there are a few things to consider.
First, the techno-centric, algorithm-centered, discourse of XAI fails to appreciate the sociotechnical reality of AI systems. When we say "AI systems," what we very often mean to say is "Human-AI assemblages, " where the "human" part of the Human-AI assemblage is often implicit [16]. No real-world AI systems work in a vacuum. Black-boxes by themselves do not do the work -humans with blackboxes do the work [19]. Even if the human contribution to the work is to just provide an input, this is a significant contribution because AI systems are useful to people as tools. Thus, the explainability of AI systems entails explainability of the Human-AI assemblage, which has at least two components: the human (or humans) and the AI [16,20]. Thus, how can we achieve the explainability of the Human-AI assemblage by just focusing on the explainability of the AI model? XAI is therefore not just technical, it is sociotechnical. It requires more than just algorithmic transparency -more than being able to open the black box. Second, what we mean by "AI" is evolving. Compared to AI systems even five years ago, the Deep Learning systems in the Foundation Model era, such as LLMs, are much more complex, have orders of magnitude more parameters, and are running at unprecedented scales. Thus, AI as a design material is tricky and is evolving [15,20,47]. Our understanding and expectations of what it means for AI-as-design-material to be explainable should also evolve. Further, XAI techniques that focus solely on the algorithm or the model face a new challenge: it is getting increasingly hard to open the black box! As AI systems are increasingly end-user facing, those that need the explanations the most are on the other side of an AI or user interface. This is the case for the most popular Large Language Models and chatbots, and it is also the case for other types of consumer-facing systems. When the initial vision of XAI was articulated, a popular framing was to "open" the (proverbial) "blackbox" of AI [9,37], so that we could see inside of it, figure out what it was doing, why it was doing it, and if it was doing it correctly. With the advent of large language models, that ability to open the black-box is increasingly limited due to the sheer complexity of the models and the increased prevalence of models behind restrictive APIs. And even if we did manage to "open" it, we will not understand what we see.
Given AI systems are bounded by their training data, by construction, they cannot incorporate the real-world dynamics "outside" the black-box. Thus, an algorithm-centered view of XAI is-by construction-a limiting view, one that handicaps the XAI system from doing what we want to do-solve real world problems. We need a paradigm that can accommodate an expansion of the epistemic canvas-an increase of the aperture of the viewing lens-to include the sociotechnical dynamics in which XAI systems are embedded so that we can do what we set out to do -solve real world problems. This is where the domain of Human-Centered Explainable AI (HCXAI) [19] can help. HCXAI is a holistic vision of AI explainability, one that is human-centered and sociotechnical in nature. Situated as a Critical Technical Practice [1,2], it draws its conceptual DNA from critical AI studies and HCI (e.g., reflective design [13,14,42], value-sensitive design [25]). HCXAI encourages us to critically reflect and question dominant assumptions and practices of a field, such as algorithm-centered XAI. It also adopts a value-sensitive approach to both users and designers in the development of technology that challenges the status quo of a field. HCXAI encapsulates the philosophy that not everything that is important lies inside the black box of AI. Critical insights can lie outside it. Why? Because that's where the humans are.
Thinking outside the black box of AI can help us meet our goals of helping people understand and calibrate their trust in AI systems. Even if we cannot meaningfully open the black box or interpret its complexities, there are a lot of things we can do to understand and explain the system around the black box. Increasing the aperture of XAI can help us focus on the most important part: who the human(s) is (are), what they are trying to achieve in seeking an explanation, and how to design XAI techniques that meet those needs. Indeed, explanations of the sociotechnical system can offer us an important affordance: actionability [18,28,43].
At its core, actionability is about what a user can do with the information in an explanation [18]. An actionable XAI system empowers the user by increasing the space of possible informed actions to achieve their end goals. This could be understanding how to change the inputs, contesting a decision, or learning when and how to use the system more appropriately. Actionability also addresses another important question: how do we know if an XAI system is useful? There are an increasing number of reports of XAI systems that are deployed and fail to have any measurable impact on their users [3,44]. Many of these systems failed because the XAI systems were not designed with user needs in mind, such as by providing users with information they could already intuit themselves, by providing information that was onerous to verify, or by providing information that users could not use. In other words, the explanations generated by the systems were not actionable.
With the reframing around human-AI assemblages and XAI systems that place the human as the central concern, and armed with actionability as the metric for success, we now lay out three possible paths forward. This list is not meant to be exhaustive or prescriptive. It is meant to be generative by providing emerging evidence for how Human-Centered XAI (HCXAI) can address the growing needs for understanding our increasingly AI-infused world.
Most consequential AI systems are embedded in organizational environments where groups of humans interact with it. These real-world AI systems, as well as the explanations they produce, are socially-situated [22,32]. Therefore, the socio-organizational context in which these systems are used is key. Why are we not incorporating socio-organizational contexts into how we think about explainability in AI? How can we tackle the explainability of Human-AI assemblages? Enter Social Transparency (ST) a sociotechnically-informed perspective that incorporates the socio-organizational context into explaining AI-mediated decision-making [16]. Social transparency allows us to augment the explainability of a human-AI assemblage without necessarily changing anything about the AI model. Social transparency allows one to annotate an output or behavior from an AI system with the 4W who did what, when and why. These annotations are shared between others using the system. They allow users to see whether and why others have accepted or rejected an AI's output. Social transparency does two important things: first, it challenges the dominant narrative of algorithm-centered notions of XAI; second, it expands our understanding of XAI beyond technical transparency by illustrating how adding social context can help people make better, more actionable decisions with AI systems.
Imagine the following scenario (Figure 3): Aziz is a software seller trying to use a powerful AI-based pricing tool to do something consequential: offer the right price to a client company. The AI suggests a price. Moreover, its suggestion has technical transparency -it explains its recommendation by showing Aziz the top features it considered, such as sales quota goals, comparative pricing with other clients, and costs. Confident with the AI's recommendation, Aziz makes a bid, but the client finds the price too high and walks out.
Despite an accurate AI model and the presence of technical transparency, why did the bid fail? There could be algorithmic reasons for it. But might also be relevant contextual factors outside the box that can help explain why the bid failed. Perhaps the history between Aziz and the client that was not honored? Or maybe there were external events that happened since the model was trained, such as a pandemic-induced budgetary crisis. Now imagine that Aziz could see that more than 65% of his peers rejected the AI's pricing recommendation (Block 2 in Fig. 3). Or, what if Aziz knew that Jess, a director in the company, sold the product at a loss due to pandemic-related budgetary cuts?(Block 5 in Fig. 3)
This peripheral vision of who did what, when and why -called the 4W -are the constitutive design elements of Social Transparency that can encode relevant socio-organizational context. The benefit of taking a holistic approach to explainability is clear: a study of realworld AI users in sales, cybersecurity, and healthcare found that social transparency, in the form of the 4W, helped people calibrate their trust in the AI's performance, provide actionable information for AI contestability and robust decision-making, and the organizational context made visible enabled better collective actions in the organization and strengthened the human-AI assemblages [16].
By incorporating the socio-organizational context, Social Transparency makes our understanding of XAI more holistic, representing the Human-AI assemblage more realistically than a purely algorithm-centered XAI view. We should note that Social Transparency is agnostic to whether an AI system is black-boxed or not. As long as there is an AI-based recommendation or decision, we can attach 4W -the socio-organizational context -to it. In a completely black-boxed AI system, there will not be any technical transparency. However, the 4W can add transparency to the social side of the Human-AI assemblage.
If the black box cannot be cracked open in any meaningful sense, there is another possibility: incorporate explainability around the edges of the black box to foster a better functional understanding in the user [38] such that it fosters actionability. One of the original formulations of rationale generation [21] postulated that there was no need to know how a black box worked as long as we could learn how to give actionable advice about the black box by looking at its inputs and outputs. It was philosophically grounded in Fodor's work on Language of Thought [24]: how is it that, despite not having a 1-1 neural correlate of thought, humans can effectively communicate by translating their thoughts into words? For Human-AI interaction, even if the exact mechanism of the (artificial) neural correlate of AI's thought was not known to the human, as long as actionable information is present in the explanation from an AI agent, the Human-AI interaction can proceed. In short, explanations that do not directly access the model can still generate actionable information.
In the case of large language models, the actionable information is whether any particular input is likely to produce a reliable response that can be trusted. Large language models might be generally capable at many tasks such as question-answering, they are not infallible, and it is always possible for a user to ask a question that results in a confabulation (also called a "hallucination") that the user is unable to vet. In this case, we can directly use the API to probe how it responds to particular stimuli [46]. It is proposed that an XAI system can decompose the original, human authored question into a series of more fine-grained, related questions that provide more opportunities for the model to confabulate responses if it is not competent at the original question. These sub-questions can be selected to be easier for the user to vet. Generating questions to challenge an LLM has been demonstrated to increase users' ability to determine whether the answer should be trusted or not.
Seams: Seamful XAI
No AI system is perfect. Mistakes are inevitable. Breakdowns in AI systems often occur when the assumptions we make in design and development do not hold true when they are deployed in the real-world. For example, an AI system can fail when it is trained on data from North America but deployed in South Asia, especially when the end user is unaware of this infrastructural mismatch. These mismatches between design assumptions and real-world usage are called seams [17]. Handling the mistakes from AI systems is hard, especially when the AI's decision-making is hidden or black-boxed. Although black-boxing AI systems can make the user experience seamless and easy to use, concealing the seams can lead to downstream harms for end-users, such as uncritical AI acceptance. What can we do differently? How do we move beyond seamless AI? And what can we gain by doing so? Seamful XAI is a design lens that incorporates the principles of seamful design [10] to augment explainability and user agency. A classic example of seamful design is a "seamful map" of WiFi coverage in your home. If you know the WiFi's dead zones in your home, you will be able to best use it because you can then avoid.Without revealing the seams, users can have reasonable expectations of perfect WiFi. The map makes the seams in the WiFi's infrastructure visible to users, which allows them to recalibrate their expectations and behavior. A seamful design principle asks us to leverage the weakness in opportunistic ways [26].
Unlike seamlessness, seamful design does not aim to hide the infrastructure. Rather, it puts the infrastructure and all its imperfections front and center. Seamful design helps us recognize and grapple with the complex infrastructures systems reside in. Conversely, seamless design ideals risks making the labor it takes to make the system work invisible (e.g., datawork, ghostwork, maintenance work). And, as invisible work is invariably unaccounted for and unappreciated, workers who conduct this work will feel undervalued or invisible. Seamfulness embraces the imperfect reality of spaces we inhabit and makes the most out of it.
In the context of AI, seams can be conceptualized as mismatches, gaps, or cracks in assumptions between the world of how AI systems are designed and the world of how AI systems are used in practice. Seamful XAI seeks to empower users with information that augments their agency by identifying gaps between ideal design assumptions and reality.
At the heart of Seamful XAI are four observations:
(1) Seams are inevitable, arising from the integration of heterogeneous sociotechnical components during technology deployments.
(2) Seams are revealed through system breakdowns.
(3) Instead of treating seams as problematic negatives to be erased, they can be used strategically to calibrate users' reliance and understanding of an AI system. (4) The goal of this strategic revelation (and concealment) is to support user agency (actionability, contestability, and appropriation).
Seamful XAI Design Process: Let's review the design process proposed by [17].
The first step of the process begins with generating "breakdowns." Breakdowns are answers to the question, "what could go wrong when this technology gets deployed?" Answers could include technology failures, unfair treatment of groups, inducing over-reliance, or deskilling.
The second step is around anticipating and crafting seams, which is done in three parts. First (2A in the diagram), we ask: "what might we (as developers, designers, researchers, etc.) do to make the breakdown happen?" While this question might seem counter-intuitive, it allows us to systematically prevent breakdowns by understanding their causes. This step inverts the problem and makes it a goal directed task, which is important to generate concrete outcomes instead of open-ended problems. Next (2B), we try to anticipate the reasons for the breakdown (the seams) in the appropriate stage in the AI's lifecycle (the colored boxes numbered 1-6 in Fig. 5). Finally (2C), we craft the seam by thinking about the gap between the ideal expectation and the reality of use.
The final step involves using the seams generated in step 2 in a way to empower user agency and explainability. Here (3A), we ask: given our end goal, which seams do we show and which do we hide (e.g. strategic revelation and concealment)? The revealed seams (3B) should empower users through better explainability. This step of the Seamful XAI process is a major differentiator from other Responsible AI processes: unlike most processes that stop at identifying gaps, this one goes beyond. It not only uncovers the gaps but also utilizes them as avenues to support users (for more details, refer to [17]).
A co-designing study [17] with 43 real-world AI users found three beneficial elements of Seamful XAI:
• It enhances explainability by helping stakeholders reveal the AI's blind spots, highlight its fallibility, and showcase the strengths and weaknesses of the system, which can calibrate reliance in AI systems. • It augments user agency by providing peripheral vision of the AI's blind spots. Seamful information expands the action space of what users can do. Information in seams can convert "unknown unknowns" to "known unknowns, " which can empower users to know "where" to start an investigation. • It is a resourceful way to not just reveal seams but also anticipate and mitigate harms from AI systems.
We began with the provocation: With the advent of Foundation Models & Large Language Models like ChatGPT, is "opening the black-box" still a reasonable and achievable goal for XAI? Do we need to shift our perspectives? Yes. The proverbial "black-box" of AI has evolved, and so should our expectations on how to make it explainable. As the box becomes more opaque and harder to "open, " the human side of the Human-AI assemblage remains as a fruitful space to explore. In the most extreme case, the human side may be all there is left to explore.
There are four important lessons from Human-centered XAI that can inform the shift in our XAI expectations.
(1) First, the human-centered XAI perspective takes a pragmatic and resourceful view of explainability, especially if black boxes are expected to persist. By considering the actions afforded to the user by the explanations, HCXAI centers the focus on the human, ensuring AI augments human abilities rather than replace them. (2) Second, explainability is not only achieved by looking inside the black box through mechanistic descriptions of how an algorithm works. Actionability can be achieved by exploring explainability outside and around the edges of the black box because human-centered XAI takes a more expansive view of what it means to provide insights into a black box that can afford a wider range of actions. (3) Third, explicitly treating AI systems as human-AI assemblages means focusing on explainability of the assemblage, not just the AI. This widened perspective opens up avenues for not just factoring in who is interacting with the black box, but also how human teams can work together -directly or indirectly -to contextualize a dynamically changing realworld AI behavior. (4) Fourth, seamful XAI turns the disadvantages and weaknesses of an AI system into advantages. The gaps between user expectations and AI capabilities are exactly the gaps that explanations address. Instead of hiding those gaps to create seamless experiences, seamful XAI leverages these gaps in an opportunistic manner to augment explainability and user agency. As we reload our expectations on XAI, we invite you to do what HCXAI asks us to do: centering the design and evaluation around the human. This positioning can reveal unmet needs that must be addressed while avoiding the costly mistake of building XAI systems that do not make a difference. While there have been many examples of XAI systems that have failed to have the intended impact of users, it is often the case that these tenets of HCXAI were overlooked. XAI is a relatively young field of research that has yet to find its footing, even as the landscape of black box AI systems is rapidly evolving. It is not yet time to give up hope on XAI. Instead, we invite you to adopt critical reflection and value-sensitivity into XAI research and evaluation, making it human-centered.
Will Human-centered XAI solve all our problems? No, but it will help us ask the right questions.