More Like this

1. Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning

   https://doi.org/10.1145/3490099.3511138  

   Gajos, Krzysztof Z.; Mamykina, Lena (March 2022, 27th International Conference on Intelligent User Interfaces (IUI ’22))

   When people receive advice while making difficult decisions, they often make better decisions in the moment and also increase their knowledge in the process. However, such incidental learning can only occur when people cognitively engage with the information they receive and process this information thoughtfully. How do people process the information and advice they receive from AI, and do they engage with it deeply enough to enable learning? To answer these questions, we conducted three experiments in which individuals were asked to make nutritional decisions and received simulated AI recommendations and explanations. In the first experiment, we found that when people were presented with both a recommendation and an explanation before making their choice, they made better decisions than they did when they received no such help, but they did not learn. In the second experiment, participants first made their own choice, and only then saw a recommendation and an explanation from AI; this condition also resulted in improved decisions, but no learning. However, in our third experiment, participants were presented with just an AI explanation but no recommendation and had to arrive at their own decision. This condition led to both more accurate decisions and learning gains. We hypothesize that learning gains in this condition were due to deeper engagement with explanations needed to arrive at the decisions. This work provides some of the most direct evidence to date that it may not be sufficient to provide people with AI-generated recommendations and explanations to ensure that people engage carefully with the AI-provided information. This work also presents one technique that enables incidental learning and, by implication, can help people process AI recommendations and explanations more carefully. 

   more » « less
2. Explaining machine learning classifiers through diverse counterfactual explanations

   https://doi.org/10.1145/3351095.3372850  

   Mothilal, Ramaravind K.; Sharma, Amit; Tan, Chenhao (January 2020, FAT*)

   Post-hoc explanations of machine learning models are crucial for people to understand and act on algorithmic predictions. An intriguing class of explanations is through counterfactuals, hypothetical examples that show people how to obtain a different prediction. We posit that effective counterfactual explanations should satisfy two properties: feasibility of the counterfactual actions given user context and constraints, and diversity among the counterfactuals presented. To this end, we propose a framework for generating and evaluating a diverse set of counterfactual explanations based on determinantal point processes. To evaluate the actionability of counterfactuals, we provide metrics that enable comparison of counterfactual-based methods to other local explanation methods. We further address necessary tradeoffs and point to causal implications in optimizing for counterfactuals. Our experiments on four real-world datasets show that our framework can generate a set of counterfactuals that are diverse and well approximate local decision boundaries, outperforming prior approaches to generating diverse counterfactuals. We provide an implementation of the framework at https://github.com/microsoft/DiCE. 

   more » « less
3. Robot Delivery of Actionable Counterfactual Neural Network Explanations: Results in Group Perception

   https://doi.org/10.1109/RO-MAN63969.2025.11217869  

   Narcomey, Austin; Claure, Houston; Vázquez, Marynel (August 2025, IEEE)

   Ensuring transparency in robot’s decision-making is increasingly important as they become better at making complex decisions when collaborating with humans. Among the most promising approaches in Human-Robot Interaction (HRI) is the use of counterfactual explanations. Within HRI, counterfactual explanations typically provide insight into robot’s models by presenting changes to the inputs of the model that influence policy decisions and resulting outcomes. While prior work presents counterfactuals that may vary features that users cannot control, we focus on actionable counterfactuals that show how the robot responds to feasible user actions and thereby reveal only what users need to understand to interact effectively. We introduce a novel explanation framework that generates actionable counterfactuals for neural network models in HRI applications. We evaluate this framework in simulation and on live sensor data during an in-person demonstration with groups of participants. Our results highlight the value of each component of our framework and demonstrate its effectiveness in real-time robotic explanations. 

   more » « less
4. Counterfactual Explanations Can Be Manipulated

   Slack, Dylan; Hilgard, Anna; Lakkaraju, Himabindu; Singh, Sameer (January 2021, Advances in Neural Information Processing Systems (NeurIPS))

   Counterfactual explanations are emerging as an attractive option for providing recourse to individuals adversely impacted by algorithmic decisions. As they are deployed in critical applications (e.g. law enforcement, financial lending), it becomes important to ensure that we clearly understand the vulnerabilties of these methods and find ways to address them. However, there is little understanding of the vulnerabilities and shortcomings of counterfactual explanations. In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated. More specifically, we show counterfactual explanations may converge to drastically different counterfactuals under a small perturbation indicating they are not robust. Leveraging this insight, we introduce a novel objective to train seemingly fair models where counterfactual explanations find much lower cost recourse under a slight perturbation. We describe how these models can unfairly provide low-cost recourse for specific subgroups in the data while appearing fair to auditors. We perform experiments on loan and violent crime prediction data sets where certain subgroups achieve up to 20x lower cost recourse under the perturbation. These results raise concerns regarding the dependability of current counterfactual explanation techniques, which we hope will inspire investigations in robust counterfactual explanations. 

   more » « less
5. Counterfactual Explanations Can Be Manipulated

   Slack, Dylan; Hilgard, Anna; Lakkaraju, Himabindu; Singh, Sameer. (December 2021, Advances in neural information processing systems)

   Counterfactual explanations are emerging as an attractive option for providing recourse to individuals adversely impacted by algorithmic decisions. As they are deployed in critical applications (e.g. law enforcement, financial lending), it becomes important to ensure that we clearly understand the vulnerabilities of these methods and find ways to address them. However, there is little understanding of the vulnerabilities and shortcomings of counterfactual explanations. In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated. More specifically, we show counterfactual explanations may converge to drastically different counterfactuals under a small perturbation indicating they are not robust. Leveraging this insight, we introduce a novel objective to train seemingly fair models where counterfactual explanations find much lower cost recourse under a slight perturbation. We describe how these models can unfairly provide low-cost recourse for specific subgroups in the data while appearing fair to auditors. We perform experiments on loan and violent crime prediction data sets where certain subgroups achieve up to 20x lower cost recourse under the perturbation. These results raise concerns regarding the dependability of current counterfactual explanation techniques, which we hope will inspire investigations in robust counterfactual explanations. 

   more » « less