Algorithmic decision-making systems are increasingly used throughout the public and private sectors to make important decisions or assist humans in making these decisions with real social consequences. While there has been substantial research in recent years to build fair decision-making algorithms, there has been less research seeking to understand the factors that affect people's perceptions of fairness in these systems, which we argue is also important for their broader acceptance. In this research, we conduct an online experiment to better understand perceptions of fairness, focusing on three sets of factors: algorithm outcomes, algorithm development and deployment procedures, and individual differences. We find that people rate the algorithm as more fair when the algorithm predicts in their favor, even surpassing the negative effects of describing algorithms that are very biased against particular demographic groups. We find that this effect is moderated by several variables, including participants' education level, gender, and several aspects of the development procedure. Our findings suggest that systems that evaluate algorithmic fairness through users' feedback must consider the possibility of "outcome favorability" bias.
more »
« less
The Effects of AI Biases and Explanations on Human Decision Fairness: A Case Study of Bidding in Rental Housing Markets
The use of AI-based decision aids in diverse domains has inspired many empirical investigations into how AI models’ decision recommendations impact humans’ decision accuracy in AI-assisted decision making, while explorations on the impacts on humans’ decision fairness are largely lacking despite their clear importance. In this paper, using a real-world business decision making scenario—bidding in rental housing markets—as our testbed, we present an experimental study on understanding how the bias level of the AI-based decision aid as well as the provision of AI explanations affect the fairness level of humans’ decisions, both during and after their usage of the decision aid. Our results suggest that when people are assisted by an AI-based decision aid, both the higher level of racial biases the decision aid exhibits and surprisingly, the presence of AI explanations, result in more unfair human decisions across racial groups. Moreover, these impacts are partly made through triggering humans’ “disparate interactions” with AI. However, regardless of the AI bias level and the presence of AI explanations, when people return to make independent decisions after their usage of the AI-based decision aid, their decisions no longer exhibit significant unfairness across racial groups.
more » « less- Award ID(s):
- 2040800
- NSF-PAR ID:
- 10488903
- Publisher / Repository:
- International Joint Conferences on Artificial Intelligence Organization
- Date Published:
- Journal Name:
- Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23)
- Page Range / eLocation ID:
- 3076 to 3084
- Format(s):
- Medium: X
- Location:
- Macau, SAR China
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
BACKGROUND Charles Darwin’s Descent of Man, and Selection in Relation to Sex tackled the two main controversies arising from the Origin of Species: the evolution of humans from animal ancestors and the evolution of sexual ornaments. Most of the book focuses on the latter, Darwin’s theory of sexual selection. Research since supports his conjecture that songs, perfumes, and intricate dances evolve because they help secure mating partners. Evidence is overwhelming for a primary role of both male and female mate choice in sexual selection—not only through premating courtship but also through intimate interactions during and long after mating. But what makes one prospective mate more enticing than another? Darwin, shaped by misogyny and sexual prudery, invoked a “taste for the beautiful” without speculating on the origin of the “taste.” How to explain when the “final marriage ceremony” is between two rams? What of oral sex in bats, cloacal rubbing in bonobos, or the sexual spectrum in humans, all observable in Darwin’s time? By explaining desire through the lens of those male traits that caught his eyes and those of his gender and culture, Darwin elided these data in his theory of sexual evolution. Work since Darwin has focused on how traits and preferences coevolve. Preferences can evolve even if attractive signals only predict offspring attractiveness, but most attention has gone to the intuitive but tenuous premise that mating with gorgeous partners yields vigorous offspring. By focusing on those aspects of mating preferences that coevolve with male traits, many of Darwin’s influential followers have followed the same narrow path. The sexual selection debate in the 1980s was framed as “good genes versus runaway”: Do preferences coevolve with traits because traits predict genetic benefits, or simply because they are beautiful? To the broader world this is still the conversation. ADVANCES Even as they evolve toward ever-more-beautiful signals and healthier offspring, mate-choice mechanisms and courter traits are locked in an arms race of coercion and resistance, persuasion and skepticism. Traits favored by sexual selection often do so at the expense of chooser fitness, creating sexual conflict. Choosers then evolve preferences in response to the costs imposed by courters. Often, though, the current traits of courters tell us little about how preferences arise. Sensory systems are often tuned to nonsexual cues like food, favoring mating signals resembling those cues. And preferences can emerge simply from selection on choosing conspecifics. Sexual selection can therefore arise from chooser biases that have nothing to do with ornaments. Choice may occur before mating, as Darwin emphasized, but individuals mate multiple times and bias fertilization and offspring care toward favored partners. Mate choice can thus occur in myriad ways after mating, through behavioral, morphological, and physiological mechanisms. Like other biological traits, mating preferences vary among individuals and species along multiple dimensions. Some of this is likely adaptive, as different individuals will have different optimal mates. Indeed, mate choice may be more about choosing compatible partners than picking the “best” mate in the absolute sense. Compatibility-based choice can drive or reinforce genetic divergence and lead to speciation. The mechanisms underlying the “taste for the beautiful” determine whether mate choice accelerates or inhibits reproductive isolation. If preferences are learned from parents, or covary with ecological differences like the sensory environment, then choice can promote genetic divergence. If everyone shares preferences for attractive ornaments, then choice promotes gene flow between lineages. OUTLOOK Two major trends continue to shift the emphasis away from male “beauty” and toward how and why individuals make sexual choices. The first integrates neuroscience, genomics, and physiology. We need not limit ourselves to the feathers and dances that dazzled Darwin, which gives us a vastly richer picture of mate choice. The second is that despite persistent structural inequities in academia, a broader range of people study a broader range of questions. This new focus confirms Darwin’s insight that mate choice makes a primary contribution to sexual selection, but suggests that sexual selection is often tangential to mate choice. This conclusion challenges a persistent belief with sinister roots, whereby mate choice is all about male ornaments. Under this view, females evolve to prefer handsome males who provide healthy offspring, or alternatively, to express flighty whims for arbitrary traits. But mate-choice mechanisms also evolve for a host of other reasons Understanding mate choice mechanisms is key to understanding how sexual decisions underlie speciation and adaptation to environmental change. New theory and technology allow us to explicitly connect decision-making mechanisms with their evolutionary consequences. A century and a half after Darwin, we can shift our focus to females and males as choosers, rather than the gaudy by-products of mate choice. Mate choice mechanisms across domains of life. Sensory periphery for stimulus detection (yellow), brain for perceptual integration and evaluation (orange), and reproductive structures for postmating choice among pollen or sperm (teal). ILLUSTRATION: KELLIE HOLOSKI/ SCIENCEmore » « less
-
While EXplainable Artificial Intelligence (XAI) approaches aim to improve human-AI collaborative decision-making by improving model transparency and mental model formations, experiential factors associated with human users can cause challenges in ways system designers do not anticipate. In this paper, we first showcase a user study on how anchoring bias can potentially affect mental model formations when users initially interact with an intelligent system and the role of explanations in addressing this bias. Using a video activity recognition tool in cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Motivated by these findings and similar existing work, we formalize and present a conceptual model of user’s past experiences that examine the relations between user’s backgrounds, experiences, and human factors in XAI systems based on usage time. Our work presents strong findings and implications, aiming to raise the awareness of AI designers towards biases associated with user impressions and backgrounds.more » « less
-
When people receive advice while making difficult decisions, they often make better decisions in the moment and also increase their knowledge in the process. However, such incidental learning can only occur when people cognitively engage with the information they receive and process this information thoughtfully. How do people process the information and advice they receive from AI, and do they engage with it deeply enough to enable learning? To answer these questions, we conducted three experiments in which individuals were asked to make nutritional decisions and received simulated AI recommendations and explanations. In the first experiment, we found that when people were presented with both a recommendation and an explanation before making their choice, they made better decisions than they did when they received no such help, but they did not learn. In the second experiment, participants first made their own choice, and only then saw a recommendation and an explanation from AI; this condition also resulted in improved decisions, but no learning. However, in our third experiment, participants were presented with just an AI explanation but no recommendation and had to arrive at their own decision. This condition led to both more accurate decisions and learning gains. We hypothesize that learning gains in this condition were due to deeper engagement with explanations needed to arrive at the decisions. This work provides some of the most direct evidence to date that it may not be sufficient to provide people with AI-generated recommendations and explanations to ensure that people engage carefully with the AI-provided information. This work also presents one technique that enables incidental learning and, by implication, can help people process AI recommendations and explanations more carefully.more » « less
-
Abstract: 100 words Jurors are increasingly exposed to scientific information in the courtroom. To determine whether providing jurors with gist information would assist in their ability to make well-informed decisions, the present experiment utilized a Fuzzy Trace Theory-inspired intervention and tested it against traditional legal safeguards (i.e., judge instructions) by varying the scientific quality of the evidence. The results indicate that jurors who viewed high quality evidence rated the scientific evidence significantly higher than those who viewed low quality evidence, but were unable to moderate the credibility of the expert witness and apply damages appropriately resulting in poor calibration. Summary: <1000 words Jurors and juries are increasingly exposed to scientific information in the courtroom and it remains unclear when they will base their decisions on a reasonable understanding of the relevant scientific information. Without such knowledge, the ability of jurors and juries to make well-informed decisions may be at risk, increasing chances of unjust outcomes (e.g., false convictions in criminal cases). Therefore, there is a critical need to understand conditions that affect jurors’ and juries’ sensitivity to the qualities of scientific information and to identify safeguards that can assist with scientific calibration in the courtroom. The current project addresses these issues with an ecologically valid experimental paradigm, making it possible to assess causal effects of evidence quality and safeguards as well as the role of a host of individual difference variables that may affect perceptions of testimony by scientific experts as well as liability in a civil case. Our main goal was to develop a simple, theoretically grounded tool to enable triers of fact (individual jurors) with a range of scientific reasoning abilities to appropriately weigh scientific evidence in court. We did so by testing a Fuzzy Trace Theory-inspired intervention in court, and testing it against traditional legal safeguards. Appropriate use of scientific evidence reflects good calibration – which we define as being influenced more by strong scientific information than by weak scientific information. Inappropriate use reflects poor calibration – defined as relative insensitivity to the strength of scientific information. Fuzzy Trace Theory (Reyna & Brainerd, 1995) predicts that techniques for improving calibration can come from presentation of easy-to-interpret, bottom-line “gist” of the information. Our central hypothesis was that laypeople’s appropriate use of scientific information would be moderated both by external situational conditions (e.g., quality of the scientific information itself, a decision aid designed to convey clearly the “gist” of the information) and individual differences among people (e.g., scientific reasoning skills, cognitive reflection tendencies, numeracy, need for cognition, attitudes toward and trust in science). Identifying factors that promote jurors’ appropriate understanding of and reliance on scientific information will contribute to general theories of reasoning based on scientific evidence, while also providing an evidence-based framework for improving the courts’ use of scientific information. All hypotheses were preregistered on the Open Science Framework. Method Participants completed six questionnaires (counterbalanced): Need for Cognition Scale (NCS; 18 items), Cognitive Reflection Test (CRT; 7 items), Abbreviated Numeracy Scale (ABS; 6 items), Scientific Reasoning Scale (SRS; 11 items), Trust in Science (TIS; 29 items), and Attitudes towards Science (ATS; 7 items). Participants then viewed a video depicting a civil trial in which the defendant sought damages from the plaintiff for injuries caused by a fall. The defendant (bar patron) alleged that the plaintiff (bartender) pushed him, causing him to fall and hit his head on the hard floor. Participants were informed at the outset that the defendant was liable; therefore, their task was to determine if the plaintiff should be compensated. Participants were randomly assigned to 1 of 6 experimental conditions: 2 (quality of scientific evidence: high vs. low) x 3 (safeguard to improve calibration: gist information, no-gist information [control], jury instructions). An expert witness (neuroscientist) hired by the court testified regarding the scientific strength of fMRI data (high [90 to 10 signal-to-noise ratio] vs. low [50 to 50 signal-to-noise ratio]) and gist or no-gist information both verbally (i.e., fairly high/about average) and visually (i.e., a graph). After viewing the video, participants were asked if they would like to award damages. If they indicated yes, they were asked to enter a dollar amount. Participants then completed the Positive and Negative Affect Schedule-Modified Short Form (PANAS-MSF; 16 items), expert Witness Credibility Scale (WCS; 20 items), Witness Credibility and Influence on damages for each witness, manipulation check questions, Understanding Scientific Testimony (UST; 10 items), and 3 additional measures were collected, but are beyond the scope of the current investigation. Finally, participants completed demographic questions, including questions about their scientific background and experience. The study was completed via Qualtrics, with participation from students (online vs. in-lab), MTurkers, and non-student community members. After removing those who failed attention check questions, 469 participants remained (243 men, 224 women, 2 did not specify gender) from a variety of racial and ethnic backgrounds (70.2% White, non-Hispanic). Results and Discussion There were three primary outcomes: quality of the scientific evidence, expert credibility (WCS), and damages. During initial analyses, each dependent variable was submitted to a separate 3 Gist Safeguard (safeguard, no safeguard, judge instructions) x 2 Scientific Quality (high, low) Analysis of Variance (ANOVA). Consistent with hypotheses, there was a significant main effect of scientific quality on strength of evidence, F(1, 463)=5.099, p=.024; participants who viewed the high quality evidence rated the scientific evidence significantly higher (M= 7.44) than those who viewed the low quality evidence (M=7.06). There were no significant main effects or interactions for witness credibility, indicating that the expert that provided scientific testimony was seen as equally credible regardless of scientific quality or gist safeguard. Finally, for damages, consistent with hypotheses, there was a marginally significant interaction between Gist Safeguard and Scientific Quality, F(2, 273)=2.916, p=.056. However, post hoc t-tests revealed significantly higher damages were awarded for low (M=11.50) versus high (M=10.51) scientific quality evidence F(1, 273)=3.955, p=.048 in the no gist with judge instructions safeguard condition, which was contrary to hypotheses. The data suggest that the judge instructions alone are reversing the pattern, though nonsignificant, those who received the no gist without judge instructions safeguard awarded higher damages in the high (M=11.34) versus low (M=10.84) scientific quality evidence conditions F(1, 273)=1.059, p=.30. Together, these provide promising initial results indicating that participants were able to effectively differentiate between high and low scientific quality of evidence, though inappropriately utilized the scientific evidence through their inability to discern expert credibility and apply damages, resulting in poor calibration. These results will provide the basis for more sophisticated analyses including higher order interactions with individual differences (e.g., need for cognition) as well as tests of mediation using path analyses. [References omitted but available by request] Learning Objective: Participants will be able to determine whether providing jurors with gist information would assist in their ability to award damages in a civil trial.more » « less
