More Like this

1. Auditing for Bias in Ad Delivery Using Inferred Demographic Attributes

   https://doi.org/10.1145/3715275.3732172  

   Imana, Basileal; Korolova, Aleksandra; Heidemann, John (June 2025, ACM)

   Auditing social-media algorithms has become a focus of public-interest research and policymaking to ensure their fairness across demographic groups such as race, age, and gender in consequential domains such as the presentation of employment opportunities. However, such demographic attributes are often unavailable to auditors and platforms. When demographics data is unavailable, auditors commonly \emph{infer} them from other available information. In this work, we study the effects of inference error on auditing for bias in one prominent application: \emph{black-box} audit of ad delivery using \emph{paired ads}. We show that inference error, if not accounted for, causes auditing to falsely miss skew that exists. We then propose a way to mitigate the inference error when evaluating skew in ad delivery algorithms. Our method works by adjusting for expected error due to demographic inference, and it makes skew detection more sensitive when attributes must be inferred. Because inference is increasingly used for auditing, our results provide an important addition to the auditing toolbox to promote correct audits of ad delivery algorithms for bias. While the impact of attribute inference on accuracy has been studied in other domains, our work is the first to consider it for black-box evaluation of ad delivery bias, when only aggregate data is available to the auditor. 

   more » « less
2. Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice

   https://doi.org/10.1145/3544548.3581026  

   Deng, Wesley Hanwen; Guo, Boyuan; Devrio, Alicia; Shen, Hong; Eslami, Motahhare; Holstein, Kenneth (April 2023, ACM)

   Recent years have seen growing interest among both researchers and practitioners in user-engaged approaches to algorithm auditing, which directly engage users in detecting problematic behaviors in algorithmic systems. However, we know little about industry practitioners’ current practices and challenges around user-engaged auditing, nor what opportunities exist for them to better leverage such approaches in practice. To investigate, we conducted a series of interviews and iterative co-design activities with practitioners who employ user-engaged auditing approaches in their work. Our findings reveal several challenges practitioners face in appropriately recruiting and incentivizing user auditors, scaffolding user audits, and deriving actionable insights from user-engaged audit reports. Furthermore, practitioners shared organizational obstacles to user-engaged auditing, surfacing a complex relationship between practitioners and user auditors. Based on these findings, we discuss opportunities for future HCI research to help realize the potential (and mitigate risks) of user-engaged auditing in industry practice. 

   more » « less
3. Individually Fair Learning with One Sided Feedback

   Bechavod, Yahav; Roth, Aaroh (January 2023, International Conference on Machine Learning)

   We consider an online learning problem with one-sided feedback, in which the learner is able to observe the true label only for positively predicted instances. On each round, k instances arrive and receive classification outcomes according to a randomized policy deployed by the learner, whose goal is to maximize accuracy while deploying individually fair policies. We first extend the framework of Bechavod et al. (2020), which relies on the existence of a human fairness auditor for detecting fairness violations, to instead incorporate feedback from dynamically-selected panels of multiple, possibly inconsistent, auditors. We then construct an efficient reduction from our problem of online learning with one-sided feedback and a panel reporting fairness violations to the contextual combinatorial semi-bandit problem (Cesa-Bianchi & Lugosi, 2009, György et al., 2007). Finally, we show how to leverage the guarantees of two algorithms in the contextual combinatorial semi-bandit setting: Exp2 (Bubeck et al., 2012) and the oracle-efficient Context-Semi-Bandit-FTPL (Syrgkanis et al., 2016), to provide multi-criteria no regret guarantees simultaneously for accuracy and fairness. Our results eliminate two potential sources of bias from prior work: the "hidden outcomes" that are not available to an algorithm operating in the full information setting, and human biases that might be present in any single human auditor, but can be mitigated by selecting a well chosen panel. 

   more » « less
4. Adaptive Risk-Limiting Comparison Audits

   Harrison, Abigail; Fuller, Benjamin; Russell, Alexander (May 2023, IEEE Security and Privacy)

   Risk-limiting audits (RLAs) are rigorous statistical procedures meant to detect invalid election results. RLAs examine paper ballots cast during the election to statistically assess the possibility of a disagreement between the winner determined by the ballots and the winner reported by tabulation. The design of an RLA must balance risk against efficiency: "risk" refers to a bound on the chance that the audit fails to detect such a disagreement when one occurs; "efficiency" refers to the total effort to conduct the audit. The most efficient approaches—when measured in terms of the number of ballots that must be inspected—proceed by "ballot comparison." However, ballot comparison requires an (untrusted) declaration of the contents of each cast ballot, rather than a simple tabulation of vote totals. This "cast-vote record table" (CVR) is then spot-checked against ballots for consistency. In many practical settings, the cost of generating a suitable CVR dominates the cost of conducting the audit which has prevented widespread adoption of these sample-efficient techniques. We introduce a new RLA procedure: an "adaptive ballot comparison" audit. In this audit, a global CVR is never produced; instead, a three-stage procedure is iterated: 1) a batch is selected, 2) a CVR is produced for that batch, and 3) a ballot within the batch is sampled, inspected by auditors, and compared with the CVR. We prove that such an audit can achieve risk commensurate with standard comparison audits while generating a fraction of the CVR. We present three main contributions: (1) a formal adversarial model for RLAs; (2) definition and analysis of an adaptive audit procedure with rigorous risk limits and an associated correctness analysis accounting for the incidental errors arising in typical audits; and (3) an analysis of efficiency. 

   more » « less
5. How Do Local Officials Conceptualize Sustainability as Practiced in Their Communities?: An Examination of U.S. Cities

   https://doi.org/10.1080/01944363.2024.2312941  

   Hofmeyer, Sarah L.; Hawkins, Christopher V.; Krause, Rachel M.; Park, Angela (March 2024, Journal of the American Planning Association)

   Although they play a key role in shaping local efforts, there is limited research on how city officials define sustainability as it is practiced in their communities. To address this limitation and contribute to planning research, we leveraged a unique nationwide data set of sustainability definitions provided by the sustainability lead in more than 400 U.S city governments. Our study’s qualitative analysis of these statements complements existing research by exploring emerging themes on how sustainability is perceived and practiced at the local level. Results indicated that practitioners’ conceptualizations reflected five general orientation categories: action, aspiration, emotion, process and organization, and progress. We evaluated the association of these general orientation categories with cities’ administrative arrangements, political environments, resources, and capacities. Findings suggested that supportive contexts were associated with city sustainability staff defining sustainability in aspirational terms, as opposed to emphasizing progress or discrete actions. 

   more » « less