More Like this

1. Governing with Algorithmic Impact Assessments: Six Observations

   Watkins, Elizabeth; Moss, Emanuel; Metcalf, Jacob; Singh, Ranjit; Elish, Madeleine Clare (May 2021, AAAI / ACM Conference on Artificial Intelligence, Ethics, and Society (AIES))

   null (Ed.)

   Algorithmic impact assessments (AIA) are increasingly being proposed as a mechanism for algorithmic accountability. These assessments are seen as potentially useful for anticipating, avoiding, and mitigating the negative consequences of algorithmic decision-making systems (ADS). At the same time, what an AIA would entail remains under-specified. While promising, AIAs raise as many questions as they answer. Choices about the methods, scope, and purpose of impact assessments structure the possible governance outcomes. Decisions about what type of effects count as an impact, when impacts are assessed, whose interests are considered, who is invited to participate, who conducts the assessment, the public availability of the assessment, and what the outputs of the assessment might be all shape the forms of accountability that AIA proponents seek to encourage. These considerations remain open, and will determine whether and how AIAs can function as a viable governance mechanism in the broader algorithmic accountability toolkit, especially with regard to furthering the public interest. Because AlAs are still an incipient governance strategy, approaching them as social constructions that do not require a single or universal approach offers a chance to produce interventions that emerge from careful deliberation. 

   more » « less
2. Taking Algorithms to Courts: A Relational Approach to Algorithmic Accountability

   https://doi.org/10.1145/3593013.3594092  

   Metcalf, Jacob; Singh, Ranjit; Moss, Emanuel; Tafesse, Emnet; Watkins, Elizabeth Anne (June 2023, 2023 ACM Conference on Fairness, Accountability, and Transparency)

   In widely used sociological descriptions of how accountability is structured through institutions, an “actor” (e.g., the developer) is accountable to a “forum” (e.g., regulatory agencies) empowered to pass judgements on and demand changes from the actor or enforce sanctions. However, questions about structuring accountability persist: why and how is a forum compelled to keep making demands of the actor when such demands are called for? To whom is a forum accountable in the performance of its responsibilities, and how can its practices and decisions be contested? In the context of algorithmic accountability, we contend that a robust accountability regime requires a triadic relationship, wherein the forum is also accountable to another entity: the public(s). Typically, as is the case with environmental impact assessments, public(s) make demands upon the forum's judgements and procedures through the courts, thereby establishing a minimum standard of due diligence. However, core challenges relating to: (1) lack of documentation, (2) difficulties in claiming standing, and (3) struggles around admissibility of expert evidence on and achieving consensus over the workings of algorithmic systems in adversarial proceedings prevent the public from approaching the courts when faced with algorithmic harms. In this paper, we demonstrate that the courts are the primary route—and the primary roadblock—in the pursuit of redress for algorithmic harms. Courts often find algorithmic harms non-cognizable and rarely require developers to address material claims of harm. To address the core challenges of taking algorithms to court, we develop a relational approach to algorithmic accountability that emphasizes not what the actors do nor the results of their actions, but rather how interlocking relationships of accountability are constituted in a triadic relationship between actors, forums, and public(s). As is the case in other regulatory domains, we believe that impact assessments (and similar accountability documentation) can provide the grounds for contestation between these parties, but only when that triad is structured such that the public(s) are able to cohere around shared experiences and interests, contest the outcomes of algorithmic systems that affect their lives, and make demands upon the other parties. Where courts now find algorithmic harms non-cognizable, an impact assessment regime can potentially create procedural rights to protect substantive rights of the public(s). This would require algorithmic accountability policies currently under consideration to provide the public(s) with adequate standing in courts, and opportunities to access and contest the actor's documentation and the forum's judgments. 

   more » « less
3. Algorithmic management in a work context

   https://doi.org/10.1177/20539517211020332  

   Jarrahi, Mohammad Hossein; Newlands, Gemma; Lee, Min Kyung; Wolf, Christine T.; Kinder, Eliscia; Sutherland, Will (July 2021, Big Data & Society)

   The rapid development of machine-learning algorithms, which underpin contemporary artificial intelligence systems, has created new opportunities for the automation of work processes and management functions. While algorithmic management has been observed primarily within the platform-mediated gig economy, its transformative reach and consequences are also spreading to more standard work settings. Exploring algorithmic management as a sociotechnical concept, which reflects both technological infrastructures and organizational choices, we discuss how algorithmic management may influence existing power and social structures within organizations. We identify three key issues. First, we explore how algorithmic management shapes pre-existing power dynamics between workers and managers. Second, we discuss how algorithmic management demands new roles and competencies while also fostering oppositional attitudes toward algorithms. Third, we explain how algorithmic management impacts knowledge and information exchange within an organization, unpacking the concept of opacity on both a technical and organizational level. We conclude by situating this piece in broader discussions on the future of work, accountability, and identifying future research steps. 

   more » « less
4. Algorithmic Impact Assessments and Accountability: The Co-construction of Impacts

   Metcalf, Jacob; Moss, Emanuel; Watkins, Elizabeth Anne; Singh, Ranjit; Elish, Madeleine Clare (March 2021, ACM Conference on Fairness, Accountability,and Transparency (FAccT ’21))

   null (Ed.)

   Algorithmic impact assessments (AIAs) are an emergent form of accountability for entities that build and deploy automated decision-support systems. These are modeled after impact assessments in other domains. Our study of the history of impact assessments shows that "impacts" are an evaluative construct that enable institutions to identify and ameliorate harms experienced because of a policy decision or system. Every domain has different expectations and norms about what constitutes impacts and harms, how potential harms are rendered as the impacts of a particular undertaking, who is responsible for conducting that assessment, and who has the authority to act on the impact assessment to demand changes to that undertaking. By examining proposals for AIAs in relation to other domains, we find that there is a distinct risk of constructing algorithmic impacts as organizationally understandable metrics that are nonetheless inappropriately distant from the harms experienced by people, and which fall short of building the relationships required for effective accountability. To address this challenge of algorithmic accountability, and as impact assessments become a commonplace process for evaluating harms, the FAccT community should A) understand impacts as objects constructed for evaluative purposes, B) attempt to construct impacts as close as possible to actual harms, and C) recognize that accountability governance requires the input of various types of expertise and affected communities. We conclude with lessons for assembling cross-expertise consensus for the co-construction of impacts and to build robust accountability relationships. 

   more » « less
5. Using non-achievement based variables to differentiate district performance grades

   https://doi.org/10.3102/IP.22.1891006  

   Folger, T.; Roberts, A.; Bostic, J. (April 2022, Annual meeting program American Educational Research Association)

   The COVID-19 pandemic disrupted many school accountability systems that rely on student- level achievement data. Many states have encountered uncertainty about how to meet federal accountability requirements without typical school data. Prior research provides an abundance of evidence that student achievement is correlated to students' social background, which raises concerns about the predictive bias of accountability systems. The focus of this quantitative study is to explore the predictive ability of non-achievement based variables (i.e., students' social background) on measures of school accountability in one Midwest state. Results suggest that social background and community demographic variables have a significant impact on measures of school accountability, and might be interpreted cautiously. Implications for policy and future research are discussed. 

   more » « less