skip to main content


This content will become publicly available on May 18, 2024

Title: Auditing Practitioner Judgment for Algorithmic Fairness Implications
The development of Artificial Intelligence (AI) systems involves a significant level of judgment and decision making on the part of engineers and designers to ensure the safety, robustness, and ethical design of such systems. However, the kinds of judgments that practitioners employ while developing AI platforms are rarely foregrounded or examined to explore areas practitioners might need ethical support. In this short paper, we employ the concept of design judgment to foreground and examine the kinds of sensemaking software engineers use to inform their decisionmaking while developing AI systems. Relying on data generated from two exploratory observation studies of student software engineers, we connect the concept of fairness to the foregrounded judgments to implicate their potential algorithmic fairness impacts. Our findings surface some ways in which the design judgment of software engineers could adversely impact the downstream goal of ensuring fairness in AI systems. We discuss the implications of these findings in fostering positive innovation and enhancing fairness in AI systems, drawing attention to the need to provide ethical guidance, support, or intervention to practitioners as they engage in situated and contextual judgments while developing AI systems.  more » « less
Award ID(s):
1909714
NSF-PAR ID:
10429711
Author(s) / Creator(s):
;
Date Published:
Journal Name:
2023 IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS)
Page Range / eLocation ID:
01 to 05
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    The engineering disciplines are rigorous in their application of scientific principles, and these principles are the ones most directly addressed in undergraduate engineering classrooms. However, engineers are also called to make decisions that implicitly account for complex criteria, including the welfare of those who use or are impacted by the systems engineers design and the economic needs of their employers. As a result, engineering is an art that requires practitioners to routinely navigate difficult tradeoffs that require professional judgments. These judgments include economic, ethical, social, and value-based dimensions. These dimensions can be conflicting, increasing the complexity of practice and foregrounding the prominence of judgment. And often, these judgements need to be explained to colleagues, managers, and clients through a range of written documents. Yet little work to date has investigated the relationship between the writing engineering students do and the development of engineering judgement, particularly in terms of how these facets intersect in students developing engineering identities . Therefore, the overall goal of this project is to elucidate the interactions between how students’ identification with the engineering profession impacts the way they convey engineering judgments to different audiences. 
    more » « less
  2. null (Ed.)
    A quiet revolution is afoot in the field of law. Technical systems employing algorithms are shaping and displacing professional decision making, and they are disrupting and restructuring relationships between law firms, lawyers, and clients. Decision-support systems marketed to legal professionals to support e-discovery—generally referred to as “technology assisted review” (TAR)—increasingly rely on “predictive coding”: machine-learning techniques to classify and predict which of the voluminous electronic documents subject to litigation should be withheld or produced to the opposing side. These systems and the companies offering them are reshaping relationships between lawyers and clients, introducing new kinds of professionals into legal practice, altering the discovery process, and shaping how lawyers construct knowledge about their cases and professional obligations. In the midst of these shifting relationships—and the ways in which these systems are shaping the construction and presentation of knowledge—lawyers are grappling with their professional obligations, ethical duties, and what it means for the future of legal practice. Through in-depth, semi-structured interviews of experts in the e-discovery technology space—the technology company representatives who develop and sell such systems to law firms and the legal professionals who decide whether and how to use them in practice—we shed light on the organizational structures, professional rules and norms, and technical system properties that are shaping and being reshaped by predictive coding systems. Our findings show that AI-supported decision systems such as these are reconfiguring professional work practices. In particular, they highlight concerns about potential loss of professional agency and skill, limited understanding and thereby both over- and under reliance on decision-support systems, and confusion about responsibility and accountability as new kinds of technical professionals and technologies are brought into legal practice. The introduction of predictive coding systems and the new professional and organizational arrangements they are ushering into legal practice compound general concerns over the opacity of technical systems with specific concerns about encroachments on the construction of expert knowledge, liability frameworks, and the potential (mis)alignment of machine reasoning with professional logic and ethics. Based on our findings, we conclude that predictive coding tools—and likely other algorithmic systems lawyers use to construct knowledge and reason about legal practice— challenge the current model for evaluating whether and how tools are appropriate for legal practice. As tools become both more complex and more consequential, it is unreasonable to rely solely on legal professionals—judges, law firms, and lawyers—to determine which technologies are appropriate for use. The legal professionals we interviewed report relying on the evaluation and judgment of a range of new technical experts within law firms and, increasingly, third-party vendors and their technical experts. This system for choosing technical systems upon which lawyers rely to make professional decisions—e.g., whether documents are responsive, or whether the standard of proportionality has been met—is no longer sufficient. As the tools of medicine are reviewed by appropriate experts before they are put out for consideration and adoption by medical professionals, we argue that the legal profession must develop new processes for determining which algorithmic tools are fit to support lawyers’ decision making. Relatedly, because predictive coding systems are used to produce lawyers’ professional judgment, we argue they must be designed for contestability— providing greater transparency, interaction, and configurability around embedded choices to ensure decisions about how to embed core professional judgments, such as relevance and proportionality, remain salient and demand engagement from lawyers, not just their technical experts. 
    more » « less
  3. Engineering judgment is critical to both engineering education and engineering practice, and the ability to practice or participate in engineering judgment is often considered central to the formation of professional engineering identities. In practice, engineers must make difficult judgments that evaluate potentially competing objectives, ambiguity, uncertainty, incomplete information, and evolving technical knowledge. Nonetheless, while engineering judgment is implicit in engineering work and so central to identification with the profession, educators and practitioners have few actionable frameworks to employ when considering how to develop and assess this capacity in students. In this paper, we propose a theoretical framework designed to inform both educators and researchers that positions engineering judgment at the intersection of the cognitive dimensions of naturalistic decision-making, and discursive dimensions of identity. Our proposed theory positions engineering judgment not only as an individual capacity practiced by individual engineers alone but also as the capacity to position oneself within the discursive community so as to participate in the construction of engineering judgments among a group of professionals working together. Our theory draws on several strands of existing research to theorize a working framework for engineering judgment that considers the cognitive processes associated with making judgments and the inextricable discursive practices associated with negotiating those judgments in context. In constructing this theory, we seek to provide engineering education practitioners and researchers with a framework that can inform the design of assignments, curricula, or experiences that are intended to foster students’ participation in the development and practice of engineering judgment. 
    more » « less
  4. In an era of ubiquitous digital interfaces and systems, technology and design practitioners must address a range of ethical dilemmas surrounding the use of persuasive design techniques and how to balance shareholder and end-user needs [2], [5]. Similarly, the increasing user concerns about unethical products and services [1] is paralleling a rise in regulatory interests in enforcing ethical design and engineering practices among technology practitioners, surfacing a need for further support. Although various scholars have developed frameworks and methods to support practitioners in navigating these challenging contexts [3], [4], often, there is a lack of resonance between these generic methods and the situated ethical complexities facing the practitioner in their everyday work. In this project, we designed and implemented a three-hour cocreation workshop with designers, engineers, and technologists to support them to develop bespoke ethics-focused action plans that are resonant with the ethical challenges they face in their everyday practice. In developing the co-creation session, we sought to answer the following questions to empower practitioners: • How can we support practitioners in developing action plans to address ethical dilemmas in their everyday work? and • How can we empower designers to design more responsibly? Building on these questions as a guide, we employed Miro, a digital whiteboard platform, to develop the co-creation experience. The final c o-creation e xperience w as d esigned w ith the visual metaphor of a “house” with four floors and multiple rooms that allowed participants to complete different tasks per room, all aimed towards the overall goal of developing participants' own personalized action plan in an interactive and collaborative way. We invited participants to share their stories and ethical dilemmas to support their creation and iteration of a personal action plan that they could later use in their everyday work context. Across the six co-creation sessions we conducted, participants (n=26) gained a better understanding of the drivers for ethical action in the context of their everyday work and developed an action plan through the co-creation workshop that enabled them to constructively engage with ethical challenges in their professional context. At the end of the session, participants were provided the action plans they created to allow them to use it in their practice. Furthermore, the co-design workshops were designed such that practitioners could take them away (the house and session guide) and run them independently at their organization or another context to support their objectives. We describe the building and the activities conducted in each floor below and will provide a pictorial representation of the house with the different floors, rooms, and activities on the poster presentation. a) First floor-Welcome, Introduction, Reflection: The first floor of the virtual house was designed to allow participants to introduce themselves and to reflect on and discuss the ethical concerns they wished to resolve during the session. b) Second floor-Shopping for ethics-focused methods: The second floor of the virtual house was designed as a “shopping” space where participants selected from range of ethicsfocused building blocks that they wish to potentially adapt or incorporate into their own action plan. They were also allowed to introduce their own methods or tools. c) Third floor-DIY Workspace: The third floor was designed as a DIY workspace to allow the participants to work in small groups to develop their own bespoke action plan based on building blocks they have gathered from their shopping trip and by using any other components they wish. The goal here was to support participants in developing methods and action plans that were resonant with their situated ethical complexities. d) Fourth floor-Gallery Space: The fourth floor was designed as a gallery to allow participants to share and discuss their action plans with other participants and to identify how their action plans could impact their future practice or educational experiences. Participants were also provided an opportunity at this stage to reflect on their experience participating in the session and provide feedback on opportunities for future improvement. 
    more » « less
  5. Traditional engineering courses typically approach teaching and problem solving by focusing on the physical dimensions of those problems without consideration of dynamic social and ethical dimensions. As such, projects can fail to consider community questions and concerns, broader impacts upon society, or otherwise result in inequitable outcomes. And, despite the fact that students in engineering receive training on the Professional Code of Ethics for Engineers, to which they are expected to adhere in practice, many students are unable to recognize and analyze real-life ethical challenges as they arise. Indeed, research has found that students are typically less engaged with ethics—defined as the awareness and judgment of microethics and macroethics, sensitivity to diversity, and interest in promoting organizational ethical culture—at the end of their engineering studies than they were at the beginning. As such, many studies have focused on developing and improving the curriculum surrounding ethics through, for instance, exposing students to ethics case studies. However, such ethics courses often present a narrow and simplified view of ethics that students may struggle to integrate with their broader experience as engineers. Thus, there is a critical need to unpack the complexity of ethical behavior amongst engineering students in order to determine how to better foster ethical judgment and behavior. Promoting ethical behavior among engineering students and developing a culture of ethical behavior within institutions have become goals of many engineering programs. Towards this goal, we present an overview of the current scholarship of engineering ethics and propose a theoretical framework of ethical behavior using a review of articles related to engineering ethics from 1990-2020. These articles were selected based upon their diversity of scope and methods until saturation was reached. A thematic analysis of articles was then performed using Nvivo. The review engages in theories across disciplines including philosophy, education and psychology. Preliminary results identify two major kinds of drivers of ethical behavior, namely individual level ethical behavior drivers (awareness of microethics, awareness of macroethics, implicit understanding, and explicit understanding) and institutional drivers (diversity and institutional ethical culture). In this paper, we present an overview and discussion of two drivers of ethical behavior at the individual level, namely awareness of microethics and awareness of macroethics, based on a review of 50 articles. Our results indicate that an awareness of both microethics and macroethics is essential in promoting ethical behavior amongst students. The review also points to a need to focus on increasing students’ awareness of macroethics. This research thus addresses the need, driven by existing scholarship, to identify a conceptual framework for explaining how ethical judgment and behavior in engineering can be further promoted. 
    more » « less