More Like this

1. "AI in healthcare: data governance challenges"

   https://doi.org/10.21037/jhmhp-2020-ai-05  

   Winter, J.S. (March 2021, Journal of hospital management and health policy)

   Reddy, S.; Winter, J.S.; Padmanabhan, S. (Ed.)

   AI applications are poised to transform health care, revolutionizing benefits for individuals, communities, and health-care systems. As the articles in this special issue aptly illustrate, AI innovations in healthcare are maturing from early success in medical imaging and robotic process automation, promising a broad range of new applications. This is evidenced by the rapid deployment of AI to address critical challenges related to the COVID-19 pandemic, including disease diagnosis and monitoring, drug discovery, and vaccine development. At the heart of these innovations is the health data required for deep learning applications. Rapid accumulation of data, along with improved data quality, data sharing, and standardization, enable development of deep learning algorithms in many healthcare applications. One of the great challenges for healthcare AI is effective governance of these data—ensuring thoughtful aggregation and appropriate access to fuel innovation and improve patient outcomes and healthcare system efficiency while protecting the privacy and security of data subjects. Yet the literature on data governance has rarely looked beyond important pragmatic issues related to privacy and security. Less consideration has been given to unexpected or undesirable outcomes of healthcare in AI, such as clinician deskilling, algorithmic bias, the “regulatory vacuum”, and lack of public engagement. Amidst growing calls for ethical governance of algorithms, Reddy et al. developed a governance model for AI in healthcare delivery, focusing on principles of fairness, accountability, and transparency (FAT), and trustworthiness, and calling for wider discussion. Winter and Davidson emphasize the need to identify underlying values of healthcare data and use, noting the many competing interests and goals for use of health data—such as healthcare system efficiency and reform, patient and community health, intellectual property development, and monetization. Beyond the important considerations of privacy and security, governance must consider who will benefit from healthcare AI, and who will not. Whose values drive health AI innovation and use? How can we ensure that innovations are not limited to the wealthiest individuals or nations? As large technology companies begin to partner with health care systems, and as personally generated health data (PGHD) (e.g., fitness trackers, continuous glucose monitors, health information searches on the Internet) proliferate, who has oversight of these complex technical systems, which are essentially a black box? To tackle these complex and important issues, it is important to acknowledge that we have entered a new technical, organizational, and policy environment due to linked data, big data analytics, and AI. Data governance is no longer the responsibility of a single organization. Rather, multiple networked entities play a role and responsibilities may be blurred. This also raises many concerns related to data localization and jurisdiction—who is responsible for data governance? In this emerging environment, data may no longer be effectively governed through traditional policy models or instruments. 

   more » « less
2. Assessment of Potential Bias in Kidney Transplant Acceptance Decision Using the Deceased Donor-Recipient Attributes Matching AI Model

   https://doi.org/10.1016/j.ajt.2025.07.839  

   Fadaiya, O; Akinola, AA; Dagli, CH; Threlkeld, R (August 2025, American Journal of Transplantation)

   Purpose: AI models for kidney transplant acceptance must be rigorously evaluated for bias to ensure equitable healthcare access. This study investigates demographic and clinical biases in the Final Acceptance Model (FAM), a donor-recipient matching deep learning model that complements surgeons’ decision-making process in predicting whether to accept available kidneys for their patients with end of stage renal disorder. Methods: AI models for kidney transplant acceptance must be rigorously evaluated for bias to ensure equitable healthcare access. This study investigates demographic and clinical biases in the Final Acceptance Model (FAM), a donor-recipient matching deep learning model that complements surgeons’ decision-making process in predicting whether to accept available kidneys for their patients with end of stage renal disorder. Results: There is no significant racial bias in the model’s predictions (p=1.0), indicating consistent outcome across all racial combinations between donors and recipients. Gender-related effects as shown in Figure 1, while statistically significant (p=0.008), showed minimal practical impact with mean differences below 1% in prediction probabilities. Significant difference Clinical factors involving diabetes and hypertension showed significant difference (p=4.21e-19). The combined presence of diabetes and hypertension in donors showed the largest effect on predictions (mean difference up to -0.0173, p<0.05), followed by diabetes-only conditions in donors (mean difference up to -0.0166, p<0.05). These variations in clinical factor predictions showed bias against groups with comorbidities. Conclusions: The biases observed in the model highlight the need to improve the algorithm to ensure absolute fairness in prediction. 

   more » « less
3. AI fairness in practice: Paradigm, challenges, and prospects

   https://doi.org/10.1002/aaai.12189  

   Zhang, Wenbin (September 2024, AI Magazine)

   Abstract Understanding and correcting algorithmic bias in artificial intelligence (AI) has become increasingly important, leading to a surge in research on AI fairness within both the AI community and broader society. Traditionally, this research operates within the constrained supervised learning paradigm, assuming the presence of class labels, independent and identically distributed (IID) data, and batch‐based learning necessitating the simultaneous availability of all training data. However, in practice, class labels may be absent due to censoring, data is often represented using non‐IID graph structures that capture connections among individual units, and data can arrive and evolve over time. These prevalent real‐world data representations limit the applicability of existing fairness literature, which typically addresses fairness in static and tabular supervised learning settings. This paper reviews recent advances in AI fairness aimed at bridging these gaps for practical deployment in real‐world scenarios. Additionally, opportunities are envisioned by highlighting the limitations and significant potential for real applications. 

   more » « less
4. How Experienced Designers of Enterprise Applications Engage AI as a Design Material

   https://doi.org/10.1145/3491102.3517491  

   Yildirim, Nur; Kass, Alex; Tung, Teresa; Upton, Connor; Costello, Donnacha; Giusti, Robert; Lacin, Sinem; Lovic, Sara; O'Neill, James M; Meehan, Rudi O'Reilly; et al (April 2022, CHI '22: CHI Conference on Human Factors in Computing Systems)

   HCI research has explored AI as a design material, suggesting that designers can envision AI’s design opportunities to improve UX. Recent research claimed that enterprise applications offer an opportunity for AI innovation at the user experience level. We conducted design workshops to explore the practices of experienced designers who work on cross-functional AI teams in the enterprise. We discussed how designers successfully work with and struggle with AI. Our findings revealed that designers can innovate at the system and service levels. We also discovered that making a case for an AI feature’s return on investment is a barrier for designers when they propose AI concepts and ideas. Our discussions produced novel insights on designers’ role on AI teams, and the boundary objects they used for collaborating with data scientists. We discuss the implications of these findings as opportunities for future research aiming to empower designers in working with data and AI. 

   more » « less
5. Trustworthy Distributed AI Systems: Robustness, Privacy, and Governance

   https://doi.org/10.1145/3645102  

   Wei, Wenqi; Liu, Ling (February 2024, ACM Computing Surveys)

   Emerging Distributed AI systems are revolutionizing big data computing and data processing capabilities with growing economic and societal impact. However, recent studies have identified new attack surfaces and risks caused by security, privacy, and fairness issues in AI systems. In this paper, we review representative techniques, algorithms, and theoretical foundations for trustworthy distributed AI through robustness guarantee, privacy protection, and fairness awareness in distributed learning. We first provide a brief overview of alternative architectures for distributed learning, discuss inherent vulnerabilities for security, privacy, and fairness of AI algorithms in distributed learning, and analyze why these problems are present in distributed learning regardless of specific architectures. Then we provide a unique taxonomy of countermeasures for trustworthy distributed AI, covering (1) robustness to evasion attacks and irregular queries at inference, and robustness to poisoning attacks, Byzantine attacks, and irregular data distribution during training; (2) privacy protection during distributed learning and model inference at deployment; and (3) AI fairness and governance with respect to both data and models. We conclude with a discussion on open challenges and future research directions toward trustworthy distributed AI, such as the need for trustworthy AI policy guidelines, the AI responsibility-utility co-design, and incentives and compliance. 

   more » « less