An official website of the United States government
Importance The marketing of health care devices enabled for use with artificial intelligence (AI) or machine learning (ML) is regulated in the US by the US Food and Drug Administration (FDA), which is responsible for approving and regulating medical devices. Currently, there are no uniform guidelines set by the FDA to regulate AI- or ML-enabled medical devices, and discrepancies between FDA-approved indications for use and device marketing require articulation. Objective To explore any discrepancy between marketing and 510(k) clearance of AI- or ML-enabled medical devices. Evidence Review This systematic review was a manually conducted survey of 510(k) approval summaries and accompanying marketing materials of devices approved between November 2021 and March 2022, conducted between March and November 2022, following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline. Analysis focused on the prevalence of discrepancies between marketing and certification material for AI/ML enabled medical devices. Findings A total of 119 FDA 510(k) clearance summaries were analyzed in tandem with their respective marketing materials. The devices were taxonomized into 3 individual categories of adherent, contentious, and discrepant devices. A total of 15 devices (12.61%) were considered discrepant, 8 devices (6.72%) were considered contentious, and 96 devices (84.03%) were consistent between marketing and FDA 510(k) clearance summaries. Most devices were from the radiological approval committees (75 devices [82.35%]), with 62 of these devices (82.67%) adherent, 3 (4.00%) contentious, and 10 (13.33%) discrepant; followed by the cardiovascular device approval committee (23 devices [19.33%]), with 19 of these devices (82.61%) considered adherent, 2 contentious (8.70%) and 2 discrepant (8.70%). The difference between these 3 categories in cardiovascular and radiological devices was statistically significant ( P < .001). Conclusions and Relevance In this systematic review, low adherence rates within committees were observed most often in committees with few AI- or ML-enabled devices. and discrepancies between clearance documentation and marketing material were present in one-fifth of devices surveyed.
more »
« less
Current clinical applications of AI in Radiology and their best-supporting evidence
Purpose: Despite tremendous gains from deep learning and the promise of AI in medicine to improve diagnosis and save costs, there exists a large translational gap to implement and use AI products in real-world clinical situations. Adoption of standards like the TRIPOD, CONSORT, and CLAIM checklists is increasing to improve the peer review process and reporting of AI tools. However, no such standards exist for product level review. Methods: A review of the clinical trials shows a paucity of evidence for radiology AI products; thus, we developed a 10-question assessment tool for reviewing AI products with an emphasis on their validation and result dissemination. We applied the assessment tool to commercial and open-source algorithms used for diagnosis to extract evidence on the clinical utility of the tools. Results: We find that there is limited technical information on methodologies for FDA approved algorithms compared to open source products, likely due to concerns of intellectual property. Furthermore, we find that FDA approved products use much smaller datasets compared to open-source AI tools, as the terms of use of public datasets are limited to academic and non-commercial entities which preclude their use in commercial products. Conclusion: Overall, we observe a broad spectrum of maturity and clinical use of AI products, but a large gap exists in exploring the actual performance of AI tools in clinical practice.
more »
« less
- Award ID(s):
- 1928481
- PAR ID:
- 10188459
- Date Published:
- Journal Name:
- Journal of the American College of Radiology
- ISSN:
- 1546-1440
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Laboratory tests seeking to improve detection of COVID-19 have been widely developed by laboratories and commercial companies. This review provides an overview of molecular and antigen tests, presents the sensitivity and specificity for 329 assays that have received US FDA Emergency Use Authorization and evaluates six sample collection methods – nasal, nasopharyngeal, oropharyngeal swabs, saliva, blood and stool. Molecular testing is preferred for diagnosis of COVID-19, but negative results do not always rule out the presence of infection, especially when clinical suspicion is high. Sensitivity and specificity ranged from 88.1 to 100% and 88 to 100%, respectively. Antigen tests may be more easy to use and rapid. However, they have reported a wide range of detection sensitivities from 16.7 to 85%, which may potentially yield many false-negative results.more » « less
-
(TSFAM) model, an adaptive human-AI teaming framework designed to enhance hard-to-place kidney acceptance decision-making by integrating transplant surgeons’ individualized expertise with advanced AI analytics (Figure 1). Methods: TSFAM is an innovative solution for complex issues in kidney transplant decision-making support. It employs fuzzy associative memory to capture and codify unique decision-making rules of transplant surgeons. Using the Deceased Donor Organ Assessment (DDOA) and Final Acceptance AI models designed to evaluate hard-to-place kidneys, TSFAM integrates fuzzy logic with deep learning techniques to manage inherent uncertainties in donor organ assessments. Surgeon-specifi c ontologies and membership functions are extracted through interviews. Similar to how a pain scale is used for understanding patients, an ontology ambiguity scale is used to develop surgeon rules (Figure 2). Fuzzy logic captures ambiguity and enables the model to adapt to evolving clinical, environmental, and policy conditions. The structured incorporation of human expertise ensures decision support remains closely aligned with local clinical practices and global best evidence. Results: This novel framework incorporates human expertise into AI decisionmaking tools to support donor organ acceptance in transplantation. Integrating surgeon-defi ned criteria into a robust decision-support tool enhances accuracy and transparency of organ allocation decision-making support. TSFAM bridges the gap between data-driven models and nuanced judgment required in complex clinical scenarios, fostering trust and promoting responsible AI adoption. Conclusions: TSFAM fuses deep learning analytics with subtleties of human expertise for a promising pathway to improve decision-making support in transplant surgery. The framework enhances clinical assessment and sets a precedent for future systems prioritizing human-AI collaboration. Prospective studies will focus on clinical implementation with dynamic interfaces for a more patient-centered, evidencebased model in organ transplantation. The intent is for this approach to be adaptable to individual case scenarios and the diverse needs of key transplant team membersmore » « less
-
Iris is one of the most widely used biometric modalities because of its uniqueness, high matching performance, and inherently secure nature. Iris segmentation is an essential preliminary step for iris-based biometric authentication. The authentication accuracy is directly connected with the iris segmentation accuracy. In the last few years, deep-learning-based iris segmentation methodologies have increasingly been adopted because of their ability to handle challenging segmentation tasks and their advantages over traditional segmentation techniques. However, the biggest challenge to the biometric community is the scarcity of open-source resources for adoption for application and reproducibility. This review provides a comprehensive examination of available open-source iris segmentation resources, including datasets, algorithms, and tools. In the process, we designed three U-Net and U-Net++ architecture-influenced segmentation algorithms as standard benchmarks, trained them on a large composite dataset (>45K samples), and created 1K manually segmented ground truth masks. Overall, eleven state-of-the-art algorithms were benchmarked against five datasets encompassing multiple sensors, environmental conditions, demography, and illumination. This assessment highlights the strengths, limitations, and practical implications of each method and identifies gaps that future studies should address to improve segmentation accuracy and robustness. To foster future research, all resources developed during this work would be made publicly available.more » « less
-
Recent developments in AI have provided assisting tools to support pathologists’ diagnoses. However, it remains challenging to incorporate such tools into pathologists’ practice; one main concern is AI’s insufficient workflow integration with medical decisions. We observed pathologists’ examination and discovered that the main hindering factor to integrate AI is its incompatibility with pathologists’ workflow. To bridge the gap between pathologists and AI, we developed a human-AI collaborative diagnosis tool — xPath — that shares a similar examination process to that of pathologists, which can improve AI’s integration into their routine examination. The viability of xPath is confirmed by a technical evaluation and work sessions with twelve medical professionals in pathology. This work identifies and addresses the challenge of incorporating AI models into pathology, which can offer first-hand knowledge about how HCI researchers can work with medical professionals side-by-side to bring technological advances to medical tasks towards practical applications.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
