An official website of the United States government
Medical vision-language models (VLMs) combine computer vision (CV) and natural language processing (NLP) to analyze visual and textual medical data. Our paper reviews recent advancements in developing VLMs specialized for healthcare, focusing on publicly available models designed for medical report generation and visual question answering (VQA). We provide background on NLP and CV, explaining how techniques from both fields are integrated into VLMs, with visual and language data often fused using Transformer-based architectures to enable effective learning from multimodal data. Key areas we address include the exploration of 18 public medical vision-language datasets, in-depth analyses of the architectures and pre-training strategies of 16 recent noteworthy medical VLMs, and comprehensive discussion on evaluation metrics for assessing VLMs' performance in medical report generation and VQA. We also highlight current challenges facing medical VLM development, including limited data availability, concerns with data privacy, and lack of proper evaluation metrics, among others, while also proposing future directions to address these obstacles. Overall, our review summarizes the recent progress in developing VLMs to harness multimodal medical data for improved healthcare applications.
more »
« less
Common Medicine for the Common Man: Picturing the “Striped Layman” in Early Vernacular Print
The disruptive figure of the “striped layman” appeared in printed texts and images around the year 1500 in the southern German lands. This essay shows how it came to represent a new kind of reader of vernacular medical publications. Exploring the illustrated books of the Strasbourg surgeon-apothecary Hieronymus Brunschwig, their local context of humanist discourse, and the visual practice of their publisher, Johann Grüninger, I argue that their oft-neglected woodcuts of people in striped clothes constitute powerful visual commentary of Brunschwig's (and others’) mission to impart medical agency and expertise to the “common man” at the eve of the Reformation.
more »
« less
- Award ID(s):
- 1734596
- PAR ID:
- 10275917
- Date Published:
- Journal Name:
- Renaissance Quarterly
- Volume:
- 74
- Issue:
- 1
- ISSN:
- 0034-4338
- Page Range / eLocation ID:
- 1 to 58
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
ABSTRACT Medical procedures are an essential part of healthcare delivery, and the acquisition of procedural skills is a critical component of medical education. Unfortunately, procedural skill is not evenly distributed among medical providers. Skills may vary within departments or institutions, and across geographic regions, depending on the provider’s training and ongoing experience. We present a mixed reality real-time communication system to increase access to procedural skill training and to improve remote emergency assistance. Our system allows a remote expert to guide a local operator through a medical procedure. RGBD cameras capture a volumetric view of the local scene including the patient, the operator, and the medical equipment. The volumetric capture is augmented onto the remote expert’s view to allow the expert to spatially guide the local operator using visual and verbal instructions. We evaluated our mixed reality communication system in a study in which experts teach the ultrasound-guided placement of a central venous catheter (CVC) to students in a simulation setting. The study compares state-of-theart video communication against our system. The results indicate that our system enhances and offers new possibilities for visual communication compared to video teleconference-based training.more » « less
-
Augmented reality (AR) has great potential for use in healthcare applications, especially remote medical training and supervision. In this paper, we analyze the usage of an AR communication system to teach a medical procedure, the placement of a central venous catheter (CVC) under ultrasound guidance. We examine various AR communication and collaboration components, including gestural communication, volumetric information, annotations, augmented objects, and augmented screens. We compare how teaching in AR differs from teaching through videoconferencing-based communication. Our results include a detailed medical training steps analysis in which we compare how verbal and visual communication differs between video and AR training. We identify procedural steps in which medical experts give visual instructions utilizing AR components. We examine the change in AR usage and interaction over time and recognize patterns between users. Moreover, AR design recommendations are given based on post-training interviews.more » « less
-
The adoption of large language models (LLMs) in healthcare has garnered significant research interest, yet their performance remains limited due to a lack of domain‐specific knowledge, medical reasoning skills, and their unimodal nature, which restricts them to text‐only inputs. To address these limitations, we propose MultiMedRes, a multimodal medical collaborative reasoning framework that simulates human physicians’ communication by incorporating a learner agent to proactively acquire information from domain‐specific expert models. MultiMedRes addresses medical multimodal reasoning problems through three steps i) Inquire: The learner agent decomposes complex medical reasoning problems into multiple domain‐specific sub‐problems; ii) Interact: The agent engages in iterative “ask‐answer” interactions with expert models to obtain domain‐specific knowledge; and iii) Integrate: The agent integrates all the acquired domain‐specific knowledge to address the medical reasoning problems (e.g., identifying the difference of disease levels and abnormality sizes between medical images). We validate the effectiveness of our method on the task of difference visual question answering for X‐ray images. The experiments show that our zero‐shot prediction achieves state‐of‐the‐art performance, surpassing fully supervised methods, which demonstrates that MultiMedRes could offer trustworthy and interpretable assistance to physicians in monitoring the treatment progression of patients, paving the way for effective human–AI interaction and collaboration.more » « less
-
Fankhauser, Sarah (Ed.)ABSTRACT Visual representations in molecular biology tend to follow a set of shared conventions for using certain shapes and symbols to convey information about the size and structure of nucleotides, genes, and chromosomes. Understanding how and why biologists use these conventions to represent DNA is a key part of visual literacy in molecular biology. Visual literacy, which is the ability to read and interpret visual representations, encompasses a set of skills that are necessary for biologists to effectively use models to communicate about molecular structures that cannot be directly observed. To gauge students’ visual literacy skills, we conducted semi-structured interviews with undergraduate students who had completed at least a year of biology courses. We asked students to draw and interpret figures of nucleotides, genes, and chromosomes, and we analyzed their drawings for adherence to conventions for representing scale and abstraction. We found that 77% of students made errors in representing scale, and 86% of students made errors in representing abstraction. We also observed that about half of the students in our sample used the conventional shapes and symbols to represent DNA in unconventional ways. These unconventional sketches may signal an incomplete understanding of the structure and function of DNA. Our findings indicate that students may need additional instructional support to interpret the conventions in common representations of DNA. We highlight opportunities for instructors to scaffold visual literacy skills into their teaching to help students better understand visual conventions for representing scale and abstraction in molecular biology.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1017/rqx.2020.315
