PurposeChallenges in teaching the engineering design process (EDP) at the high-school level, such as promoting good documentation practices, are well-documented. While developments in educational artificial intelligence (AI) systems have the potential to assist in addressing these challenges, the open-ended nature of the EDP leads to challenges that often lack the specificity required for actionable AI development. In addition, conventional educational AI systems (e.g. intelligent tutoring systems) primarily target procedural domain tasks with well-defined outcomes and problem-solving strategies, while the EDP involves open-ended problems and multiple correct solutions, making AI intervention timing and appropriateness complex. Design/methodology/approachAuthors conducted a six-week-long Research through Co-Design (RtCD) process (i.e. a co-design process rooted in Research through Design) with two experienced high-school engineering teachers to co-construct actionable insight in the form of AI intervention points (AI-IPs) in engineering education where an AI system can effectively intervene to support them while highlighting their pedagogical practices. FindingsThis paper leveraged the design of task models to iteratively refine our prior understanding of teachers’ experiences with teaching the EDP into three AI-IPs related to documentation, ephemeral interactions between teachers and students and disruptive failures that can serve as a focus for intelligent educational system designs. Originality/valueThis paper discusses the implications of these AI-IPs for designing educational AI systems to support engineering education as well as the importance of leveraging RtCD methodologies to engage teachers in developing intelligent educational systems that align with their needs and afford them control over computational interventions in their classrooms.
more »
« less
Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findings
Abstract ObjectivesTo evaluate the proficiency of a HIPAA-compliant version of GPT-4 in identifying actionable, incidental findings from unstructured radiology reports of Emergency Department patients. To assess appropriateness of artificial intelligence (AI)-generated, patient-facing summaries of these findings. Materials and MethodsRadiology reports extracted from the electronic health record of a large academic medical center were manually reviewed to identify non-emergent, incidental findings with high likelihood of requiring follow-up, further sub-stratified as “definitely actionable” (DA) or “possibly actionable—clinical correlation” (PA-CC). Instruction prompts to GPT-4 were developed and iteratively optimized using a validation set of 50 reports. The optimized prompt was then applied to a test set of 430 unseen reports. GPT-4 performance was primarily graded on accuracy identifying either DA or PA-CC findings, then secondarily for DA findings alone. Outputs were reviewed for hallucinations. AI-generated patient-facing summaries were assessed for appropriateness via Likert scale. ResultsFor the primary outcome (DA or PA-CC), GPT-4 achieved 99.3% recall, 73.6% precision, and 84.5% F-1. For the secondary outcome (DA only), GPT-4 demonstrated 95.2% recall, 77.3% precision, and 85.3% F-1. No findings were “hallucinated” outright. However, 2.8% of cases included generated text about recommendations that were inferred without specific reference. The majority of True Positive AI-generated summaries required no or minor revision. ConclusionGPT-4 demonstrates proficiency in detecting actionable, incidental findings after refined instruction prompting. AI-generated patient instructions were most often appropriate, but rarely included inferred recommendations. While this technology shows promise to augment diagnostics, active clinician oversight via “human-in-the-loop” workflows remains critical for clinical implementation.
more »
« less
- PAR ID:
- 10521960
- Publisher / Repository:
- AMIA
- Date Published:
- Journal Name:
- Journal of the American Medical Informatics Association
- ISSN:
- 1067-5027
- Subject(s) / Keyword(s):
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
ObjectivesTo identify what patient-related characteristics have been reported to be associated with the occurrence of shared decision-making (SDM) about treatment. DesignScoping review. Eligibility criteriaPeer-reviewed articles in English or Dutch reporting on associations between patient-related characteristics and the occurrence of SDM for actual treatment decisions. Information sourcesCOCHRANE Library, Embase, MEDLINE, PsycInfo, PubMed and Web of Science were systematically searched for articles published until 25 March 2019. ResultsThe search yielded 5289 hits of which 53 were retained. Multiple categories of patient characteristics were identified: (1) sociodemographic characteristics (eg, gender), (2) general health and clinical characteristics (eg, symptom severity), (3) psychological characteristics and coping with illness (eg, self-efficacy) and (4) SDM style or preference. Many characteristics showed no association or unclear relationships with SDM occurrence. For example, for female gender positive, negative and, most frequently, non-significant associations were seen. ConclusionsA large variety of patient-related characteristics have been studied, but for many the association with SDM occurrence remains unclear. The results will caution often-made assumptions about associations and provide an important step to target effective interventions to foster SDM with all patients.more » « less
-
Abstract The concentration of dopamine (DA) and tyrosine (Tyr) reflects the condition of patients with Parkinson's disease, whereas moderate paracetamol (PA) can help relieve their pain. Therefore, real‐time measurements of these bioanalytes have important clinical implications for patients with Parkinson's disease. However, previous sensors suffer from either limited sensitivity or complex fabrication and integration processes. This work introduces a simple and cost‐effective method to prepare high‐quality, flexible titanium dioxide (TiO2) thin films with highly reactive (001)‐facets. The as‐fabricated TiO2film supported by a carbon cloth electrode (i.e., TiO2–CC) allows excellent electrochemical specificity and sensitivity to DA (1.390 µA µM−1 cm−2), Tyr (0.126 µA µM−1 cm−2), and PA (0.0841 µA µM−1 cm−2). More importantly, accurate DA concentration in varied pH conditions can be obtained by decoupling them within a single differential pulse voltammetry measurement without additional sensing units. The TiO2–CC electrochemical sensor can be integrated into a smart diaper to detect the trace amount of DA or an integrated skin‐interfaced patch with microfluidic sampling and wireless transmission units for real‐time detection of the sweat Try and PA concentration. The wearable sensor based on TiO2–CC prepared by facile manufacturing methods holds great potential in the daily health monitoring and care of patients with neurological disorders.more » « less
-
BackgroundGenerative artificial intelligence (AI) use has been suggested to have adverse mental health consequences but a causal relationship has not been examined. ObjectiveTo simulate a randomised controlled trial of AI use in a work, school or personal context by applying target trial emulation to multiple waves of data from a nationally representative survey. MethodsWe conducted a target trial emulation using non-probability survey data from three waves of a nationally representative survey conducted between 18 June 2024 and 8 January 2025. Participants aged ≥18 years reported generative AI use frequency at baseline. High-frequency use was defined as multiple times per week or more. The primary outcome was depressive symptom severity measured using the Patient Health Questionnaire 9-item (PHQ-9) at follow-up. Generalised causal forests assessed heterogeneity of treatment effects. FindingsAmong 19 099 participants assessed at baseline, 2862 (15.0%) reported AI use at least multiple times per week. A subset of 3109 (16.3%) returned for follow-up. In the primary weighted analysis, high-frequency use was not significantly associated with change in PHQ-9 score at follow-up (mean difference −0.18, 95% CI −0.94 to 0.59; p=0.65). Multiple sensitivity analyses using alternate outcome definitions also did not identify significant causal effects. Generalised causal forests yielded no significant evidence of heterogeneity of effect (p=0.81). ConclusionsIn an emulated randomised trial among US adults, generative AI use was not associated with subsequent depressive symptoms. This result does not support the premise that AI use causes greater depressive symptoms, although adverse outcomes among vulnerable individuals cannot be excluded. Clinical implicationsAI use is unlikely to cause increased depressive symptoms among most US adults. Continued monitoring should clarify potential risks among vulnerable populations.more » « less
-
Wren, Jonathan (Ed.)Abstract MotivationThousands of genomes are publicly available, however, most genes in those genomes have poorly defined functions. This is partly due to a gap between previously published, experimentally characterized protein activities and activities deposited in databases. This activity deposition is bottlenecked by the time-consuming biocuration process. The emergence of large language models presents an opportunity to speed up the text-mining of protein activities for biocuration. ResultsWe developed FuncFetch—a workflow that integrates NCBI E-Utilities, OpenAI’s GPT-4, and Zotero—to screen thousands of manuscripts and extract enzyme activities. Extensive validation revealed high precision and recall of GPT-4 in determining whether the abstract of a given paper indicates the presence of a characterized enzyme activity in that paper. Provided the manuscript, FuncFetch extracted data such as species information, enzyme names, sequence identifiers, substrates, and products, which were subjected to extensive quality analyses. Comparison of this workflow against a manually curated dataset of BAHD acyltransferase activities demonstrated a precision/recall of 0.86/0.64 in extracting substrates. We further deployed FuncFetch on nine large plant enzyme families. Screening 26 543 papers, FuncFetch retrieved 32 605 entries from 5459 selected papers. We also identified multiple extraction errors including incorrect associations, nontarget enzymes, and hallucinations, which highlight the need for further manual curation. The BAHD activities were verified, resulting in a comprehensive functional fingerprint of this family and revealing that ∼70% of the experimentally characterized enzymes are uncurated in the public domain. FuncFetch represents an advance in biocuration and lays the groundwork for predicting the functions of uncharacterized enzymes. Availability and implementationCode and minimally curated activities are available at: https://github.com/moghelab/funcfetch and https://tools.moghelab.org/funczymedb.more » « less
An official website of the United States government

