skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • Machine learning to predict notes for chart review in the oncology setting: a proof of concept strategy for improving clinician note-writing
Title: Machine learning to predict notes for chart review in the oncology setting: a proof of concept strategy for improving clinician note-writing
Abstract ObjectiveLeverage electronic health record (EHR) audit logs to develop a machine learning (ML) model that predicts which notes a clinician wants to review when seeing oncology patients. Materials and MethodsWe trained logistic regression models using note metadata and a Term Frequency Inverse Document Frequency (TF-IDF) text representation. We evaluated performance with precision, recall, F1, AUC, and a clinical qualitative assessment. ResultsThe metadata only model achieved an AUC 0.930 and the metadata and TF-IDF model an AUC 0.937. Qualitative assessment revealed a need for better text representation and to further customize predictions for the user. DiscussionOur model effectively surfaces the top 10 notes a clinician wants to review when seeing an oncology patient. Further studies can characterize different types of clinician users and better tailor the task for different care settings. ConclusionEHR audit logs can provide important relevance data for training ML models that assist with note-writing in the oncology setting.  more » « less
Award ID(s):
2205320 2205306
PAR ID:
10504580
Author(s) / Creator(s):
; ; ; ; ; ; ;
Publisher / Repository:
Oxford University Press
Date Published:
Journal Name:
Journal of the American Medical Informatics Association
Volume:
31
Issue:
7
ISSN:
1067-5027
Format(s):
Medium: X Size: p. 1578-1582
Size(s):
p. 1578-1582
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ; (, Machine Learning for Healthcare 2023 conference)
    The large amount of time clinicians spend sifting through patient notes and documenting in electronic health records (EHRs) is a leading cause of clinician burnout. By proactively and dynamically retrieving relevant notes during the documentation process, we can reduce the effort required to find relevant patient history. In this work, we conceptualize the use of EHR audit logs for machine learning as a source of supervision of note relevance in a specific clinical context, at a particular point in time. Our evaluation focuses on the dynamic retrieval in the emergency department, a high acuity setting with unique patterns of information retrieval and note writing. We show that our methods can achieve an AUC of 0.963 for predicting which notes will be read in an individual note writing session. We additionally conduct a user study with several clinicians and find that our framework can help clinicians retrieve relevant information more efficiently. Demonstrating that our framework and methods can perform well in this demanding setting is a promising proof of concept that they will translate to other clinical settings and data modalities (e.g., labs, medications, imaging). 
    more » « less
  2. ; ; ; ; ; ; (, Frontiers in Psychiatry)
    IntroductionWith the increasing utilization of text-based suicide crisis counseling, new means of identifying at risk clients must be explored. Natural language processing (NLP) holds promise for evaluating the content of crisis counseling; here we use a data-driven approach to evaluate NLP methods in identifying client suicide risk. MethodsDe-identified crisis counseling data from a regional text-based crisis encounter and mobile tipline application were used to evaluate two modeling approaches in classifying client suicide risk levels. A manual evaluation of model errors and system behavior was conducted. ResultsThe neural model outperformed a term frequency-inverse document frequency (tf-idf) model in the false-negative rate. While 75% of the neural model’s false negative encounters had some discussion of suicidality, 62.5% saw a resolution of the client’s initial concerns. Similarly, the neural model detected signals of suicidality in 60.6% of false-positive encounters. DiscussionThe neural model demonstrated greater sensitivity in the detection of client suicide risk. A manual assessment of errors and model performance reflected these same findings, detecting higher levels of risk in many of the false-positive encounters and lower levels of risk in many of the false negatives. NLP-based models can detect the suicide risk of text-based crisis encounters from the encounter’s content. 
    more » « less
  3. ; ; ; ; ; ; ; (, Obstetrics & Gynecology)
    OBJECTIVE:To determine biomarkers other than CA 125 that could be used in identifying early-stage ovarian cancer. DATA SOURCES:Ovid MEDLINE ALL, EMBASE, Web of Science Core Collection, ScienceDirect, Clinicaltrials.gov, and CAB Direct were searched for English-language studies between January 2008 and April 2023 for the concepts of high-grade serous ovarian cancer, testing, and prevention or early diagnosis. METHODS OF STUDY SELECTION:The 5,523 related articles were uploaded to Covidence. Screening by two independent reviewers of the article abstracts led to the identification of 245 peer-reviewed primary research articles for full-text review. Full-text review by those reviewers led to the identification of 131 peer-reviewed primary research articles used for this review. TABULATION, INTEGRATION, AND RESULTSOf 131 studies, only 55 reported sensitivity, specificity, or area under the curve (AUC), with 36 of the studies reporting at least one biomarker with a specificity of 80% or greater specificity or 0.9 or greater AUC. CONCLUSION:These findings suggest that although many types of biomarkers are being tested in ovarian cancer, most have similar or worse detection rates compared with CA 125 and have the same limitations of poor detection rates in early-stage disease. However, 27.5% of articles (36/131) reported biomarkers with better sensitivity and an AUC greater than 0.9 compared with CA 125 alone and deserve further exploration. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; ; et al (, Clinical and Translational Science)
    Abstract Pharmacogenomic (PGx) biomarkers integrated using machine learning can be embedded within the electronic health record (EHR) to provide clinicians with individualized predictions of drug treatment outcomes. Currently, however, drug alerts in the EHR are largely generic (not patient‐specific) and contribute to increased clinician stress and burnout. Improving the usability of PGx alerts is an urgent need. Therefore, this work aimed to identify principles for optimal PGx alert design through a health‐system‐wide, mixed‐methods study. Clinicians representing multiple practices and care settings (N = 1062) in urban, rural, and underserved regions were invited to complete an electronic survey comparing the usability of three drug alerts for citalopram, as a case study. Alert 1 contained a generic warning of pharmacogenomic effects on citalopram metabolism. Alerts 2 and 3 provided patient‐specific predictions of citalopram efficacy with varying depth of information. Primary outcomes included the System's Usability Scale score (0–100 points) of each alert, the perceived impact of each alert on stress and decision‐making, and clinicians' suggestions for alert improvement. Secondary outcomes included the assessment of alert preference by clinician age, practice type, and geographic setting. Qualitative information was captured to provide context to quantitative information. The final cohort comprised 305 geographically and clinically diverse clinicians. A simplified, individualized alert (Alert 2) was perceived as beneficial for decision‐making and stress compared with a more detailed version (Alert 3) and the generic alert (Alert 1) regardless of age, practice type, or geographic setting. Findings emphasize the need for clinician‐guided design of PGx alerts in the era of digital medicine. 
    more » « less
  5. ; ; ; ; (, Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles)
    Given a private string q and a remote server that holds a set of public documents D, how can one of the K most relevant documents to q in D be selected and viewed without anyone (not even the server) learning anything about q or the document? This is the oblivious document ranking and retrieval problem. In this paper, we describe Coeus, a system that solves this problem. At a high level, Coeus composes two cryptographic primitives: secure matrix-vector product for scoring document relevance using the widely-used term frequency-inverse document frequency (tf-idf) method, and private information retrieval (PIR) for obliviously retrieving documents. However, Coeus reduces the time to run these protocols, thereby improving the user-perceived latency, which is a key performance metric. Coeus first reduces the PIR overhead by separating out private metadata retrieval from document retrieval, and it then scales secure matrix-vector product to tf-idf matrices with several hundred billion elements through a series of novel cryptographic refinements. For a corpus of English Wikipedia containing 5 million documents, a keyword dictionary with 64K keywords, and on a cluster of 143 machines on AWS, Coeus enables a user to obliviously rank and retrieve a document in 3.9 seconds---a 24x improvement over a baseline system. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email