skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Attention:

The NSF Public Access Repository (PAR) system and access will be unavailable from 10:00 PM ET on Friday, February 6 until 10:00 AM ET on Saturday, February 7 due to maintenance. We apologize for the inconvenience.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: A machine learning approach to identifying suicide risk among text-based crisis counseling encounters
IntroductionWith the increasing utilization of text-based suicide crisis counseling, new means of identifying at risk clients must be explored. Natural language processing (NLP) holds promise for evaluating the content of crisis counseling; here we use a data-driven approach to evaluate NLP methods in identifying client suicide risk. MethodsDe-identified crisis counseling data from a regional text-based crisis encounter and mobile tipline application were used to evaluate two modeling approaches in classifying client suicide risk levels. A manual evaluation of model errors and system behavior was conducted. ResultsThe neural model outperformed a term frequency-inverse document frequency (tf-idf) model in the false-negative rate. While 75% of the neural model’s false negative encounters had some discussion of suicidality, 62.5% saw a resolution of the client’s initial concerns. Similarly, the neural model detected signals of suicidality in 60.6% of false-positive encounters. DiscussionThe neural model demonstrated greater sensitivity in the detection of client suicide risk. A manual assessment of errors and model performance reflected these same findings, detecting higher levels of risk in many of the false-positive encounters and lower levels of risk in many of the false negatives. NLP-based models can detect the suicide risk of text-based crisis encounters from the encounter’s content.  more » « less
Award ID(s):
1822877
PAR ID:
10501428
Author(s) / Creator(s):
; ; ; ; ; ;
Publisher / Repository:
Frontiers in Psychiatry
Date Published:
Journal Name:
Frontiers in Psychiatry
Volume:
14
ISSN:
1664-0640
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ; ; ; ; et al (, Bipolar Disorders)
    ABSTRACT ObjectivesMost bipolar disorder (BD) patients initially present with depressive symptoms, resulting in a delayed diagnosis of BD and poor clinical outcomes. This study aims to identify features predictive of the conversion from Major Depressive Disorder (MDD) to BD by leveraging electronic health record (EHR) data from the Clínica San Juan de Dios Manizales in Colombia. MethodsWe employed a multivariable Cox regression model to identify important predictors of conversion from MDD to BD. ResultsAnalyzing 15 years of EHR data from 13,607 patients diagnosed with MDD, a total of 1610 (11.8%) transitioned to BD. Predictive features of the conversion to BD included severity of the initial MDD episode, presence of psychosis and hospitalization at first episode, family history of BD, and female gender. Additionally, we observed associations with medication classes (positive associations with prescriptions of mood stabilizers, antipsychotics, and negative associations with antidepressants) and a positive association with suicidality, a feature derived from natural language processing (NLP) of clinical notes. Together, these risk factors predicted BD conversion within 5 years of the initial MDD diagnosis, with a recall of 72% and a precision of 38%. ConclusionsOur study confirms previously identified risk factors identified through registry‐based studies (female gender and psychotic depression at the index MDD episode) and identifies novel ones (suicidality extracted from clinical notes). These results simultaneously demonstrate the validity of using EHR data for predicting BD conversion and underscore its potential for the identification of novel risk factors, thereby improving early diagnosis. 
    more » « less
  2. ; ; ; ; ; ; (, Frontiers in Endocrinology)
    IntroductionPredictive models have been used to aid early diagnosis of PCOS, though existing models are based on small sample sizes and limited to fertility clinic populations. We built a predictive model using machine learning algorithms based on an outpatient population at risk for PCOS to predict risk and facilitate earlier diagnosis, particularly among those who meet diagnostic criteria but have not received a diagnosis. MethodsThis is a retrospective cohort study from a SafetyNet hospital’s electronic health records (EHR) from 2003-2016. The study population included 30,601 women aged 18-45 years without concurrent endocrinopathy who had any visit to Boston Medical Center for primary care, obstetrics and gynecology, endocrinology, family medicine, or general internal medicine. Four prediction outcomes were assessed for PCOS. The first outcome was PCOS ICD-9 diagnosis with additional model outcomes of algorithm-defined PCOS. The latter was based on Rotterdam criteria and merging laboratory values, radiographic imaging, and ICD data from the EHR to define irregular menstruation, hyperandrogenism, and polycystic ovarian morphology on ultrasound. ResultsWe developed predictive models using four machine learning methods: logistic regression, supported vector machine, gradient boosted trees, and random forests. Hormone values (follicle-stimulating hormone, luteinizing hormone, estradiol, and sex hormone binding globulin) were combined to create a multilayer perceptron score using a neural network classifier. Prediction of PCOS prior to clinical diagnosis in an out-of-sample test set of patients achieved an average AUC of 85%, 81%, 80%, and 82%, respectively in Models I, II, III and IV. Significant positive predictors of PCOS diagnosis across models included hormone levels and obesity; negative predictors included gravidity and positive bHCG. ConclusionMachine learning algorithms were used to predict PCOS based on a large at-risk population. This approach may guide early detection of PCOS within EHR-interfaced populations to facilitate counseling and interventions that may reduce long-term health consequences. Our model illustrates the potential benefits of an artificial intelligence-enabled provider assistance tool that can be integrated into the EHR to reduce delays in diagnosis. However, model validation in other hospital-based populations is necessary. 
    more » « less
  3. ; ; ; ; ; ; (, JAMIA Open)
    Abstract BackgroundWhile most health-care providers now use electronic health records (EHRs) to document clinical care, many still treat them as digital versions of paper records. As a result, documentation often remains unstructured, with free-text entries in progress notes. This limits the potential for secondary use and analysis, as machine-learning and data analysis algorithms are more effective with structured data. ObjectiveThis study aims to use advanced artificial intelligence (AI) and natural language processing (NLP) techniques to improve diagnostic information extraction from clinical notes in a periodontal use case. By automating this process, the study seeks to reduce missing data in dental records and minimize the need for extensive manual annotation, a long-standing barrier to widespread NLP deployment in dental data extraction. Materials and MethodsThis research utilizes large language models (LLMs), specifically Generative Pretrained Transformer 4, to generate synthetic medical notes for fine-tuning a RoBERTa model. This model was trained to better interpret and process dental language, with particular attention to periodontal diagnoses. Model performance was evaluated by manually reviewing 360 clinical notes randomly selected from each of the participating site’s dataset. ResultsThe results demonstrated high accuracy of periodontal diagnosis data extraction, with the sites 1 and 2 achieving a weighted average score of 0.97-0.98. This performance held for all dimensions of periodontal diagnosis in terms of stage, grade, and extent. DiscussionSynthetic data effectively reduced manual annotation needs while preserving model quality. Generalizability across institutions suggests viability for broader adoption, though future work is needed to improve contextual understanding. ConclusionThe study highlights the potential transformative impact of AI and NLP on health-care research. Most clinical documentation (40%-80%) is free text. Scaling our method could enhance clinical data reuse. 
    more » « less
  4. ; ; ; ; ; ; (, Addiction)
    Abstract AimsTo develop machine‐learning algorithms for predicting the risk of a hospitalization or emergency department (ED) visit for opioid use disorder (OUD) (i.e. OUD acute events) in Pennsylvania Medicaid enrollees in the Opioid Use Disorder Centers of Excellence (COE) program and to evaluate the fairness of model performance across racial groups. MethodsWe studied 20 983 United States Medicaid enrollees aged 18 years or older who had COE visits between April 2019 and March 2021. We applied multivariate logistic regression, least absolute shrinkage and selection operator models, random forests, and eXtreme Gradient Boosting (XGB), to predict OUD acute events following the initial COE visit. Our models included predictors at the system, patient, and regional levels. We assessed model performance using multiple metrics by racial groups. Individuals were divided into a low, medium and high‐risk group based on predicted risk scores. ResultsThe training (n = 13 990) and testing (n = 6993) samples displayed similar characteristics (mean age 38.1 ± 9.3 years, 58% male, 80% White enrollees) with 4% experiencing OUD acute events at baseline. XGB demonstrated the best prediction performance (C‐statistic = 76.6% [95% confidence interval = 75.6%–77.7%] vs. 72.8%–74.7% for other methods). At the balanced cutoff, XGB achieved a sensitivity of 68.2%, specificity of 70.0%, and positive predictive value of 8.3%. The XGB model classified the testing sample into high‐risk (6%), medium‐risk (30%), and low‐risk (63%) groups. In the high‐risk group, 40.7% had OUD acute events vs. 16.5% and 5.0% in the medium‐ and low‐risk groups. The high‐ and medium‐risk groups captured 44% and 26% of individuals with OUD events. The XGB model exhibited lower false negative rates and higher false positive rates in racial/ethnic minority groups than White enrollees. ConclusionsNew machine‐learning algorithms perform well to predict risks of opioid use disorder (OUD) acute care use among United States Medicaid enrollees and improve fairness of prediction across racial and ethnic groups compared with previous OUD‐related models. 
    more » « less
  5. ; ; ; ; (, Sports Medicine)
    Abstract BackgroundEmerging evidence indicates an elevated risk of post-concussion musculoskeletal injuries in collegiate athletes; however, identifying athletes at highest risk remains to be elucidated. ObjectiveThe purpose of this study was to model post-concussion musculoskeletal injury risk in collegiate athletes by integrating a comprehensive set of variables by machine learning. MethodsA risk model was developed and tested on a dataset of 194 athletes (155 in the training set and 39 in the test set) with 135 variables entered into the analysis, which included participant’s heath and athletic history, concussion injury and recovery-specific criteria, and outcomes from a diverse array of concussion assessments. The machine learning approach involved transforming variables by the weight of evidence method, variable selection using L1-penalized logistic regression, model selection via the Akaike Information Criterion, and a final L2-regularized logistic regression fit. ResultsA model with 48 predictive variables yielded significant predictive performance of subsequent musculoskeletal injury with an area under the curve of 0.82. Top predictors included cognitive, balance, and reaction at baseline and acute timepoints. At a specified false-positive rate of 6.67%, the model achieves a true-positive rate (sensitivity) of 79% and a precision (positive predictive value) of 95% for identifying at-risk athletes via a well-calibrated composite risk score. ConclusionsThese results support the development of a sensitive and specific injury risk model using standard data combined with a novel methodological approach that may allow clinicians to target high injury risk student athletes. The development and refinement of predictive models, incorporating machine learning and utilizing comprehensive datasets, could lead to improved identification of high-risk athletes and allow for the implementation of targeted injury risk reduction strategies by identifying student athletes most at risk for post-concussion musculoskeletal injury. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Text Encoded Conversion
  • XML
  • Save / Share this Record
  • Send to Email