skip to main content

Title: Evaluating the Effect of a COVID-19 Predictive Model to Facilitate Discharge: A Randomized Controlled Trial
Abstract Background We previously developed and validated a predictive model to help clinicians identify hospitalized adults with coronavirus disease 2019 (COVID-19) who may be ready for discharge given their low risk of adverse events. Whether this algorithm can prompt more timely discharge for stable patients in practice is unknown. Objectives The aim of the study is to estimate the effect of displaying risk scores on length of stay (LOS). Methods We integrated model output into the electronic health record (EHR) at four hospitals in one health system by displaying a green/orange/red score indicating low/moderate/high-risk in a patient list column and a larger COVID-19 summary report visible for each patient. Display of the score was pseudo-randomized 1:1 into intervention and control arms using a patient identifier passed to the model execution code. Intervention effect was assessed by comparing LOS between intervention and control groups. Adverse safety outcomes of death, hospice, and re-presentation were tested separately and as a composite indicator. We tracked adoption and sustained use through daily counts of score displays. Results Enrolling 1,010 patients from May 15, 2020 to December 7, 2020, the trial found no detectable difference in LOS. The intervention had no impact on safety indicators of death, hospice or re-presentation after discharge. The scores were displayed consistently throughout the study period but the study lacks a causally linked process measure of provider actions based on the score. Secondary analysis revealed complex dynamics in LOS temporally, by primary symptom, and hospital location. Conclusion An AI-based COVID-19 risk score displayed passively to clinicians during routine care of hospitalized adults with COVID-19 was safe but had no detectable impact on LOS. Health technology challenges such as insufficient adoption, nonuniform use, and provider trust compounded with temporal factors of the COVID-19 pandemic may have contributed to the null result. Trial registration identifier: NCT04570488.  more » « less
Award ID(s):
Author(s) / Creator(s):
; ; ; ; ; ; ; ;
Date Published:
Journal Name:
Applied Clinical Informatics
Page Range / eLocation ID:
632 to 640
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Background

    The Mexican Institute of Social Security (IMSS) is the largest health care provider in Mexico, covering about 48% of the Mexican population. In this report, we describe the epidemiological patterns related to confirmed cases, hospitalizations, intubations, and in-hospital mortality due to COVID-19 and associated factors, during five epidemic waves recorded in the IMSS surveillance system.


    We analyzed COVID-19 laboratory-confirmed cases from the Online Epidemiological Surveillance System (SINOLAVE) from March 29th, 2020, to August 27th, 2022. We constructed weekly epidemic curves describing temporal patterns of confirmed cases and hospitalizations by age, gender, and wave. We also estimated hospitalization, intubation, and hospital case fatality rates. The mean days of in-hospital stay and hospital admission delay were calculated across five pandemic waves. Logistic regression models were employed to assess the association between demographic factors, comorbidities, wave, and vaccination and the risk of severe disease and in-hospital death.


    A total of 3,396,375 laboratory-confirmed COVID-19 cases were recorded across the five waves. The introduction of rapid antigen testing at the end of 2020 increased detection and modified epidemiological estimates. Overall, 11% (95% CI 10.9, 11.1) of confirmed cases were hospitalized, 20.6% (95% CI 20.5, 20.7) of the hospitalized cases were intubated, and the hospital case fatality rate was 45.1% (95% CI 44.9, 45.3). The mean in-hospital stay was 9.11 days, and patients were admitted on average 5.07 days after symptoms onset. The most recent waves dominated by the Omicron variant had the highest incidence. Hospitalization, intubation, and mean hospitalization days decreased during subsequent waves. The in-hospital case fatality rate fluctuated across waves, reaching its highest value during the second wave in winter 2020. A notable decrease in hospitalization was observed primarily among individuals ≥ 60 years. The risk of severe disease and death was positively associated with comorbidities, age, and male gender; and declined with later waves and vaccination status.


    During the five pandemic waves, we observed an increase in the number of cases and a reduction in severity metrics. During the first three waves, the high in-hospital fatality rate was associated with hospitalization practices for critical patients with comorbidities.

    more » « less
  2. Abstract Objective

    To develop predictive models of coronavirus disease 2019 (COVID-19) outcomes, elucidate the influence of socioeconomic factors, and assess algorithmic racial fairness using a racially diverse patient population with high social needs.

    Materials and Methods

    Data included 7,102 patients with positive (RT-PCR) severe acute respiratory syndrome coronavirus 2 test at a safety-net system in Massachusetts. Linear and nonlinear classification methods were applied. A score based on a recurrent neural network and a transformer architecture was developed to capture the dynamic evolution of vital signs. Combined with patient characteristics, clinical variables, and hospital occupancy measures, this dynamic vital score was used to train predictive models.


    Hospitalizations can be predicted with an area under the receiver-operating characteristic curve (AUC) of 92% using symptoms, hospital occupancy, and patient characteristics, including social determinants of health. Parsimonious models to predict intensive care, mechanical ventilation, and mortality that used the most recent labs and vitals exhibited AUCs of 92.7%, 91.2%, and 94%, respectively. Early predictive models, using labs and vital signs closer to admission had AUCs of 81.1%, 84.9%, and 92%, respectively.


    The most accurate models exhibit racial bias, being more likely to falsely predict that Black patients will be hospitalized. Models that are only based on the dynamic vital score exhibited accuracies close to the best parsimonious models, although the latter also used laboratories.


    This large study demonstrates that COVID-19 severity may accurately be predicted using a score that accounts for the dynamic evolution of vital signs. Further, race, social determinants of health, and hospital occupancy play an important role.

    more » « less
  3. Background Participation in ambulatory cardiac rehabilitation remains low, especially among older adults. Although mobile health cardiac rehabilitation (mHealth-CR) provides a novel opportunity to deliver care, age-specific impairments may limit older adults’ uptake, and efficacy data are currently lacking. Objective This study aims to describe the design of the rehabilitation using mobile health for older adults with ischemic heart disease in the home setting (RESILIENT) trial. Methods RESILIENT is a multicenter randomized clinical trial that is enrolling patients aged ≥65 years with ischemic heart disease in a 3:1 ratio to either an intervention (mHealth-CR) or control (usual care) arm, with a target sample size of 400 participants. mHealth-CR consists of a commercially available mobile health software platform coupled with weekly exercise therapist sessions to review progress and set new activity goals. The primary outcome is a change in functional mobility (6-minute walk distance), which is measured at baseline and 3 months. Secondary outcomes are health status, goal attainment, hospital readmission, and mortality. Among intervention participants, engagement with the mHealth-CR platform will be analyzed to understand the characteristics that determine different patterns of use (eg, persistent high engagement and declining engagement). Results As of December 2021, the RESILIENT trial had enrolled 116 participants. Enrollment is projected to continue until October 2023. The trial results are expected to be reported in 2024. Conclusions The RESILIENT trial will generate important evidence about the efficacy of mHealth-CR among older adults in multiple domains and characteristics that determine the sustained use of mHealth-CR. These findings will help design future precision medicine approaches to mobile health implementation in older adults. This knowledge is especially important in light of the COVID-19 pandemic that has shifted much of health care to a remote, internet-based setting. Trial Registration NCT03978130; International Registered Report Identifier (IRRID) DERR1-10.2196/32163 
    more » « less
  4. null (Ed.)
    Background The novel coronavirus SARS-CoV-2 and its associated disease, COVID-19, have caused worldwide disruption, leading countries to take drastic measures to address the progression of the disease. As SARS-CoV-2 continues to spread, hospitals are struggling to allocate resources to patients who are most at risk. In this context, it has become important to develop models that can accurately predict the severity of infection of hospitalized patients to help guide triage, planning, and resource allocation. Objective The aim of this study was to develop accurate models to predict the mortality of hospitalized patients with COVID-19 using basic demographics and easily obtainable laboratory data. Methods We performed a retrospective study of 375 hospitalized patients with COVID-19 in Wuhan, China. The patients were randomly split into derivation and validation cohorts. Regularized logistic regression and support vector machine classifiers were trained on the derivation cohort, and accuracy metrics (F1 scores) were computed on the validation cohort. Two types of models were developed: the first type used laboratory findings from the entire length of the patient’s hospital stay, and the second type used laboratory findings that were obtained no later than 12 hours after admission. The models were further validated on a multicenter external cohort of 542 patients. Results Of the 375 patients with COVID-19, 174 (46.4%) died of the infection. The study cohort was composed of 224/375 men (59.7%) and 151/375 women (40.3%), with a mean age of 58.83 years (SD 16.46). The models developed using data from throughout the patients’ length of stay demonstrated accuracies as high as 97%, whereas the models with admission laboratory variables possessed accuracies of up to 93%. The latter models predicted patient outcomes an average of 11.5 days in advance. Key variables such as lactate dehydrogenase, high-sensitivity C-reactive protein, and percentage of lymphocytes in the blood were indicated by the models. In line with previous studies, age was also found to be an important variable in predicting mortality. In particular, the mean age of patients who survived COVID-19 infection (50.23 years, SD 15.02) was significantly lower than the mean age of patients who died of the infection (68.75 years, SD 11.83; P<.001). Conclusions Machine learning models can be successfully employed to accurately predict outcomes of patients with COVID-19. Our models achieved high accuracies and could predict outcomes more than one week in advance; this promising result suggests that these models can be highly useful for resource allocation in hospitals. 
    more » « less
  5. Abstract

    Adverse patient safety events, unintended injuries resulting from medical therapy, were associated with 110,000 deaths in the United States in 2019. A nationwide pandemic (such as COVID-19) further challenges the ability of healthcare systems to ensure safe medication use and the pandemic’s effects on safety events remain poorly understood. Here, we investigate drug safety events across demographic groups before and during a pandemic using a dataset of 1,425,371 reports involving 2,821 drugs and 7,761 adverse events. Among 64 adverse events identified by our analyses, we find 54 increased in frequency during the pandemic, despite a 4.4% decrease in the total number of reports. Out of 53 adverse events with a pre-pandemic gender gap, 33 have seen their gap increase with the pandemic onset. We find that the number of adverse events with an increased reporting ratio is higher in adults (by 16.8%) than in older patients. Our findings have implications for safe medication use and preventable healthcare inequality in public health emergencies.

    more » « less