skip to main content

Explore Research Products in the NSF-PAR

This content will become publicly available on May 15, 2024

Title: Bayesian approaches to include real-world data in clinical studies
Randomized clinical trials have been the mainstay of clinical research, but are prohibitively expensive and subject to increasingly difficult patient recruitment. Recently, there is a movement to use real-world data (RWD) from electronic health records, patient registries, claims data and other sources in lieu of or supplementing controlled clinical trials. This process of combining information from diverse sources calls for inference under a Bayesian paradigm. We review some of the currently used methods and a novel non-parametric Bayesian (BNP) method. Carrying out the desired adjustment for differences in patient populations is naturally done with BNP priors that facilitate understanding of and adjustment for population heterogeneities across different data sources. We discuss the particular problem of using RWD to create a synthetic control arm to supplement single-arm treatment only studies. At the core of the proposed approach is the model-based adjustment to achieve equivalent patient populations in the current study and the (adjusted) RWD. This is implemented using common atoms mixture models. The structure of such models greatly simplifies inference. The adjustment for differences in the populations can be reduced to ratios of weights in such mixtures. This article is part of the theme issue ‘Bayesian inference: challenges, perspectives, and prospects’.  more » « less
Award ID(s):
1952679
NSF-PAR ID:
10416676
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
Volume:
381
Issue:
2247
ISSN:
1364-503X
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ( , The Milbank Quarterly)
    Context

    The Medicare Shared Savings Program (MSSP) establishes incentives for participating accountable care organizations (ACOs) to lower spending for their attributed fee‐for‐service Medicare patients. Turnover in ACO physicians and patient panels has raised concerns that ACOs may be earning shared‐savings bonuses by selecting lower‐risk patients or providers with lower‐risk panels.

     Methods

    We conducted three sets of analyses of Medicare claims data. First, we estimated overall MSSP savings through 2015 using a difference‐in‐differences approach and methods that eliminated selection bias from ACO program exit or changes in the practices or physicians included in ACO contracts. We then checked for residual risk selection at the patient level. Second, we reestimated savings with methods that address undetected risk selection but could introduce bias from other sources. These included patient fixed effects, baseline or prospective assignment, and area‐level MSSP exposure to hold patient populations constant. Third, we tested for changes in provider composition or provider billing that may have contributed to bonuses, even if they were eliminated as sources of bias in the evaluation analyses.

     Findings

    MSSP participation was associated with modest and increasing annual gross savings in the 2012‐2013 entry cohorts of ACOs that reached $139 to $302 per patient by 2015. Savings in the 2014 entry cohort were small and not statistically significant. Robustness checks revealed no evidence of residual risk selection. Alternative methods to address risk selection produced results that were substantively consistent with our primary analysis but varied somewhat and were more sensitive to adjustment for patient characteristics, suggesting the introduction of bias from within‐patient changes in time‐varying characteristics. We found no evidence of ACO manipulation of provider composition or billing to inflate savings. Finally, larger savings for physician group ACOs were robust to consideration of differential changes in organizational structure among non‐ACO providers (eg, from consolidation).

     Conclusions

    Participation in the original MSSP program was associated with modest savings and not with favorable risk selection. These findings suggest an opportunity to build on early progress. Understanding the effect of new opportunities and incentives for risk selection in the revamped MSSP will be important for guiding future program reforms.

     
    more » « less
  2. ; ; ( , PLOS Digital Health)
    Pollard, Tom J. (Ed.)
    Modern predictive models require large amounts of data for training and evaluation, absence of which may result in models that are specific to certain locations, populations in them and clinical practices. Yet, best practices for clinical risk prediction models have not yet considered such challenges to generalizability. Here we ask whether population- and group-level performance of mortality prediction models vary significantly when applied to hospitals or geographies different from the ones in which they are developed. Further, what characteristics of the datasets explain the performance variation? In this multi-center cross-sectional study, we analyzed electronic health records from 179 hospitals across the US with 70,126 hospitalizations from 2014 to 2015. Generalization gap, defined as difference between model performance metrics across hospitals, is computed for area under the receiver operating characteristic curve (AUC) and calibration slope. To assess model performance by the race variable, we report differences in false negative rates across groups. Data were also analyzed using a causal discovery algorithm “Fast Causal Inference” that infers paths of causal influence while identifying potential influences associated with unmeasured variables. When transferring models across hospitals, AUC at the test hospital ranged from 0.777 to 0.832 (1st-3rd quartile or IQR; median 0.801); calibration slope from 0.725 to 0.983 (IQR; median 0.853); and disparity in false negative rates from 0.046 to 0.168 (IQR; median 0.092). Distribution of all variable types (demography, vitals, and labs) differed significantly across hospitals and regions. The race variable also mediated differences in the relationship between clinical variables and mortality, by hospital/region. In conclusion, group-level performance should be assessed during generalizability checks to identify potential harms to the groups. Moreover, for developing methods to improve model performance in new environments, a better understanding and documentation of provenance of data and health processes are needed to identify and mitigate sources of variation. 
    more » « less
  3. ; ; ; ; ( , Scientific Reports)
    Abstract Overly restrictive eligibility criteria for clinical trials may limit the generalizability of the trial results to their target real-world patient populations. We developed a novel machine learning approach using large collections of real-world data (RWD) to better inform clinical trial eligibility criteria design. We extracted patients’ clinical events from electronic health records (EHRs), which include demographics, diagnoses, and drugs, and assumed certain compositions of these clinical events within an individual’s EHRs can determine the subphenotypes—homogeneous clusters of patients, where patients within each subgroup share similar clinical characteristics. We introduced an outcome-guided probabilistic model to identify those subphenotypes, such that the patients within the same subgroup not only share similar clinical characteristics but also at similar risk levels of encountering severe adverse events (SAEs). We evaluated our algorithm on two previously conducted clinical trials with EHRs from the OneFlorida+ Clinical Research Consortium. Our model can clearly identify the patient subgroups who are more likely to suffer or not suffer from SAEs as subphenotypes in a transparent and interpretable way. Our approach identified a set of clinical topics and derived novel patient representations based on them. Each clinical topic represents a certain clinical event composition pattern learned from the patient EHRs. Tested on both trials, patient subgroup (#SAE=0) and patient subgroup (#SAE>0) can be well-separated by k-means clustering using the inferred topics. The inferred topics characterized as likely to align with the patient subgroup (#SAE>0) revealed meaningful combinations of clinical features and can provide data-driven recommendations for refining the exclusion criteria of clinical trials. The proposed supervised topic modeling approach can infer the clinical topics from the subphenotypes with or without SAEs. The potential rules for describing the patient subgroups with SAEs can be further derived to inform the design of clinical trial eligibility criteria. 
    more » « less
  4. ; ; ( , Biometrics)
    Abstract

    “Covariate adjustment” in the randomized trial context refers to an estimator of the average treatment effect that adjusts for chance imbalances between study arms in baseline variables (called “covariates”). The baseline variables could include, for example, age, sex, disease severity, and biomarkers. According to two surveys of clinical trial reports, there is confusion about the statistical properties of covariate adjustment. We focus on the analysis of covariance (ANCOVA) estimator, which involves fitting a linear model for the outcome given the treatment arm and baseline variables, and trials that use simple randomization with equal probability of assignment to treatment and control. We prove the following new (to the best of our knowledge) robustness property of ANCOVA to arbitrary model misspecification: Not only is the ANCOVA point estimate consistent (as proved by Yang and Tsiatis, 2001) but so is its standard error. This implies that confidence intervals and hypothesis tests conducted as if the linear model were correct are still asymptotically valid even when the linear model is arbitrarily misspecified, for example, when the baseline variables are nonlinearly related to the outcome or there is treatment effect heterogeneity. We also give a simple, robust formula for the variance reduction (equivalently, sample size reduction) from using ANCOVA. By reanalyzing completed randomized trials for mild cognitive impairment, schizophrenia, and depression, we demonstrate how ANCOVA can achieve variance reductions of 4 to 32%.

     
    more » « less
  5. ; ; ; ( , Statistics in Medicine)

    Interest in incorporating historical data in the clinical trial has increased with the rising cost of conducting clinical trials. The intervention arm for the current trial often requires prospective data to assess a novel treatment, and thus borrowing historical control data commensurate in distribution to current control data is motivated in order to increase the allocation ratio to the current intervention arm. Existing historical control borrowing adaptive designs adjust allocation ratios based on the commensurability assessed through study‐level summary statistics of the response agnostic of the distributions of the trial subject characteristics in the current and historical trials. This can lead to distributional imbalance of the current trial subject characteristics across the treatment arms as well as between current control data and borrowed historical control data. Such covariate imbalance may threaten the internal validity of the current trial by introducing confounding factors that affect study endpoints. In this article, we propose a Bayesian design which borrows and updates the treatment allocation ratios both covariate‐adaptively and commensurate to covariate dependently assessed similarity between the current and historical control data. We employ covariate‐dependent discrepancy parameters which are allowed to grow with the sample size and propose a regularized local regression procedure for the estimation of the parameters. The proposed design also permits the current and the historical controls to be similar to varying degree, depending on the subject level characteristics. We evaluate the proposed design extensively under the settings derived from two placebo‐controlled randomized trials on vertebral fracture risk in post‐menopausal women.

     
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email