skip to main content


Title: Cost-Effectiveness and Value-of-Information Analysis Using Machine Learning–Based Metamodeling: A Case of Hepatitis C Treatment
Background

Metamodels can address some of the limitations of complex simulation models by formulating a mathematical relationship between input parameters and simulation model outcomes. Our objective was to develop and compare the performance of a machine learning (ML)–based metamodel against a conventional metamodeling approach in replicating the findings of a complex simulation model.

Methods

We constructed 3 ML-based metamodels using random forest, support vector regression, and artificial neural networks and a linear regression-based metamodel from a previously validated microsimulation model of the natural history hepatitis C virus (HCV) consisting of 40 input parameters. Outcomes of interest included societal costs and quality-adjusted life-years (QALYs), the incremental cost-effectiveness (ICER) of HCV treatment versus no treatment, cost-effectiveness analysis curve (CEAC), and expected value of perfect information (EVPI). We evaluated metamodel performance using root mean squared error (RMSE) and Pearson’s R2on the normalized data.

Results

The R2values for the linear regression metamodel for QALYs without treatment, QALYs with treatment, societal cost without treatment, societal cost with treatment, and ICER were 0.92, 0.98, 0.85, 0.92, and 0.60, respectively. The corresponding R2values for our ML-based metamodels were 0.96, 0.97, 0.90, 0.95, and 0.49 for support vector regression; 0.99, 0.83, 0.99, 0.99, and 0.82 for artificial neural network; and 0.99, 0.99, 0.99, 0.99, and 0.98 for random forest. Similar trends were observed for RMSE. The CEAC and EVPI curves produced by the random forest metamodel matched the results of the simulation output more closely than the linear regression metamodel.

Conclusions

ML-based metamodels generally outperformed traditional linear regression metamodels at replicating results from complex simulation models, with random forest metamodels performing best.

Highlights

Decision-analytic models are frequently used by policy makers and other stakeholders to assess the impact of new medical technologies and interventions. However, complex models can impose limitations on conducting probabilistic sensitivity analysis and value-of-information analysis, and may not be suitable for developing online decision-support tools. Metamodels, which accurately formulate a mathematical relationship between input parameters and model outcomes, can replicate complex simulation models and address the above limitation. The machine learning–based random forest model can outperform linear regression in replicating the findings of a complex simulation model. Such a metamodel can be used for conducting cost-effectiveness and value-of-information analyses or developing online decision support tools.

 
more » « less
Award ID(s):
1722665 1722614
NSF-PAR ID:
10371514
Author(s) / Creator(s):
 ;  ;  
Publisher / Repository:
SAGE Publications
Date Published:
Journal Name:
Medical Decision Making
Volume:
43
Issue:
1
ISSN:
0272-989X
Page Range / eLocation ID:
p. 68-77
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Context

    The US Food and Drug Administration (FDA) set draft voluntary targets to reduce sodium levels in processed foods. We aimed to determine cost effectiveness of meeting these draft sodium targets, from the perspective of US food system workers.

    Methods

    We employed a microsimulation cost‐effectiveness analysis using the US IMPACT Food Policy model with two scenarios: (1) short term, achieving two‐year FDA reformulation targets only, and (2) long term, achieving 10‐year FDA reformulation targets.

    We modeled four close‐to‐reality populations: food system “ever” workers; food system “current” workers in 2017; and subsets of processed food “ever” and “current” workers. Outcomes included cardiovascular disease cases prevented and postponed as well as incremental cost‐effectiveness ratio per quality‐adjusted life year (QALY) gained from 2017 to 2036.

    Findings

    Among food system ever workers, achieving long‐term sodium reduction targets could produce 20‐year health gains of approximately 180,000 QALYs (95% uncertainty interval [UI]: 150,000 to 209,000) and health cost savings of approximately $5.2 billion (95% UI: $3.5 billion to $8.3 billion), with an incremental cost‐effectiveness ratio (ICER) of $62,000 (95% UI: $1,000 to $171,000) per QALY gained. For the subset of processed food industry workers, health gains would be approximately 32,000 QALYs (95% UI: 27,000 to 37,000); cost savings, $1.0 billion (95% UI: $0.7bn to $1.6bn); and ICER, $486,000 (95% UI: $148,000 to $1,094,000) per QALY gained. Because many health benefits may occur in individuals older than 65 or the uninsured, these health savings would be shared among individuals, industry, and government.

    Conclusions

    The benefits of implementing the FDA voluntary sodium targets extend to food companies and food system workers, with the value of health gains and health care cost savings outweighing the costs of reformulation, although not for the processed food industry.

     
    more » « less
  2. Abstract

    Hepatitis C virus (HCV) is 15 times more prevalent among persons in Spain’s prisons than in the community. Recently, Spain initiated a pilot program, JAILFREE-C, to treat HCV in prisons using direct-acting antivirals (DAAs). Our aim was to identify a cost-effective strategy to scale-up HCV treatment in all prisons. Using a validated agent-based model, we simulated the HCV landscape in Spain’s prisons considering disease transmission, screening, treatment, and prison-community dynamics. Costs and disease outcomes under status quo were compared with strategies to scale-up treatment in prisons considering prioritization (HCV fibrosis stage vs. HCV prevalence of prisons), treatment capacity (2,000/year vs. unlimited) and treatment initiation based on sentence lengths (>6 months vs. any). Scaling-up treatment by treating all incarcerated persons irrespective of their sentence length provided maximum health benefits–preventing 10,200 new cases of HCV, and 8,300 HCV-related deaths between 2019–2050; 90% deaths prevented would have occurred in the community. Compared with status quo, this strategy increased quality-adjusted life year (QALYs) by 69,700 and costs by €670 million, yielding an incremental cost-effectiveness ratio of €9,600/QALY. Scaling-up HCV treatment with DAAs for the entire Spanish prison population, irrespective of sentence length, is cost-effective and would reduce HCV burden.

     
    more » « less
  3. Abstract Background

    Tension in the spinal cord is a trademark of tethered cord syndrome. Unfortunately, existing tests cannot quantify tension across the bulk of the cord, making the diagnostic evaluation of stretch ambiguous. A potential non-destructive metric for spinal cord tension is ultrasound-derived shear wave velocity (SWV). The velocity is sensitive to tissue elasticity and boundary conditions including strain. We use the term Ultrasound Tensography to describe the acoustic evaluation of tension with SWV.

    Methods

    Our solution Tethered cord Assessment with Ultrasound Tensography (TAUT) was utilized in three sub-studies: finite element simulations, a cadaveric benchtop validation, and a neurosurgical case series. The simulation computed SWV for given tensile forces. The cadaveric model with induced tension validated the SWV-tension relationship. Lastly, SWV was measured intraoperatively in patients diagnosed with tethered cords who underwent treatment (spinal column shortening). The surgery alleviates tension by decreasing the vertebral column length.

    Results

    Here we observe a strong linear relationship between tension and squared SWV across the preclinical sub-studies. Higher tension induces faster shear waves in the simulation (R2 = 0.984) and cadaveric (R2 = 0.951) models. The SWV decreases in all neurosurgical procedures (p < 0.001). Moreover, TAUT has a c-statistic of 0.962 (0.92-1.00), detecting all tethered cords.

    Conclusions

    This study presents a physical, clinical metric of spinal cord tension. Strong agreement among computational, cadaveric, and clinical studies demonstrates the utility of ultrasound-induced SWV for quantitative intraoperative feedback. This technology is positioned to enhance tethered cord diagnosis, treatment, and postoperative monitoring as it differentiates stretched from healthy cords.

     
    more » « less
  4. Abstract Purpose

    One standard method, proton resonance frequency shift, for measuring temperature using magnetic resonance imaging (MRI), in MRI‐guided surgeries, fails completely below the freezing point of water. Because of this, we have developed a new methodology for monitoring temperature with MRI below freezing. The purpose of this paper is to show that a strong temperature dependence of the nuclear relaxation timeT1in soft silicone polymers can lead to temperature‐dependent changes of MRI intensity acquired withT1weighting. We propose the use of silicone filaments inserted in tissue for measuring temperature during MRI‐guided cryoablations.

    Methods

    The temperature dependence ofT1in bio‐compatible soft silicone polymers was measured using nuclear magnetic resonance spectroscopy and MRI. Phantoms, made of bulk silicone materials and put in an MRI‐compatible thermal container with dry ice, allowed temperature measurements ranging from –60°C to + 20°C.T1‐weighted gradient echo images of the phantoms were acquired at spatially uniform temperatures and with a gradient in temperature to determine the efficacy of using these materials as temperature indicators in MRI. Ex vivo experiments on silicone rods, 4 mm in diameter, inserted in animal tissue were conducted to assess the practical feasibility of the method.

    Results

    Measurements of nuclear relaxation times of protons in soft silicone polymers show a monotonic, nearly linear, change with temperature (R2 > 0.98) and have a significant correlation with temperature (Pearson'sr > 0.99,p < 0.01). Similarly, the intensity of the MR images in these materials, taken with a gradient echo sequence, are also temperature dependent. There is again a monotonic change in MRI intensity that correlates well with the measured temperature (Pearson'sr < ‐0.98 andp < 0.01). The MRI experiments show that a temperature change of 3°C can be resolved in a distance of about 2.5 mm. Based on MRI images and external sensor calibrations for a sample with a gradient in temperature, temperature maps with 3°C isotherms are created for a bulk phantom. Experiments demonstrate that these changes in MRI intensity with temperature can also be seen in 4 mm silicone rods embedded in ex vivo animal tissue.

    Conclusions

    We have developed a new method for measuring temperature in MRI that potentially could be used during MRI‐guided cryoablation operations, reducing both procedure time and cost, and making these surgeries safer.

     
    more » « less
  5. Abstract

    This study investigates whether coupling crop modeling and machine learning (ML) improves corn yield predictions in the US Corn Belt. The main objectives are to explore whether a hybrid approach (crop modeling + ML) would result in better predictions, investigate which combinations of hybrid models provide the most accurate predictions, and determine the features from the crop modeling that are most effective to be integrated with ML for corn yield prediction. Five ML models (linear regression, LASSO, LightGBM, random forest, and XGBoost) and six ensemble models have been designed to address the research question. The results suggest that adding simulation crop model variables (APSIM) as input features to ML models can decrease yield prediction root mean squared error (RMSE) from 7 to 20%. Furthermore, we investigated partial inclusion of APSIM features in the ML prediction models and we found soil moisture related APSIM variables are most influential on the ML predictions followed by crop-related and phenology-related variables. Finally, based on feature importance measure, it has been observed that simulated APSIM average drought stress and average water table depth during the growing season are the most important APSIM inputs to ML. This result indicates that weather information alone is not sufficient and ML models need more hydrological inputs to make improved yield predictions.

     
    more » « less