An official website of the United States government
BackgroundMetamodels can address some of the limitations of complex simulation models by formulating a mathematical relationship between input parameters and simulation model outcomes. Our objective was to develop and compare the performance of a machine learning (ML)–based metamodel against a conventional metamodeling approach in replicating the findings of a complex simulation model. MethodsWe constructed 3 ML-based metamodels using random forest, support vector regression, and artificial neural networks and a linear regression-based metamodel from a previously validated microsimulation model of the natural history hepatitis C virus (HCV) consisting of 40 input parameters. Outcomes of interest included societal costs and quality-adjusted life-years (QALYs), the incremental cost-effectiveness (ICER) of HCV treatment versus no treatment, cost-effectiveness analysis curve (CEAC), and expected value of perfect information (EVPI). We evaluated metamodel performance using root mean squared error (RMSE) and Pearson’s R2on the normalized data. ResultsThe R2values for the linear regression metamodel for QALYs without treatment, QALYs with treatment, societal cost without treatment, societal cost with treatment, and ICER were 0.92, 0.98, 0.85, 0.92, and 0.60, respectively. The corresponding R2values for our ML-based metamodels were 0.96, 0.97, 0.90, 0.95, and 0.49 for support vector regression; 0.99, 0.83, 0.99, 0.99, and 0.82 for artificial neural network; and 0.99, 0.99, 0.99, 0.99, and 0.98 for random forest. Similar trends were observed for RMSE. The CEAC and EVPI curves produced by the random forest metamodel matched the results of the simulation output more closely than the linear regression metamodel. ConclusionsML-based metamodels generally outperformed traditional linear regression metamodels at replicating results from complex simulation models, with random forest metamodels performing best. HighlightsDecision-analytic models are frequently used by policy makers and other stakeholders to assess the impact of new medical technologies and interventions. However, complex models can impose limitations on conducting probabilistic sensitivity analysis and value-of-information analysis, and may not be suitable for developing online decision-support tools. Metamodels, which accurately formulate a mathematical relationship between input parameters and model outcomes, can replicate complex simulation models and address the above limitation. The machine learning–based random forest model can outperform linear regression in replicating the findings of a complex simulation model. Such a metamodel can be used for conducting cost-effectiveness and value-of-information analyses or developing online decision support tools.
more »
« less
This content will become publicly available on September 3, 2025
Soybean prediction using computationally efficient Bayesian spatial regression models and satellite imagery
Abstract Preharvest yield estimates can be used for harvest planning, marketing, and prescribing in‐season fertilizer and pesticide applications. One approach that is being widely tested is the use of machine learning (ML) or artificial intelligence (AI) algorithms to estimate yields. However, one barrier to the adoption of this approach is that ML/AI algorithms behave as a black block. An alternative approach is to create an algorithm using Bayesian statistics. In Bayesian statistics, prior information is used to help create the algorithm. However, algorithms based on Bayesian statistics are not often computationally efficient. The objective of the current study was to compare the accuracy and computational efficiency of four Bayesian models that used different assumptions to reduce the execution time. In this paper, the Bayesian multiple linear regression (BLR), Bayesian spatial, Bayesian skewed spatial regression, and the Bayesian nearest neighbor Gaussian process (NNGP) models were compared with ML non‐Bayesian random forest model. In this analysis, soybean (Glycine max) yields were the response variable (y), and spaced‐based blue, green, red, and near‐infrared reflectance that was measured with the PlanetScope satellite were the predictor (x). Among the models tested, the Bayesian (NNGP;R2‐testing = 0.485) model, which captures the short‐range correlation, outperformed the (BLR;R2‐testing = 0.02), Bayesian spatial regression (SRM;R2‐testing = 0.087), and Bayesian skewed spatial regression (sSRM;R2‐testing = 0.236) models. However, associated with improved accuracy was an increase in run time from 534 s for the BLR model to 2047 s for the NNGP model. These data show that relatively accurate within‐field yield estimates can be obtained without sacrificing computational efficiency and that the coefficients have biological meaning. However, all Bayesian models had lowerR2values and higher execution times than the random forest model.
more »
« less
- Award ID(s):
- 2202706
- PAR ID:
- 10539514
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Agronomy Journal
- Volume:
- 116
- Issue:
- 6
- ISSN:
- 0002-1962
- Format(s):
- Medium: X Size: p. 2841-2849
- Size(s):
- p. 2841-2849
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract A key challenge in spatial data science is the analysis for massive spatially‐referenced data sets. Such analyses often proceed from Gaussian process specifications that can produce rich and robust inference, but involve dense covariance matrices that lack computationally exploitable structures. Recent developments in spatial statistics offer a variety of massively scalable approaches. Bayesian inference and hierarchical models, in particular, have gained popularity due to their richness and flexibility in accommodating spatial processes. Our current contribution is to provide computationally efficient exact algorithms for spatial interpolation of massive data sets using scalable spatial processes. We combine low‐rank Gaussian processes with efficient sparse approximations. Following recent work by Zhang et al. (2019), we model the low‐rank process using a Gaussian predictive process (GPP) and the residual process as a sparsity‐inducing nearest‐neighbor Gaussian process (NNGP). A key contribution here is to implement these models using exact conjugate Bayesian modeling to avoid expensive iterative algorithms. Through the simulation studies, we evaluate performance of the proposed approach and the robustness of our models, especially for long range prediction. We implement our approaches for remotely sensed light detection and ranging (LiDAR) data collected over the US Forest Service Tanana Inventory Unit (TIU) in a remote portion of Interior Alaska.more » « less
-
Abstract Private groundwater wells can be unmonitored sources of contaminated water that can harm human health. Developing models that predict exposure could allow residents to take action to reduce risk. Machine learning models have been successful in predicting nitrate contamination using geospatial information such as proximity to nitrate sources, but previous models have not considered meteorological factors that change temporally. In this study, we test random forest (regression and classification) and linear regression models to predict nitrate contamination using rainfall, temperature, and readily available soil parameters. We trained and tested models for (1) all of North Carolina, (2) each geographic region in North Carolina, (3) a three‐county region with a high density of animal agriculture, and (4) a three‐county region with a low density of animal agriculture. All regression models had poor predictive performance (R2 < 0.09). The random forest classification model for the coastal plain showed fair agreement (Cohen'sκ = 0.23) when trying to predict whether contamination occurred. All other classification models had slight or poor predictive performance. Our results show that temporal changes in rainfall and temperature, or in combination with soil data, are not enough to predict nitrate contamination in most areas of North Carolina. The low level of contamination (<25%) measured during the study could have contributed to the poor performance of the models.more » « less
-
Abstract Gaussian process (GP) is a staple in the toolkit of a spatial statistician. Well‐documented computing roadblocks in the analysis of large geospatial datasets using GPs have now largely been mitigated via several recent statistical innovations. Nearest neighbor Gaussian process (NNGP) has emerged as one of the leading candidates for such massive‐scale geospatial analysis owing to their empirical success. This article reviews the connection of NNGP to sparse Cholesky factors of the spatial precision (inverse‐covariance) matrix. Focus of the review is on these sparse Cholesky matrices which are versatile and have recently found many diverse applications beyond the primary usage of NNGP for fast parameter estimation and prediction in the spatial (generalized) linear models. In particular, we discuss applications of sparse NNGP Cholesky matrices to address multifaceted computational issues in spatial bootstrapping, simulation of large‐scale realizations of Gaussian random fields, and extensions to nonparametric mean function estimation of a GP using random forests. We also review a sparse‐Cholesky‐based model for areal (geographically aggregated) data that addresses long‐established interpretability issues of existing areal models. Finally, we highlight some yet‐to‐be‐addressed issues of such sparse Cholesky approximations that warrant further research. This article is categorized under:Algorithms and Computational Methods > AlgorithmsAlgorithms and Computational Methods > Numerical Methodsmore » « less
-
Abstract The timeliness of monitoring is essential to algal bloom management. However, acquiring algal bio-indicators can be time-consuming and laborious, and bloom biomass data often contain a large proportion of extreme values limiting the predictive models. Therefore, to predict algal blooms from readily water quality parameters (i.e. dissolved oxygen, pH, etc), and to provide a novel solution to the modeling challenges raised by the extremely distributed biomass data, a Bayesian scale-mixture of skew-normal (SMSN) model was proposed. In this study, our SMSN model accurately predicted over-dispersed biomass variations with skewed distributions in both rivers and lakes (in-sample and out-of-sample predictionR2ranged from 0.533 to 0.706 and 0.412 to 0.742, respectively). Moreover, we successfully achieve a probabilistic assessment of algal blooms with the Bayesian framework (accuracy >0.77 and macro-F1score >0.72), which robustly decreased the classic point-prediction-based inaccuracy by up to 34%. This work presented a promising Bayesian SMSN modeling technique, allowing for real-time prediction of algal biomass variations andin-situprobabilistic assessment of algal bloom.more » « less
