An official website of the United States government
Abstract Neural networks (NN) have become an important tool for prediction tasks—both regression and classification—in environmental science. Since many environmental-science problems involve life-or-death decisions and policy making, it is crucial to provide not only predictions but also an estimate of the uncertainty in the predictions. Until recently, very few tools were available to provide uncertainty quantification (UQ) for NN predictions. However, in recent years the computer-science field has developed numerous UQ approaches, and several research groups are exploring how to apply these approaches in environmental science. We provide an accessible introduction to six of these UQ approaches, then focus on tools for the next step, namely, to answer the question:Once we obtain an uncertainty estimate (using any approach), how do we know whether it is good or bad?To answer this question, we highlight four evaluation graphics and eight evaluation scores that are well suited for evaluating and comparing uncertainty estimates (NN based or otherwise) for environmental-science applications. We demonstrate the UQ approaches and UQ-evaluation methods for two real-world problems: 1) estimating vertical profiles of atmospheric dewpoint (a regression task) and 2) predicting convection over Taiwan based onHimawari-8satellite imagery (a classification task). We also provide Jupyter notebooks with Python code for implementing the UQ approaches and UQ-evaluation methods discussed herein. This article provides the environmental-science community with the knowledge and tools to start incorporating the large number of emerging UQ methods into their research. Significance StatementNeural networks are used for many environmental-science applications, some involving life-or-death decision-making. In recent years new methods have been developed to provide much-needed uncertainty estimates for NN predictions. We seek to accelerate the adoption of these methods in the environmental-science community with an accessible introduction to 1) methods for computing uncertainty estimates in NN predictions and 2) methods for evaluating such estimates.
more »
« less
Sensitivity Analysis and Bayesian Calibration of OpenSees Models Using quoFEM
The NHERI SimCenter is a nine-year research project that aims to advance the simulation of natural hazard impact on the built environment and communities. The SimCenter is developing several open-source workflow applications and an underlying scientific application framework. All applications built on this framework provide an OpenSees interface that enables users to use their existing models in advanced simulation studies, such as local and regional performance assessment, and uncertainty quantification (UQ). SimCenter applications provide researchers an opportunity to explore different extensions of their models by lowering the interdisciplinary barrier and encouraging collaboration. Among the applications, quoFEM provides access to UQ analyses with an easy-to-use, standardized interface. This work demonstrates the research enabled by quoFEM through the example of model calibration using PM4Sand, a soil constitutive model available in OpenSees. After an initial sensitivity analysis, the model is calibrated using Bayesian inference based on observations of hysteretic soil response from cyclic direct simple shear tests. The uncertainty in the model parameters is used in forward propagation to explore plausible lateral spreading scenarios due to seismic liquefaction. The results demonstrate the utility of quoFEM to the OpenSees community as a UQ-enabling tool.
more »
« less
- Award ID(s):
- 2131111
- PAR ID:
- 10501938
- Publisher / Repository:
- Springer, Cham
- Date Published:
- Journal Name:
- Proceedings of the 2022 Eurasian OpenSees Days
- ISBN:
- 978-3-031-30125-4
- Format(s):
- Medium: X
- Location:
- https://doi.org/10.1007/978-3-031-30125-4_6
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Modeling and simulation for additive manufacturing (AM) are critical enablers for understanding process physics, conducting process planning and optimization, and streamlining qualification and certification. It is often the case that a suite of hierarchically linked (or coupled) simulation models is needed to achieve the above tasks, as the entirety of the complex physical phenomena relevant to the understanding of process-structure-property-performance relationships in the context of AM precludes the use of a single simulation framework. In this study using a Bayesian network approach, we address the important problem of conducting uncertainty quantification (UQ) analysis for multiple hierarchical models to establish process-microstructure relationships in laser powder bed fusion (LPBF) AM. More significantly, we present the framework to calibrate and analyze simulation models that have experimentally unmeasurable variables, which are quantities of interest predicted by an upstream model and deemed necessary for the downstream model in the chain. We validate the framework using a case study on predicting the microstructure of a binary nickel-niobium alloy processed using LPBF as a function of processing parameters. Our framework is shown to be able to predict segregation of niobium with up to 94.3% prediction accuracy on test data.more » « less
-
Abstract Reservoir operations for gas extraction, fluid disposal, carbon dioxide storage, or geothermal energy production are capable of inducing seismicity. Modeling tools exist for seismicity forecasting using operational data, but the computational costs and uncertainty quantification (UQ) pose challenges. We address this issue in the context of seismicity induced by gas production from the Groningen gas field using an integrated modeling framework, which combines reservoir modeling, geomechanical modeling, and stress-based earthquake forecasting. The framework is computationally efficient thanks to a 2D finite-element reservoir model, which assumes vertical flow equilibrium, and the use of semianalytical solutions to calculate poroelastic stress changes and predict seismicity rate. The earthquake nucleation model is based on rate-and-state friction and allows for an initial strength excess so that the faults are not assumed initially critically stressed. We estimate uncertainties in the predicted number of earthquakes and magnitudes. To reduce the computational costs, we assume that the stress model is true, but our UQ algorithm is general enough that the uncertainties in reservoir and stress models could be incorporated. We explore how the selection of either a Poisson or a Gaussian likelihood influences the forecast. We also use a synthetic catalog to estimate the improved forecasting performance that would have resulted from a better seismicity detection threshold. Finally, we use tapered and nontapered Gutenberg–Richter distributions to evaluate the most probable maximum magnitude over time and account for uncertainties in its estimation. Although we did not formally account for uncertainties in the stress model, we tested several alternative stress models, and found negligible impact on the predicted temporal evolution of seismicity and forecast uncertainties. Our study shows that the proposed approach yields realistic estimates of the uncertainties of temporal seismicity and is applicable for operational forecasting or induced seismicity monitoring. It can also be used in probabilistic traffic light systems.more » « less
-
Abstract Climate models are generally calibrated manually by comparing selected climate statistics, such as the global top‐of‐atmosphere energy balance, to observations. The manual tuning only targets a limited subset of observational data and parameters. Bayesian calibration can estimate climate model parameters and their uncertainty using a larger fraction of the available data and automatically exploring the parameter space more broadly. In Bayesian learning, it is natural to exploit the seasonal cycle, which has large amplitude compared with anthropogenic climate change in many climate statistics. In this study, we develop methods for the calibration and uncertainty quantification (UQ) of model parameters exploiting the seasonal cycle, and we demonstrate a proof‐of‐concept with an idealized general circulation model (GCM). UQ is performed using the calibrate‐emulate‐sample approach, which combines stochastic optimization and machine learning emulation to speed up Bayesian learning. The methods are demonstrated in a perfect‐model setting through the calibration and UQ of a convective parameterization in an idealized GCM with a seasonal cycle. Calibration and UQ based on seasonally averaged climate statistics, compared to annually averaged, reduces the calibration error by up to an order of magnitude and narrows the spread of the non‐Gaussian posterior distributions by factors between two and five, depending on the variables used for UQ. The reduction in the spread of the parameter posterior distribution leads to a reduction in the uncertainty of climate model predictions.more » « less
-
Quantifying the impact of parametric and model-form uncertainty on the predictions of stochastic models is a key challenge in many applications. Previous work has shown that the relative entropy rate is an effective tool for deriving path-space uncertainty quantification (UQ) bounds on ergodic averages. In this work we identify appropriate information-theoretic objects for a wider range of quantities of interest on path-space, such as hitting times and exponentially discounted observables, and develop the corresponding UQ bounds. In addition, our method yields tighter UQ bounds, even in cases where previous relative-entropy-based methods also apply, e.g. , for ergodic averages. We illustrate these results with examples from option pricing, non-reversible diffusion processes, stochastic control, semi-Markov queueing models, and expectations and distributions of hitting times.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
