More Like this

1. The Implicit Delta Method

   Nathan Kallus, James McInerney (January 2022, Advances in neural information processing systems)

   Epistemic uncertainty quantification is a crucial part of drawing credible conclusions from predictive models, whether concerned about the prediction at a given point or any downstream evaluation that uses the model as input. When the predictive model is simple and its evaluation differentiable, this task is solved by the delta method, where we propagate the asymptotically-normal uncertainty in the predictive model through the evaluation to compute standard errors and Wald confidence intervals. However, this becomes difficult when the model and/or evaluation becomes more complex. Remedies include the bootstrap, but it can be computationally infeasible when training the model even once is costly. In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of the predictive model to automatically assess downstream uncertainty. We show that the change in the evaluation due to regularization is consistent for the asymptotic variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference. This provides both a reliable quantification of uncertainty in terms of standard errors as well as permits the construction of calibrated confidence intervals. We discuss connections to other approaches to uncertainty quantification, both Bayesian and frequentist, and demonstrate our approach empirically. 

   more » « less
2. Active subspace analysis and uncertainty quantification for a polydomain ferroelectric phase-field model

   https://doi.org/10.1177/1045389X19853636  

   Leon, Lider_S; Miles, Paul_R; Smith, Ralph_C; Oates, William_S (July 2019, Journal of Intelligent Material Systems and Structures)

   We perform parameter subset selection and uncertainty analysis for phase-field models that are applied to the ferroelectric material lead titanate. A motivating objective is to determine which parameters are influential in the sense that their uncertainties directly affect the uncertainty in the model response, and fix noninfluential parameters at nominal values for subsequent uncertainty propagation. We employ Bayesian inference to quantify the uncertainties of gradient exchange parameters governing 180° and 90° tetragonal phase domain wall energies. The uncertainties of influential parameters determined by parameter subset selection are then propagated through the models to obtain credible intervals when estimating energy densities quantifying polarization and strain across domain walls. The results illustrate various properties of Landau and electromechanical coupling parameters and their influence on domain wall interactions. We employ energy statistics, which quantify distances between statistical observations, to compare credible intervals constructed using a complete set of parameters against an influential subset of parameters. These intervals are obtained from the uncertainty propagation of the model input parameters on the domain wall energy densities. The investigation provides critical insight into the development of parameter subset selection, uncertainty quantification, and propagation methodologies for material modeling domain wall structure evolution, informed by density functional theory simulations. 

   more » « less
3. Learning‐based adaptive‐scenario‐tree model predictive control with improved probabilistic safety using robust Bayesian neural networks

   https://doi.org/10.1002/rnc.6560  

   Bao, Yajie; Chan, Kimberly J.; Mesbah, Ali; Velni, Javad Mohammadpour (December 2022, International Journal of Robust and Nonlinear Control)

   Summary Scenario‐based model predictive control (MPC) methods can mitigate the conservativeness inherent to open‐loop robust MPC. Yet, the scenarios are often generated offline based on worst‐case uncertainty descriptions obtaineda priori, which can in turn limit the improvements in the robust control performance. To this end, this paper presents a learning‐based, adaptive‐scenario‐tree model predictive control approach for uncertain nonlinear systems with time‐varying and/or hard‐to‐model dynamics. Bayesian neural networks (BNNs) are used to learn a state‐ and input‐dependent description of model uncertainty, namely the mismatch between a nominal (physics‐based or data‐driven) model of a system and its actual dynamics. We first present a new approach for training robust BNNs (RBNNs) using probabilistic Lipschitz bounds to provide a less conservative uncertainty quantification. Then, we present an approach to evaluate the credible intervals of RBNN predictions and determine the number of samples required for estimating the credible intervals given a credible level. The performance of RBNNs is evaluated with respect to that of standard BNNs and Gaussian process (GP) as a basis of comparison. The RBNN description of plant‐model mismatch with verified accurate credible intervals is employed to generate adaptive scenarios online for scenario‐based MPC (sMPC). The proposed sMPC approach with adaptive scenario tree can improve the robust control performance with respect to sMPC with a fixed, worst‐case scenario tree and with respect to an adaptive‐scenario‐based MPC (asMPC) using GP regression on a cold atmospheric plasma system. Furthermore, closed‐loop simulation results illustrate that robust model uncertainty learning via RBNNs can enhance the probability of constraint satisfaction of asMPC. 

   more » « less
4. Comparing traditional and Bayesian approaches to ecological meta‐analysis

   https://doi.org/10.1111/2041-210X.13445  

   Pappalardo, Paula; Ogle, Kiona; Hamman, Elizabeth A.; Bence, James R.; Hungate, Bruce A.; Osenberg, Craig W.; O'Hara, ed., Robert B. (July 2020, Methods in Ecology and Evolution)

   Abstract Despite the wide application of meta‐analysis in ecology, some of the traditional methods used for meta‐analysis may not perform well given the type of data characteristic of ecological meta‐analyses.We reviewed published meta‐analyses on the ecological impacts of global climate change, evaluating the number of replicates used in the primary studies (n_i) and the number of studies or records (k) that were aggregated to calculate a mean effect size. We used the results of the review in a simulation experiment to assess the performance of conventional frequentist and Bayesian meta‐analysis methods for estimating a mean effect size and its uncertainty interval.Our literature review showed thatn_iandkwere highly variable, distributions were right‐skewed and were generally small (mediann_i = 5, mediank = 44). Our simulations show that the choice of method for calculating uncertainty intervals was critical for obtaining appropriate coverage (close to the nominal value of 0.95). Whenkwas low (<40), 95% coverage was achieved by a confidence interval (CI) based on thetdistribution that uses an adjusted standard error (the Hartung–Knapp–Sidik–Jonkman, HKSJ), or by a Bayesian credible interval, whereas bootstrap orzdistribution CIs had lower coverage. Despite the importance of the method to calculate the uncertainty interval, 39% of the meta‐analyses reviewed did not report the method used, and of the 61% that did, 94% used a potentially problematic method, which may be a consequence of software defaults.In general, for a simple random‐effects meta‐analysis, the performance of the best frequentist and Bayesian methods was similar for the same combinations of factors (kand mean replication), though the Bayesian approach had higher than nominal (>95%) coverage for the mean effect whenkwas very low (k < 15). Our literature review suggests that many meta‐analyses that usedzdistribution or bootstrapping CIs may have overestimated the statistical significance of their results when the number of studies was low; more appropriate methods need to be adopted in ecological meta‐analyses. 

   more » « less
5. Photometric redshift uncertainties in weak gravitational lensing shear analysis: models and marginalization

   https://doi.org/10.1093/mnras/stac3090  

   Zhang, Tianqing; Rau, Markus Michael; Mandelbaum, Rachel; Li, Xiangchong; Moews, Ben (November 2022, Monthly Notices of the Royal Astronomical Society)

   ABSTRACT Recovering credible cosmological parameter constraints in a weak lensing shear analysis requires an accurate model that can be used to marginalize over nuisance parameters describing potential sources of systematic uncertainty, such as the uncertainties on the sample redshift distribution n(z). Due to the challenge of running Markov chain Monte Carlo (MCMC) in the high-dimensional parameter spaces in which the n(z) uncertainties may be parametrized, it is common practice to simplify the n(z) parametrization or combine MCMC chains that each have a fixed n(z) resampled from the n(z) uncertainties. In this work, we propose a statistically principled Bayesian resampling approach for marginalizing over the n(z) uncertainty using multiple MCMC chains. We self-consistently compare the new method to existing ones from the literature in the context of a forecasted cosmic shear analysis for the HSC three-year shape catalogue, and find that these methods recover statistically consistent error bars for the cosmological parameter constraints for predicted HSC three-year analysis, implying that using the most computationally efficient of the approaches is appropriate. However, we find that for data sets with the constraining power of the full HSC survey data set (and, by implication, those upcoming surveys with even tighter constraints), the choice of method for marginalizing over n(z) uncertainty among the several methods from the literature may modify the 1σ uncertainties on Ωm–S8 constraints by ∼4 per cent, and a careful model selection is needed to ensure credible parameter intervals. 

   more » « less