An official website of the United States government
Deep Learning (DL) methods have been transforming computer vision with innovative adaptations to other domains including climate change. For DL to pervade Science and Engineering (S&EE) applications where risk management is a core component, well-characterized uncertainty estimates must accompany predictions. However, S&E observations and model-simulations often follow heavily skewed distributions and are not well modeled with DL approaches, since they usually optimize a Gaussian, or Euclidean, likelihood loss. Recent developments in Bayesian Deep Learning (BDL), which attempts to capture uncertainties from noisy observations, aleatoric, and from unknown model parameters, epistemic, provide us a foundation. Here we present a discrete-continuous BDL model with Gaussian and lognormal likelihoods for uncertainty quantification (UQ). We demonstrate the approach by developing UQ estimates on “DeepSD’‘, a super-resolution based DL model for Statistical Downscaling (SD) in climate applied to precipitation, which follows an extremely skewed distribution. We find that the discrete-continuous models outperform a basic Gaussian distribution in terms of predictive accuracy and uncertainty calibration. Furthermore, we find that the lognormal distribution, which can handle skewed distributions, produces quality uncertainty estimates at the extremes. Such results may be important across S&E, as well as other domains such as finance and economics, where extremes are often of significant interest. Furthermore, to our knowledge, this is the first UQ model in SD where both aleatoric and epistemic uncertainties are characterized.
more »
« less
Creating and Evaluating Uncertainty Estimates with Neural Networks for Environmental-Science Applications
Abstract Neural networks (NN) have become an important tool for prediction tasks—both regression and classification—in environmental science. Since many environmental-science problems involve life-or-death decisions and policy making, it is crucial to provide not only predictions but also an estimate of the uncertainty in the predictions. Until recently, very few tools were available to provide uncertainty quantification (UQ) for NN predictions. However, in recent years the computer-science field has developed numerous UQ approaches, and several research groups are exploring how to apply these approaches in environmental science. We provide an accessible introduction to six of these UQ approaches, then focus on tools for the next step, namely, to answer the question:Once we obtain an uncertainty estimate (using any approach), how do we know whether it is good or bad?To answer this question, we highlight four evaluation graphics and eight evaluation scores that are well suited for evaluating and comparing uncertainty estimates (NN based or otherwise) for environmental-science applications. We demonstrate the UQ approaches and UQ-evaluation methods for two real-world problems: 1) estimating vertical profiles of atmospheric dewpoint (a regression task) and 2) predicting convection over Taiwan based onHimawari-8satellite imagery (a classification task). We also provide Jupyter notebooks with Python code for implementing the UQ approaches and UQ-evaluation methods discussed herein. This article provides the environmental-science community with the knowledge and tools to start incorporating the large number of emerging UQ methods into their research. Significance StatementNeural networks are used for many environmental-science applications, some involving life-or-death decision-making. In recent years new methods have been developed to provide much-needed uncertainty estimates for NN predictions. We seek to accelerate the adoption of these methods in the environmental-science community with an accessible introduction to 1) methods for computing uncertainty estimates in NN predictions and 2) methods for evaluating such estimates.
more »
« less
- Award ID(s):
- 1934668
- PAR ID:
- 10405615
- Publisher / Repository:
- American Meteorological Society
- Date Published:
- Journal Name:
- Artificial Intelligence for the Earth Systems
- Volume:
- 2
- Issue:
- 2
- ISSN:
- 2769-7525
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Neural networks (NNs) are increasingly used for data‐driven subgrid‐scale parameterizations in weather and climate models. While NNs are powerful tools for learning complex non‐linear relationships from data, there are several challenges in using them for parameterizations. Three of these challenges are (a) data imbalance related to learning rare, often large‐amplitude, samples; (b) uncertainty quantification (UQ) of the predictions to provide an accuracy indicator; and (c) generalization to other climates, for example, those with different radiative forcings. Here, we examine the performance of methods for addressing these challenges using NN‐based emulators of the Whole Atmosphere Community Climate Model (WACCM) physics‐based gravity wave (GW) parameterizations as a test case. WACCM has complex, state‐of‐the‐art parameterizations for orography‐, convection‐, and front‐driven GWs. Convection‐ and orography‐driven GWs have significant data imbalance due to the absence of convection or orography in most grid points. We address data imbalance using resampling and/or weighted loss functions, enabling the successful emulation of parameterizations for all three sources. We demonstrate that three UQ methods (Bayesian NNs, variational auto‐encoders, and dropouts) provide ensemble spreads that correspond to accuracy during testing, offering criteria for identifying when an NN gives inaccurate predictions. Finally, we show that the accuracy of these NNs decreases for a warmer climate (4 × CO2). However, their performance is significantly improved by applying transfer learning, for example, re‐training only one layer using ∼1% new data from the warmer climate. The findings of this study offer insights for developing reliable and generalizable data‐driven parameterizations for various processes, including (but not limited to) GWs.more » « less
-
Abstract Probabilistic near‐term forecasting facilitates evaluation of model predictions against observations and is of pressing need in ecology to inform environmental decision‐making and effect societal change. Despite this imperative, many ecologists are unfamiliar with the widely used tools for evaluating probabilistic forecasts developed in other fields. We address this gap by reviewing the literature on probabilistic forecast evaluation from diverse fields including climatology, economics, and epidemiology. We present established practices for selecting evaluation data (end‐sample hold out), graphical forecast evaluation (times‐series plots with uncertainty, probability integral transform plots), quantitative evaluation using scoring rules (log, quadratic, spherical, and ranked probability scores), and comparing scores across models (skill score, Diebold–Mariano test). We cover common approaches, highlight mathematical concepts to follow, and note decision points to allow application of general principles to specific forecasting endeavors. We illustrate these approaches with an application to a long‐term rodent population time series currently used for ecological forecasting and discuss how ecology can continue to learn from and drive the cross‐disciplinary field of forecasting science.more » « less
-
Abstract IntroductionIgA nephropathy (IgAN) can cause end‐stage kidney disease (ESKD). This study assesses the impact of induction and maintenance immunosuppression on IgAN recurrence, graft survival, and mortality in living and deceased donor kidney transplants (LDKT and DDKT). MethodsRetrospective analysis of the UNOS database in adults with ESKD secondary to IgAN who received kidney transplants between January 2000 and June 30, 2022. Patients with thymoglobulin (ATG), alemtuzumab, or basiliximab/daclizumab induction with calcineurin inhibitor (CNI) and mycophenolate mofetil (MMF) with or without prednisone maintenance were analyzed. Multivariate logistic regression was performed to identify factors correlated with IgA recurrence. Multivariable Cox regression analyses were performed for clinically suspected risk factors. Kaplan Meir Analysis was utilized for overall graft survival. ResultsCompared to ATG with steroid maintenance, alemtuzumab with steroid increased the odds of IgAN recurrence in DDKTs (OR 1.90,p <.010, 95% CI [1.169–3.101]). Alemtuzumab with and without steroid increased the odds of recurrence by 52% (p = .036) and 56% (p = .005), respectively, in LDKTs. ATG without steroids was associated with less risk of IgAN recurrence (HR .665,p = .044, 95% CI [.447–.989]), graft failure (HR .758,p = .002, 95% CI [.633–.907]), and death (HR .619,p <.001, 95% CI [.490–.783]) in DDKTs. Recurrence was strongly associated with risks of graft failure in DDKTs and LDKTs and death in LDKTs. ConclusionIn patients with IgAN requiring a kidney transplant, Alemtuzumab induction correlates with increased IgAN recurrence. Relapse significantly affects graft survival and mortality. ATG without steroids is associated with the least graft loss and mortality.more » « less
-
Abstract ObjectivesRecurrent famine events during the medieval period might have contributed to excess mortality during the Black Death in London, England (c. 1349–1350). Previous research using conventional methods of age estimation revealed that adult males experienced lower risks of mortality under “normal” (attritional) but not famine mortality conditions following the Black Death. However, given the biases inherent in conventional age estimation methods, this study reassesses sex differences in risks of medieval adult famine mortality using ages estimated via transition analysis, which avoids some of the limitations of conventional age estimation methods. Materials and MethodsWe apply hazards analysis (the Gompertz model of adult mortality) to ages estimated for human skeletal remains (n= 1245) from London cemeteries dated to the pre‐Black Death (c. 1000–1250 CE) and post‐Black Death (c. 1350–1540 CE) periods. ResultsThe results reveal no sex differences in risks of mortality before the Black Death but indicate that adult males faced lower risks of mortality after the Black Death during conditions of normal and famine mortality. ConclusionsThese findings largely support those of our previous research, which suggested that selective mortality during the Black Death or sex‐biased improvements in standard of living following the epidemic reduced risk of mortality for adult males in the post‐Black Death period under normal mortality conditions. However, the use of transition analysis age estimates also revealed a reduced risk of mortality for post‐Black Death adult males under famine conditions.more » « less
