An official website of the United States government
Uncertainty decomposition refers to the task of decomposing the total uncertainty of a predictive model into aleatoric (data) uncertainty, resulting from inherent randomness in the data-generating process, and epistemic (model) uncertainty, resulting from missing information in the model’s training data. In large language models (LLMs) specifically, identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability, but remains an important open research question. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling, which can be applied to any pre-trained LLM. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions. We show that, when aleatoric uncertainty arises from ambiguity or under-specification in LLM inputs, this approach makes it possible to factor an (un-clarified) LLM’s predictions into separate aleatoric and epistemic terms, using a decomposition similar to the one employed by Bayesian neural networks. Empirical evaluations demonstrate that input clarification ensembling provides accurate and reliable uncertainty quantification on several language processing tasks.
more »
« less
This content will become publicly available on March 13, 2026
Quantification of total uncertainty in the physics-informed reconstruction of CVSim-6 physiology
When predicting physical phenomena through simulation, quantification of the total uncertainty due to multiple sources is as crucial as making sure the underlying numerical model is accurate. Possible sources include irreduciblealeatoricuncertainty due to noise in the data,epistemicuncertainty induced by insufficient data or inadequate parameterization andmodel-formuncertainty related to the use of misspecified model equations. In addition, recently proposed approaches provide flexible ways to combine information from data with full or partial satisfaction of equations that typically encode physical principles. Physics-based regularization interacts in non-trivial ways with aleatoric, epistemic and model-form uncertainty and their combination, and a better understanding of this interaction is needed to improve the predictive performance of physics-informed digital twins that operate under real conditions. To better understand this interaction, with a specific focus on biological and physiological models, this study investigates the decomposition of total uncertainty in the estimation of states and parameters of a differential system simulated with MC X-TFC, a new physics-informed approach for uncertainty quantification based on random projections and Monte Carlo sampling. After an introductory comparison between approaches for physics-informed estimation, MC X-TFC is applied to a six-compartment stiff ODE system, the CVSim-6 model, developed in the context of human physiology. The system is first analysed by progressively removing data while estimating an increasing number of parameters, and subsequently by investigating total uncertainty under model-form misspecification of nonlinear resistance in the pulmonary compartment. In particular, we focus on the interaction between the formulation of the discrepancy term and quantification of model-form uncertainty, and show how additional physics can help in the estimation process. The method demonstrates robustness and efficiency in estimating unknown states and parameters, even with limited, sparse and noisy data. It also offers great flexibility in integrating data with physics for improved estimation, even in cases of model misspecification. This article is part of the theme issue ‘Uncertainty quantification for healthcare and biological systems (Part 1)’.
more »
« less
- Award ID(s):
- 1942662
- PAR ID:
- 10632890
- Publisher / Repository:
- The Royal Society
- Date Published:
- Journal Name:
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Volume:
- 383
- Issue:
- 2292
- ISSN:
- 1364-503X
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Non-pharmaceutical interventions (NPI) have been proven vital in the fight against the COVID-19 pandemic before the massive rollout of vaccinations. Considering the inherent epistemic-aleatoric uncertainty of parameters, accurate simulation and modeling of the interplay between the NPI and contagion dynamics are critical to the optimal design of intervention policies. We propose a modified SIRD-MPC model that combines a modified stochastic Susceptible-Infected-Recovered-Deceased (SIRD) compartment model with mixed epistemic-aleatoric parameters and Model Predictive Control (MPC), to develop robust NPI control policies to contain the infection of the COVID-19 pandemic with minimum economic impact. The simulation result indicates that our proposed model can significantly decrease the infection rate compared to the practical results under the same initial conditions.more » « less
-
Estimating and disentangling epistemic uncertainty, uncertainty that is reducible with more training data, and aleatoric uncertainty, uncertainty that is inherent to the task at hand, is critically important when applying machine learning to highstakes applications such as medical imaging and weather forecasting. Conditional diffusion models’ breakthrough ability to accurately and efficiently sample from the posterior distribution of a dataset now makes uncertainty estimation conceptually straightforward: One need only train and sample from a large ensemble of diffusion models. Unfortunately, training such an ensemble becomes computationally intractable as the complexity of the model architecture grows. In this work we introduce a new approach to ensembling, hyper-diffusion models (HyperDM), which allows one to accurately estimate both epistemic and aleatoric uncertainty with a single model. Unlike existing single-model uncertainty methods like Monte-Carlo dropout and Bayesian neural networks, HyperDM offers prediction accuracy on par with, and in some cases superior to, multi-model ensembles. Furthermore, our proposed approach scales to modern network architectures such as Attention U-Net and yields more accurate uncertainty estimates compared to existing methods. We validate our method on two distinct real-world tasks: x-ray computed tomography reconstruction and weather temperature forecasting. Source code is publicly available at https://github.com/matthewachan/hyperdm.more » « less
-
Estimation of cardiovascular model parameters from electronic health records (EHRs) poses a significant challenge primarily due to lack of identifiability. Structural non-identifiability arises when a manifold in the space of parameters is mapped to a common output, while practical non-identifiability can result due to limited data, model misspecification or noise corruption. To address the resulting ill-posed inverse problem, optimization-based or Bayesian inference approaches typically use regularization, thereby limiting the possibility of discovering multiple solutions. In this study, we use inVAErt networks, a neural network-based, data-driven framework for enhanced digital twin analysis of stiff dynamical systems. We demonstrate the flexibility and effectiveness of inVAErt networks in the context of physiological inversion of a six-compartment lumped‐parameter haemodynamic model from synthetic data to real data with missing components. This article is part of the theme issue ‘Uncertainty quantification for healthcare and biological systems (Part 2)’.more » « less
-
Abstract Estimating uncertainty in flood model predictions is important for many applications, including risk assessment and flood forecasting. We focus on uncertainty in physics‐based urban flooding models. We consider the effects of the model's complexity and uncertainty in key input parameters. The effect of rainfall intensity on the uncertainty in water depth predictions is also studied. As a test study, we choose the Interconnected Channel and Pond Routing (ICPR) model of a part of the city of Minneapolis. The uncertainty in the ICPR model's predictions of the floodwater depth is quantified in terms of the ensemble variance using the multilevel Monte Carlo (MC) simulation method. Our results show that uncertainties in the studied domain are highly localized. Model simplifications, such as disregarding the groundwater flow, lead to overly confident predictions, that is, predictions that are both less accurate and uncertain than those of the more complex model. We find that for the same number of uncertain parameters, increasing the model resolution reduces uncertainty in the model predictions (and increases the MC method's computational cost). We employ the multilevel MC method to reduce the cost of estimating uncertainty in a high‐resolution ICPR model. Finally, we use the ensemble estimates of the mean and covariance of the flood depth for real‐time flood depth forecasting using the physics‐informed Gaussian process regression method. We show that even with few measurements, the proposed framework results in a more accurate forecast than that provided by the mean prediction of the ICPR model.more » « less
