Title: Simple and Effective Approaches for Uncertainty Prediction in Facial Action Unit Intensity Regression
Knowing how much to trust a prediction is important for many critical applications. We describe two simple approaches to estimate uncertainty in regression prediction tasks and compare their performance and complexity against popular approaches. We operationalize uncertainty in regression as the absolute error between a model's prediction and the ground truth. Our two proposed approaches use a secondary model to predict the uncertainty of a primary predictive model. Our first approach leverages the assumption that similar observations are likely to have similar uncertainty and predicts uncertainty with a non-parametric method. Our second approach trains a secondary model to directly predict the uncertainty of the primary predictive model. Both approaches outperform other established uncertainty estimation approaches on the MNIST, DISFA, and BP4D+ datasets. Furthermore, we observe that approaches that directly predict the uncertainty generally perform better than approaches that indirectly estimate uncertainty. more »« less
Subseasonal climate forecasting is the task of predicting climate variables, such as temperature and precipitation, in a two-week to two-month time horizon. The primary predictors for such prediction problem are spatio-temporal satellite and ground measurements of a variety of climate variables in the atmosphere, ocean, and land, which however have rather limited predictive signal at the subseasonal time horizon. We propose a carefully constructed spatial hierarchical Bayesian regression model that makes use of the inherent spatial structure of the subseasonal climate prediction task. We use our Bayesian model to then derive decision-theoretically optimal point estimates with respect to various performance measures of interest to climate science. As we show, our approach handily improves on various off-the-shelf ML baselines. Since our method is based on a Bayesian frame- work, we are also able to quantify the uncertainty in our predictions, which is particularly crucial for difficult tasks such as the subseasonal prediction, where we expect any model to have considerable uncertainty at different test locations under differ- ent scenarios.
Paul, S.N.; Sheridan, L.; Mehta, P.; Huzurbazar, S.
(, American Astronomical Society meeting)
The rapidly increasing congestion in the low Earth environment makes the modeling of uncertainty in atmospheric drag force a critical task, affecting space situational awareness (SSA) activities like the probability of collision estimation. A key element in atmospheric drag modeling is the assessment of uncertainty in the atmospheric drag coefficient estimate. While atmospheric drag coefficients for space objects with known characteristics can be computed numerically, they suffer from large computational costs for practical applications. In this work, we use cost-effective data-driven stochastic methods for modeling the drag coefficients of objects in the low Earth orbit (LEO) region. The training data is generated using the numerical Test Particle Monte Carlo (TPMC) method. TPMC is simulated with Cercignani–Lampis–Lord (CLL) gas-surface interaction (GSI) model. Mehta et al. [1] use a Gaussian process regression (GPR) model to predict satellite drag coefficient, but the authors did not estimate the predictive uncertainty. The first part of this research extends the work by Mehta et al. [1] by fitting a GPR model to the training data and performing predictive uncertainty estimation. The results of the Gaussian fit are then compared against a deep neural network (DNN) model aided by the Monte Carlo dropout approach. To the best of our knowledge, this is the first study to use the aforementioned stochastic deep learning algorithm to perform predictive uncertainty estimation of the estimated satellite drag coefficient. Apart from the accuracy of the models, we also undertake the task of calibrating the models. Simulations are carried out for a spherical satellite followed by the Champ satellite. Finally, quantification of the effect of drag coefficient uncertainty on orbit prediction is carried out for different solar activity and geomagnetic activity levels.
Pandey, Chetraj; Ji, Anli; Angryk, Rafal A.; Georgoulis, Manolis K.; Aydin, Berkay
(, Frontiers in Astronomy and Space Sciences)
Solar flare prediction is a central problem in space weather forecasting and has captivated the attention of a wide spectrum of researchers due to recent advances in both remote sensing as well as machine learning and deep learning approaches. The experimental findings based on both machine and deep learning models reveal significant performance improvements for task specific datasets. Along with building models, the practice of deploying such models to production environments under operational settings is a more complex and often time-consuming process which is often not addressed directly in research settings. We present a set of new heuristic approaches to train and deploy an operational solar flare prediction system for ≥M1.0-class flares with two prediction modes: full-disk and active region-based. In full-disk mode, predictions are performed on full-disk line-of-sight magnetograms using deep learning models whereas in active region-based models, predictions are issued for each active region individually using multivariate time series data instances. The outputs from individual active region forecasts and full-disk predictors are combined to a final full-disk prediction result with a meta-model. We utilized an equal weighted average ensemble of two base learners’ flare probabilities as our baseline meta learner and improved the capabilities of our two base learners by training a logistic regression model. The major findings of this study are: 1) We successfully coupled two heterogeneous flare prediction models trained with different datasets and model architecture to predict a full-disk flare probability for next 24 h, 2) Our proposed ensembling model, i.e., logistic regression, improves on the predictive performance of two base learners and the baseline meta learner measured in terms of two widely used metrics True Skill Statistic (TSS) and Heidke Skill Score (HSS), and 3) Our result analysis suggests that the logistic regression-based ensemble (Meta-FP) improves on the full-disk model (base learner) by ∼9% in terms TSS and ∼10% in terms of HSS. Similarly, it improves on the AR-based model (base learner) by ∼17% and ∼20% in terms of TSS and HSS respectively. Finally, when compared to the baseline meta model, it improves on TSS by ∼10% and HSS by ∼15%.
Effective assisted living environments must be able to perform inferences on how their occupants interact with their environment. Gaze direction provides strong indications of how people interact with their surroundings. In this paper, we propose a gaze tracking method that uses a neural network regressor to estimate gazes from keypoints and integrates them over time using a moving average mechanism. Our gaze regression model uses confidence gated units to handle cases of keypoint occlusion and estimate its own prediction uncertainty. Our temporal approach for gaze tracking incorporates these prediction uncertainties as weights in the moving average scheme. Experimental results on a dataset collected in an assisted living facility demonstrate that our gaze regression network performs on par with a complex, dataset-specific baseline, while its uncertainty predictions are highly correlated with the actual angular error of corresponding estimations. Finally, experiments on videos sequences show that our temporal approach generates more accurate and stable gaze predictions.
Schreck, John S; Gagne, David John; Becker, Charlie; Chapman, William E; Elmore, Kim; Fan, Da; Gantos, Gabrielle; Kim, Eliot; Kimpara, Dhamma; Martin, Thomas; et al
(, Artificial Intelligence for the Earth Systems)
Abstract Robust quantification of predictive uncertainty is a critical addition needed for machine learning applied to weather and climate problems to improve the understanding of what is driving prediction sensitivity. Ensembles of machine learning models provide predictive uncertainty estimates in a conceptually simple way but require multiple models for training and prediction, increasing computational cost and latency. Parametric deep learning can estimate uncertainty with one model by predicting the parameters of a probability distribution but does not account for epistemic uncertainty. Evidential deep learning, a technique that extends parametric deep learning to higher-order distributions, can account for both aleatoric and epistemic uncertainties with one model. This study compares the uncertainty derived from evidential neural networks to that obtained from ensembles. Through applications of the classification of winter precipitation type and regression of surface-layer fluxes, we show evidential deep learning models attaining predictive accuracy rivaling standard methods while robustly quantifying both sources of uncertainty. We evaluate the uncertainty in terms of how well the predictions are calibrated and how well the uncertainty correlates with prediction error. Analyses of uncertainty in the context of the inputs reveal sensitivities to underlying meteorological processes, facilitating interpretation of the models. The conceptual simplicity, interpretability, and computational efficiency of evidential neural networks make them highly extensible, offering a promising approach for reliable and practical uncertainty quantification in Earth system science modeling. To encourage broader adoption of evidential deep learning, we have developed a new Python package, Machine Integration and Learning for Earth Systems (MILES) group Generalized Uncertainty for Earth System Science (GUESS) (MILES-GUESS) (https://github.com/ai2es/miles-guess), that enables users to train and evaluate both evidential and ensemble deep learning. Significance StatementThis study demonstrates a new technique, evidential deep learning, for robust and computationally efficient uncertainty quantification in modeling the Earth system. The method integrates probabilistic principles into deep neural networks, enabling the estimation of both aleatoric uncertainty from noisy data and epistemic uncertainty from model limitations using a single model. Our analyses reveal how decomposing these uncertainties provides valuable insights into reliability, accuracy, and model shortcomings. We show that the approach can rival standard methods in classification and regression tasks within atmospheric science while offering practical advantages such as computational efficiency. With further advances, evidential networks have the potential to enhance risk assessment and decision-making across meteorology by improving uncertainty quantification, a longstanding challenge. This work establishes a strong foundation and motivation for the broader adoption of evidential learning, where properly quantifying uncertainties is critical yet lacking.
Wörtwein, Torsten, and Morency, Louis-Philippe. Simple and Effective Approaches for Uncertainty Prediction in Facial Action Unit Intensity Regression. Retrieved from https://par.nsf.gov/biblio/10169266. Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition 1. Web. doi:10.1109/FG47880.2020.00045.
Wörtwein, Torsten, & Morency, Louis-Philippe. Simple and Effective Approaches for Uncertainty Prediction in Facial Action Unit Intensity Regression. Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition, 1 (). Retrieved from https://par.nsf.gov/biblio/10169266. https://doi.org/10.1109/FG47880.2020.00045
Wörtwein, Torsten, and Morency, Louis-Philippe.
"Simple and Effective Approaches for Uncertainty Prediction in Facial Action Unit Intensity Regression". Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition 1 (). Country unknown/Code not available. https://doi.org/10.1109/FG47880.2020.00045.https://par.nsf.gov/biblio/10169266.
@article{osti_10169266,
place = {Country unknown/Code not available},
title = {Simple and Effective Approaches for Uncertainty Prediction in Facial Action Unit Intensity Regression},
url = {https://par.nsf.gov/biblio/10169266},
DOI = {10.1109/FG47880.2020.00045},
abstractNote = {Knowing how much to trust a prediction is important for many critical applications. We describe two simple approaches to estimate uncertainty in regression prediction tasks and compare their performance and complexity against popular approaches. We operationalize uncertainty in regression as the absolute error between a model's prediction and the ground truth. Our two proposed approaches use a secondary model to predict the uncertainty of a primary predictive model. Our first approach leverages the assumption that similar observations are likely to have similar uncertainty and predicts uncertainty with a non-parametric method. Our second approach trains a secondary model to directly predict the uncertainty of the primary predictive model. Both approaches outperform other established uncertainty estimation approaches on the MNIST, DISFA, and BP4D+ datasets. Furthermore, we observe that approaches that directly predict the uncertainty generally perform better than approaches that indirectly estimate uncertainty.},
journal = {Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition},
volume = {1},
author = {Wörtwein, Torsten and Morency, Louis-Philippe},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
An official website of the United States government
