An official website of the United States government
Abstract Machine learning (ML) has been applied to space weather problems with increasing frequency in recent years, driven by an influx of in-situ measurements and a desire to improve modeling and forecasting capabilities throughout the field. Space weather originates from solar perturbations and is comprised of the resulting complex variations they cause within the numerous systems between the Sun and Earth. These systems are often tightly coupled and not well understood. This creates a need for skillful models with knowledge about the confidence of their predictions. One example of such a dynamical system highly impacted by space weather is the thermosphere, the neutral region of Earth’s upper atmosphere. Our inability to forecast it has severe repercussions in the context of satellite drag and computation of probability of collision between two space objects in low Earth orbit (LEO) for decision making in space operations. Even with (assumed) perfect forecast of model drivers, our incomplete knowledge of the system results in often inaccurate thermospheric neutral mass density predictions. Continuing efforts are being made to improve model accuracy, but density models rarely provide estimates of confidence in predictions. In this work, we propose two techniques to develop nonlinear ML regression models to predict thermospheric density while providing robust and reliable uncertainty estimates: Monte Carlo (MC) dropout and direct prediction of the probability distribution, both using the negative logarithm of predictive density (NLPD) loss function. We show the performance capabilities for models trained on both local and global datasets. We show that the NLPD loss provides similar results for both techniques but the direct probability distribution prediction method has a much lower computational cost. For the global model regressed on the Space Environment Technologies High Accuracy Satellite Drag Model (HASDM) density database, we achieve errors of approximately 11% on independent test data with well-calibrated uncertainty estimates. Using an in-situ CHAllenging Minisatellite Payload (CHAMP) density dataset, models developed using both techniques provide test error on the order of 13%. The CHAMP models—on validation and test data—are within 2% of perfect calibration for the twenty prediction intervals tested. We show that this model can also be used to obtain global density predictions with uncertainties at a given epoch.
more »
« less
Measuring Stochastic Data Complexity with Boltzmann Influence Functions
Estimating the uncertainty of a model’s prediction on a test point is a crucial part of ensuring reliability and calibration under distribution shifts. A minimum description length approach to this problem uses the predictive normalized maximum likelihood (pNML) distribution, which considers every possible label for a data point, and decreases confidence in a prediction if other labels are also consistent with the model and training data. In this work we propose IF-COMP, a scalable and efficient approximation of the pNML distribution that linearizes the model with a temperature- scaled Boltzmann influence function. IF-COMP can be used to produce well-calibrated predictions on test points as well as measure complexity in both labelled and unlabelled settings. We experimentally validate IF-COMP on uncertainty calibration, mislabel detection, and OOD detection tasks, where it consistently matches or beats strong baseline methods.
more »
« less
- Award ID(s):
- 2339381
- PAR ID:
- 10580636
- Publisher / Repository:
- Proceedings of the 41st International Conference on Machine Learning (ICML)
- Date Published:
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Wang, L; Zhang, JM; Wang, R (Ed.)Liquefaction under cyclic loads can be predicted through advanced (liquefaction-capable) material constitutive models. However, such constitutive models have several input parameters whose values are often unknown or imprecisely known, requiring calibration via lab/in-situ test data. This study proposes a Bayesian updating framework that integrates probabilistic calibration of the soil model and probabilistic prediction of lateral spreading due to seismic liquefaction. In particular, the framework consists of three main parts: (1) Parametric study based on global sensitivity analysis, (2) Bayesian calibration of the primary input parameters of the constitutive model, and (3) Forward uncertainty propagation through a computational model simulating the response of a soil column under earthquake loading. For demonstration, the PM4Sand model is adopted, and cyclic strength data of Ottawa F-65 sand from cyclic direct simple shear tests are utilized to calibrate the model. The three main uncertainty analyses are performed using quoFEM, a SimCenter open-source software application for uncertainty quantification and optimization in the field of natural hazard engineering. The results demonstrate the potential of the framework linked with quoFEM to perform calibration and uncertainty propagation using sophisticated simulation models that can be part of a performance-based design workflow.more » « less
-
Liquefaction under cyclic loads can be predicted through advanced (liquefaction-capable) material constitutive models. However, such constitutive models have several input parameters whose values are often unknown or imprecisely known, requiring calibration via lab/in-situ test data. This study proposes a Bayesian updating framework that integrates probabilistic calibration of the soil model and probabilistic prediction of lateral spreading due to seismic liquefaction. In particular, the framework consists of three main parts: (1) Parametric study based on global sensitivity analysis, (2) Bayesian calibration of the primary input parameters of the constitutive model, and (3) Forward uncertainty propagation through a computational model simulating the response of a soil column under earthquake loading. For demonstration, the PM4Sand model is adopted, and cyclic strength data of Ottawa F-65 sand from cyclic direct simple shear tests are utilized to calibrate the model. The three main uncertainty analyses are performed using quoFEM, a SimCenter open-source software application for uncertainty quantification and optimization in the field of natural hazard engineering. The results demonstrate the potential of the framework linked with quoFEM to perform calibration and uncertainty propagation using sophisticated simulation models that can be part of a performance-based design workflow.more » « less
-
Learning multi-agent dynamics is a core AI problem with broad applications in robotics and autonomous driving. While most existing works focus on deterministic prediction, producing probabilistic forecasts to quantify uncertainty and assess risks is critical for downstream decision-making tasks such as motion planning and collision avoidance. Multi-agent dynamics often contains internal symmetry. By leveraging symmetry, specifically rotation equivariance, we can improve not only the prediction accuracy but also uncertainty calibration. We introduce Energy Score, a proper scoring rule, to evaluate probabilistic predictions. We propose a novel deep dynamics model, Probabilistic Equivariant Continuous COnvolution (PECCO) for probabilistic prediction of multi-agent trajectories. PECCO extends equivariant continuous convolution to model the joint velocity distribution of multiple agents. It uses dynamics integration to propagate the uncertainty from velocity to position. On both synthetic and real-world datasets, PECCO shows significant improvements in accuracy and calibration compared to non-equivariant baselines.more » « less
-
Abstract Conformal prediction builds marginally valid prediction intervals that cover the unknown outcome of a randomly drawn test point with a prescribed probability. However, in practice, data-driven methods are often used to identify specific test unit(s) of interest, requiring uncertainty quantification tailored to these focal units. In such cases, marginally valid conformal prediction intervals may fail to provide valid coverage for the focal unit(s) due to selection bias. This article presents a general framework for constructing a prediction set with finite-sample exact coverage, conditional on the unit being selected by a given procedure. The general form of our method accommodates arbitrary selection rules that are invariant to the permutation of the calibration units and generalizes Mondrian Conformal Prediction to multiple test units and non-equivariant classifiers. We also work out computationally efficient implementation of our framework for a number of realistic selection rules, including top-K selection, optimization-based selection, selection based on conformal p-values, and selection based on properties of preliminary conformal prediction sets. The performance of our methods is demonstrated via applications in drug discovery and health risk prediction.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
