skip to main content


The NSF Public Access Repository (NSF-PAR) system and access will be unavailable from 11:00 PM ET on Thursday, May 23 until 2:00 AM ET on Friday, May 24 due to maintenance. We apologize for the inconvenience.

Title: Uncertainty quantification techniques for data-driven space weather modeling: thermospheric density application

Machine learning (ML) has been applied to space weather problems with increasing frequency in recent years, driven by an influx of in-situ measurements and a desire to improve modeling and forecasting capabilities throughout the field. Space weather originates from solar perturbations and is comprised of the resulting complex variations they cause within the numerous systems between the Sun and Earth. These systems are often tightly coupled and not well understood. This creates a need for skillful models with knowledge about the confidence of their predictions. One example of such a dynamical system highly impacted by space weather is the thermosphere, the neutral region of Earth’s upper atmosphere. Our inability to forecast it has severe repercussions in the context of satellite drag and computation of probability of collision between two space objects in low Earth orbit (LEO) for decision making in space operations. Even with (assumed) perfect forecast of model drivers, our incomplete knowledge of the system results in often inaccurate thermospheric neutral mass density predictions. Continuing efforts are being made to improve model accuracy, but density models rarely provide estimates of confidence in predictions. In this work, we propose two techniques to develop nonlinear ML regression models to predict thermospheric density while providing robust and reliable uncertainty estimates: Monte Carlo (MC) dropout and direct prediction of the probability distribution, both using the negative logarithm of predictive density (NLPD) loss function. We show the performance capabilities for models trained on both local and global datasets. We show that the NLPD loss provides similar results for both techniques but the direct probability distribution prediction method has a much lower computational cost. For the global model regressed on the Space Environment Technologies High Accuracy Satellite Drag Model (HASDM) density database, we achieve errors of approximately 11% on independent test data with well-calibrated uncertainty estimates. Using an in-situ CHAllenging Minisatellite Payload (CHAMP) density dataset, models developed using both techniques provide test error on the order of 13%. The CHAMP models—on validation and test data—are within 2% of perfect calibration for the twenty prediction intervals tested. We show that this model can also be used to obtain global density predictions with uncertainties at a given epoch.

more » « less
Award ID(s):
Author(s) / Creator(s):
Publisher / Repository:
Nature Publishing Group
Date Published:
Journal Name:
Scientific Reports
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. The rapidly increasing congestion in the low Earth environment makes the modeling of uncertainty in atmospheric drag force a critical task, affecting space situational awareness (SSA) activities like the probability of collision estimation. A key element in atmospheric drag modeling is the assessment of uncertainty in the atmospheric drag coefficient estimate. While atmospheric drag coefficients for space objects with known characteristics can be computed numerically, they suffer from large computational costs for practical applications. In this work, we use cost-effective data-driven stochastic methods for modeling the drag coefficients of objects in the low Earth orbit (LEO) region. The training data is generated using the numerical Test Particle Monte Carlo (TPMC) method. TPMC is simulated with Cercignani–Lampis–Lord (CLL) gas-surface interaction (GSI) model. Mehta et al. [1] use a Gaussian process regression (GPR) model to predict satellite drag coefficient, but the authors did not estimate the predictive uncertainty. The first part of this research extends the work by Mehta et al. [1] by fitting a GPR model to the training data and performing predictive uncertainty estimation. The results of the Gaussian fit are then compared against a deep neural network (DNN) model aided by the Monte Carlo dropout approach. To the best of our knowledge, this is the first study to use the aforementioned stochastic deep learning algorithm to perform predictive uncertainty estimation of the estimated satellite drag coefficient. Apart from the accuracy of the models, we also undertake the task of calibrating the models. Simulations are carried out for a spherical satellite followed by the Champ satellite. Finally, quantification of the effect of drag coefficient uncertainty on orbit prediction is carried out for different solar activity and geomagnetic activity levels. 
    more » « less
  2. null (Ed.)
    The specification and prediction of density fluctuations in the thermosphere, especially during geomagnetic storms, is a key challenge for space weather observations and modeling. It is of great operational importance for tracking objects orbiting in near-Earth space. For low-Earth orbit, variations in neutral density represent the most important uncertainty for propagation and prediction of satellite orbits. An international conference in 2018 conducted under the auspices of the NASA Community Coordinated Modeling Center (CCMC) included a workshop on neutral density modeling, using both empirical and numerical methods, and resulted in the organization of an initial effort of model comparison and evaluation. Here, we present an updated metric for model assessment under geomagnetic storm conditions by dividing a storm in four phases with respect to the time of minimum Dst and then calculating the mean density ratios and standard deviations and correlations. Comparisons between three empirical (NRLMSISE-00, JB2008 and DTM2013) and two first-principles models (TIE-GCM and CTIPe) and neutral density data sets that include measurements by the CHAMP, GRACE, and GOCE satellites for 13 storms are presented. The models all show reduced performance during storms, notably much increased standard deviations, but DTM2013, JB2008 and CTIPe did not on average reveal a significant bias in the four phases of our metric. DTM2013 and TIE-GCM driven with the Weimer model achieved the best results taking the entire storm event into account, while NRLMSISE-00 systematically and significantly underestimates the storm densities. Numerical models are still catching up to empirical methods on a statistical basis, but as their drivers become more accurate and they become available at higher resolutions, they will surpass them in the foreseeable future. 
    more » « less
  3. Abstract

    The space weather research community relies heavily on thermospheric density data to understand long‐term thermospheric variability, construct assimilative, empirical, and semiempirical global atmospheric models and validate model performance. One of the challenges in resolving accurate thermospheric density data sets from satellite orbital drag measurements is modeling appropriate physical aerodynamic drag force coefficients. The drag coefficient may change throughout the thermosphere due to model dependencies on composition and altitude. As such, existing drag coefficient model errors and corresponding errors in orbit‐derived density data sets and models may be altitude and solar cycle dependent with greater errors at higher altitudes around 500 km near the oxygen‐to‐helium transition region. In this paper, inter‐satellite observed‐to‐modeled density comparisons at ∼500 km are evaluated to constrain drag coefficient modeling assumptions. Observed densities are derived from accelerometer data for the Gravity Recovery and Climate Experiment (GRACE) satellites and Two‐Line Element data for a set of compact satellites, while the NRLMSISE‐00 atmospheric model is used to obtain modeled densities and composition information. Density consistency results indicate that drag coefficient models with incomplete energy and momentum accommodation produce the most consistent densities, while the standard diffuse modeling approach may not be appropriate at these altitudes. Models with momentum accommodation between 0.5 and 0.9 and energy accommodation between 0.83 and 0.96 may be most appropriate at upper thermospheric altitudes. Modeling drag coefficients with diffuse gas‐surface interactions for the GRACE satellites could lead to errors in derived density of ∼25% and in‐track satellite orbit prediction uncertainty during solar maximum conditions on the order of kilometers.

    more » « less
  4. Abstract

    Machine learning (ML) models are universal function approximators and—if used correctly—can summarize the information content of observational data sets in a functional form for scientific and engineering applications. A benefit to ML over parametric models is that there are no a priori assumptions about particular basis functions which can potentially limit the phenomena that can be modeled. In this work, we develop ML models on three data sets: the Space Environment Technologies High Accuracy Satellite Drag Model (HASDM) density database, a spatiotemporally matched data set of outputs from the Jacchia‐Bowman 2008 Empirical Thermospheric Density Model (JB2008), and an accelerometer‐derived density data set from CHAllenging Minisatellite Payload (CHAMP). These ML models are compared to the Naval Research Laboratory Mass Spectrometer and Incoherent Scatter radar (NRLMSIS 2.0) model to study the presence of post‐storm cooling in the middle‐thermosphere. We find that both NRLMSIS 2.0 and JB2008‐ML do not account for post‐storm cooling and consequently perform poorly in periods following strong geomagnetic storms (e.g., the 2003 Halloween storms). Conversely, HASDM‐ML and CHAMP‐ML do show evidence of post‐storm cooling indicating that this phenomenon is present in the original data sets. Results show that density reductions up to 40% can occur 1–3 days post‐storm depending on the location and strength of the storm.

    more » « less
  5. Abstract

    Thermospheric mass density perturbations are commonly observed during geomagnetic storms and fundamental to upper atmosphere dynamics, but the sources of these perturbations are not well understood. Large neutral density perturbations during storms greatly affect the drag experienced by low Earth orbit. We investigated the thermospheric density perturbations at all latitudes observed along the CHAMP and GRACE satellite trajectories during the August 24–25, 2005 geomagnetic storm. Observations show that large neutral density enhancements occurred not only at high latitudes, but also globally. Large density perturbations were seen in the equatorial regions away from the high‐latitude, magnetospheric energy sources. We used the high‐resolution Multiscale Atmosphere Geospace Environment (MAGE) model to simulate consecutive neutral density changes observed by satellites during the storm. The MAGE simulation, which resolved mesoscale high‐latitude convection electric fields and field‐aligned currents, and included physics‐based specification of auroral precipitation, was contrasted with a standalone ionosphere‐thermosphere simulation driven by a high‐latitude electrodynamics empirical model. The comparison demonstrates that first‐principles representations of highly dynamic and localized Joule heating events in a fully coupled whole geospace model is critical to accurately capture both generation and propagation of traveling atmospheric disturbances (TADs) that produce neutral density perturbations globally. The MAGE simulation shows that larger density peaks in the equatorial region observed by CHAMP and GRACE are the result of TADs generated at high‐latitudes in both hemispheres, and intersect at low‐latitudes. This study reveals the importance of investigating thermospheric density variations at all latitudes in a fully coupled geospace model with sufficiently high resolving power.

    more » « less