More Like this

1. Lognormal and Mixed Gaussian–Lognormal Kalman Filters

   https://doi.org/10.1175/MWR-D-22-0072.1  

   Fletcher, Steven J.; Zupanski, Milija; Goodliff, Michael R.; Kliewer, Anton J.; Jones, Andrew S.; Forsythe, John M.; Wu, Ting-Chi; Hossen, Md. Jakir; Van Loon, Senne (March 2023, Monthly Weather Review)

   Abstract In this paper we present the derivation of two new forms of the Kalman filter equations; the first is for a pure lognormally distributed random variable, while the second set of Kalman filter equations will be for a combination of Gaussian and lognormally distributed random variables. We show that the appearance is similar to that of the Gaussian-based equations, but that the analysis state is a multivariate median and not the mean. We also show results of the mixed distribution Kalman filter with the Lorenz 1963 model with lognormal errors for the background and observations of the z component, and compare them to analysis results from a traditional Gaussian-based extended Kalman filter and show that under certain circumstances the new approach produces more accurate results. 

   more » « less
2. Dynamical Gaussian, lognormal, and reverse lognormal maximum‐likelihood ensemble filter

   https://doi.org/10.1002/qj.4706  

   Van_Loon, Senne; Fletcher, Steven J; Zupanski, Milija (April 2024, Quarterly Journal of the Royal Meteorological Society)

   We present a non‐Gaussian ensemble data assimilation method based on the maximum‐likelihood ensemble filter, which allows for any combination of Gaussian, lognormal, and reverse lognormal errors in both the background and the observations. The technique is fully nonlinear, does not require a tangent linear model, and uses a Hessian preconditioner to minimise the cost function efficiently in ensemble space. When the Gaussian assumption is relaxed, the results show significant improvements in the analysis skill within two atmospheric toy models, and the performance of data assimilation systems for (semi)bounded variables is expected to improve. 

   more » « less
3. A Statistical Hypothesis Testing Strategy for Adaptively Blending Particle Filters and Ensemble Kalman Filters for Data Assimilation

   https://doi.org/10.1175/MWR-D-22-0108.1  

   Kurosawa, Kenta; Poterjoy, Jonathan (January 2023, Monthly Weather Review)

   Abstract Particle filters avoid parametric estimates for Bayesian posterior densities, which alleviates Gaussian assumptions in nonlinear regimes. These methods, however, are more sensitive to sampling errors than Gaussian-based techniques such as ensemble Kalman filters. A recent study by the authors introduced an iterative strategy for particle filters that match posterior moments—where iterations improve the filter’s ability to draw samples from non-Gaussian posterior densities. The iterations follow from a factorization of particle weights, providing a natural framework for combining particle filters with alternative filters to mitigate the impact of sampling errors. The current study introduces a novel approach to forming an adaptive hybrid data assimilation methodology, exploiting the theoretical strengths of nonparametric and parametric filters. At each data assimilation cycle, the iterative particle filter performs a sequence of updates while the prior sample distribution is non-Gaussian, then an ensemble Kalman filter provides the final adjustment when Gaussian distributions for marginal quantities are detected. The method employs the Shapiro–Wilk test to determine when to make the transition between filter algorithms, which has outstanding power for detecting departures from normality. Experiments using low-dimensional models demonstrate that the approach has a significant value, especially for nonhomogeneous observation networks and unknown model process errors. Moreover, hybrid factors are extended to consider marginals of more than one collocated variables using a test for multivariate normality. Findings from this study motivate the use of the proposed method for geophysical problems characterized by diverse observation networks and various dynamic instabilities, such as numerical weather prediction models. Significance Statement Data assimilation statistically processes observation errors and model forecast errors to provide optimal initial conditions for the forecast, playing a critical role in numerical weather forecasting. The ensemble Kalman filter, which has been widely adopted and developed in many operational centers, assumes Gaussianity of the prior distribution and solves a linear system of equations, leading to bias in strong nonlinear regimes. On the other hand, particle filters avoid many of those assumptions but are sensitive to sampling errors and are computationally expensive. We propose an adaptive hybrid strategy that combines their advantages and minimizes the disadvantages of the two methods. The hybrid particle filter–ensemble Kalman filter is achieved with the Shapiro–Wilk test to detect the Gaussianity of the ensemble members and determine the timing of the transition between these filter updates. Demonstrations in this study show that the proposed method is advantageous when observations are heterogeneous and when the model has an unknown bias. Furthermore, by extending the statistical hypothesis test to the test for multivariate normality, we consider marginals of more than one collocated variable. These results encourage further testing for real geophysical problems characterized by various dynamic instabilities, such as real numerical weather prediction models. 

   more » « less
4. A hybrid particle-ensemble Kalman filter for problems with medium nonlinearity

   https://doi.org/10.1371/journal.pone.0248266  

   Grooms, Ian; Robinson, Gregor (March 2021, PLOS ONE)

   Hoteit, Ibrahim (Ed.)

   A hybrid particle ensemble Kalman filter is developed for problems with medium non-Gaussianity, i.e. problems where the prior is very non-Gaussian but the posterior is approximately Gaussian. Such situations arise, e.g., when nonlinear dynamics produce a non-Gaussian forecast but a tight Gaussian likelihood leads to a nearly-Gaussian posterior. The hybrid filter starts by factoring the likelihood. First the particle filter assimilates the observations with one factor of the likelihood to produce an intermediate prior that is close to Gaussian, and then the ensemble Kalman filter completes the assimilation with the remaining factor. How the likelihood gets split between the two stages is determined in such a way to ensure that the particle filter avoids collapse, and particle degeneracy is broken by a mean-preserving random orthogonal transformation. The hybrid is tested in a simple two-dimensional (2D) problem and a multiscale system of ODEs motivated by the Lorenz-‘96 model. In the 2D problem it outperforms both a pure particle filter and a pure ensemble Kalman filter, and in the multiscale Lorenz-‘96 model it is shown to outperform a pure ensemble Kalman filter, provided that the ensemble size is large enough. 

   more » « less
5. Acoustic Float Tracking with the Kalman Smoother

   https://doi.org/10.1175/JTECH-D-21-0063.1  

   Chamberlain, Paul; Cornuelle, Bruce; Talley, Lynne D; Speer, Kevin; Hancock, Cathrine; Riser, Stephen (July 2022, Journal of Atmospheric and Oceanic Technology)

   Abstract Acoustically-tracked subsurface floats provide insights into ocean complexity and were first deployed over 60 years ago. A standard tracking method uses a Least-Squares algorithm to estimate float trajectories based on acoustic ranging from moored sound sources. However, infrequent or imperfect data challenge such estimates, and Least-Squares algorithms are vulnerable to non-Gaussian errors. Acoustic tracking is currently the only feasible strategy for recovering float positions in the sea ice region, a focus of this study. Acoustic records recovered from under-ice floats frequently lack continuous sound source coverage. This is because environmental factors such as surface sound channels and rough sea ice attenuate acoustic signals, while operational considerations make polar sound sources expensive and difficult to deploy. Here we present a Kalman Smoother approach that, by including some estimates of float behavior, extends tracking to situations with more challenging data sets. The Kalman Smoother constructs dynamically constrained, error-minimized float tracks and variance ellipses using all possible position data. This algorithm outperforms the Least-Squares approach and a Kalman Filter in numerical experiments. The Kalman Smoother is applied to previously-tracked floats from the southeast Pacific (DIMES experiment), and the results are compared with existing trajectories constructed using the Least- Squares algorithm. The Kalman Smoother is also used to reconstruct the trajectories of a set of previously untracked, acoustically-enabled Argo floats in the Weddell Sea. 

   more » « less