An official website of the United States government
Abstract The ensemble Kalman filter (EnKF) is a popular technique for data assimilation in high-dimensional nonlinear state-space models. The EnKF represents distributions of interest by an ensemble, which is a form of dimension reduction that enables straightforward forecasting even for complicated and expensive evolution operators. However, the EnKF update step involves estimation of the forecast covariance matrix based on the (often small) ensemble, which requires regularization. Many existing regularization techniques rely on spatial localization, which may ignore long-range dependence. Instead, our proposed approach assumes a sparse Cholesky factor of the inverse covariance matrix, and the nonzero Cholesky entries are further regularized. The resulting method is highly flexible and computationally scalable. In our numerical experiments, our approach was more accurate and less sensitive to misspecification of tuning parameters than tapering-based localization.
more »
« less
This content will become publicly available on June 2, 2026
Inaccuracy of the variance evolution associated with discrete covariance propagation
Abstract Data assimilation methods often also employ the same discrete dynamical model used to evolve the state estimate in time to propagate an approximation of the state estimation error covariance matrix. Four‐dimensional variational methods, for instance, evolve the covariance matrix implicitly via discrete tangent linear dynamics. Ensemble methods, while not forming this matrix explicitly, approximate its evolution at low rank from the evolution of the ensemble members. Such approximate evolution schemes for the covariance matrix imply an approximate evolution of the estimation error variances along its diagonal. For states that satisfy the advection equation, the continuity equation, or related hyperbolic partial differential equations (PDEs), the estimation error variance itself satisfies a known PDE, so the accuracy of the various approximations to the variances implied by the discrete covariance propagation can be determined directly. Experiments conducted by the atmospheric chemical constituent data assimilation community have indicated that such approximate variance evolution can be highly inaccurate. Through careful analysis and simple numerical experiments, we show that such poor accuracy must be expected, due to the inherent nature of discrete covariance propagation, coupled with a special property of the continuum covariance dynamics for states governed by these types of hyperbolic PDE. The intuitive explanation for this inaccuracy is that discrete covariance propagation involves approximating diagonal elements of the covariance matrix with combinations of off‐diagonal elements, even when there is a discontinuity in the continuum covariance dynamics along the diagonal. Our analysis uncovers the resulting error terms that depend on the ratio of the grid spacing to the correlation length, and these terms become very large when correlation lengths begin to approach the grid scale, for instance, as gradients steepen near the diagonal. We show that inaccurate variance evolution can manifest itself as both spurious loss and gain of variance.
more »
« less
- Award ID(s):
- 1848544
- PAR ID:
- 10614414
- Publisher / Repository:
- Wiley
- Date Published:
- Journal Name:
- Quarterly Journal of the Royal Meteorological Society
- ISSN:
- 0035-9009
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
A new type of ensemble Kalman filter is developed, which is based on replacing the sample covariance in the analysis step by its diagonal in a spectral basis. It is proved that this technique improves the approximation of the covariance when the covariance itself is diagonal in the spectral basis, as is the case, e.g., for a second-order stationary random field and the Fourier basis. The method is extended by wavelets to the case when the state variables are random fields which are not spatially homogeneous. Efficient implementations by the fast Fourier transform (FFT) and discrete wavelet transform (DWT) are presented for several types of observations, including high-dimensional data given on a part of the domain, such as radar and satellite images. Computational experiments confirm that the method performs well on the Lorenz 96 problem and the shallow water equations with very small ensembles and over multiple analysis cycles.more » « less
-
Melenk, J.M.; Perugia, I.; Schöberl, J.; Schwab, C (Ed.)The matrix valued exponential function can be used for time-stepping numerically stiff discretization, such as the discontinuous Galerkin method but this approach is expensive as the matrix is dense and necessitates global communication. In this paper, we propose a local low-rank approximation to this matrix. The local low-rank construction is motivated by the nature of wave propagation and costs significantly less to apply than full exponentiation. The accuracy of this time stepping method is inherited from the exponential integrator and the local property of it allows parallel implementation. The method is expected to be useful in design and inverse problems where many solves of the PDE are required. We demonstrate the error convergence of the method for the one-dimensional (1D) Maxwell’s equation on a uniform grid.more » « less
-
Abstract Covariance matrices are fundamental to the analysis and forecast of economic, physical and biological systems. Although the eigenvalues $$\{\lambda _i\}$$ and eigenvectors $$\{\boldsymbol{u}_i\}$$ of a covariance matrix are central to such endeavours, in practice one must inevitably approximate the covariance matrix based on data with finite sample size $$n$$ to obtain empirical eigenvalues $$\{\tilde{\lambda }_i\}$$ and eigenvectors $$\{\tilde{\boldsymbol{u}}_i\}$$, and therefore understanding the error so introduced is of central importance. We analyse eigenvector error $$\|\boldsymbol{u}_i - \tilde{\boldsymbol{u}}_i \|^2$$ while leveraging the assumption that the true covariance matrix having size $$p$$ is drawn from a matrix ensemble with known spectral properties—particularly, we assume the distribution of population eigenvalues weakly converges as $$p\to \infty $$ to a spectral density $$\rho (\lambda )$$ and that the spacing between population eigenvalues is similar to that for the Gaussian orthogonal ensemble. Our approach complements previous analyses of eigenvector error that require the full set of eigenvalues to be known, which can be computationally infeasible when $$p$$ is large. To provide a scalable approach for uncertainty quantification of eigenvector error, we consider a fixed eigenvalue $$\lambda $$ and approximate the distribution of the expected square error $$r= \mathbb{E}\left [\| \boldsymbol{u}_i - \tilde{\boldsymbol{u}}_i \|^2\right ]$$ across the matrix ensemble for all $$\boldsymbol{u}_i$$ associated with $$\lambda _i=\lambda $$. We find, for example, that for sufficiently large matrix size $$p$$ and sample size $n> p$, the probability density of $$r$$ scales as $1/nr^2$. This power-law scaling implies that the eigenvector error is extremely heterogeneous—even if $$r$$ is very small for most eigenvectors, it can be large for others with non-negligible probability. We support this and further results with numerical experiments.more » « less
-
Hughes, Kim (Ed.)Abstract Evolvable traits of organisms can alter the environment those organisms experience. While it is well appreciated that those modified environments can influence natural selection to which organisms are exposed, they can also influence the expression of genetic variances and covariances of traits under selection. When genetic variance and covariance change in response to changes in the evolving, modified environment, rates and outcomes of evolution also change. Here we discuss the basic mechanisms whereby organisms modify their environments, review how those modified environments have been shown to alter genetic variance and covariance, and discuss potential evolutionary consequences of such dynamics. With these dynamics, responses to selection can be more rapid and sustained, leading to more extreme phenotypes, or they can be slower and truncated, leading to more conserved phenotypes. Patterns of correlated selection can also change, leading to greater or less evolutionary independence of traits, or even causing convergence or divergence of traits, even when selection on them is consistent across environments. Developing evolutionary models that incorporate changes in genetic variances and covariances when environments themselves evolve requires developing methods to predict how genetic parameters respond to environments—frequently multifactorial environments. It also requires a population-level analysis of how traits of collections of individuals modify environments for themselves and/or others in a population, possibly in spatially explicit ways. Despite the challenges of elucidating the mechanisms and nuances of these processes, even qualitative predictions of how environment-modifying traits alter evolutionary potential are likely to improve projections of evolutionary outcomes.more » « less
