More Like this

1. Approximate MLE of High-Dimensional STAP Covariance Matrices With Banded and Spiked Structure—A Convex Relaxation Approach

   https://doi.org/10.1109/TAES.2025.3636131  

   Jain, Shashwat; Krishnamurthy, Vikram; Rangaswamy, Muralidhar; Gogineni, Sandeep; Kang, Bosung; O'Rourke, Sean M (January 2026, IEEE Transactions on Aerospace and Electronic Systems)

   Estimating the clutter-plus-noise covariance matrix in highdimensional space-time adaptive processing is challenging in the presence of internal clutter motion (ICM) and a high noise floor. The problem becomes more difficult in low-sample regimes, where the sample covariance matrix becomes ill-conditioned. To capture the ICM and high noise floor, we model the covariance matrix using a “Banded+Spiked” structure. Since the maximum likelihood estimation for this model is nonconvex, we propose a convex relaxation, which is formulated as a Frobenius norm minimization with nonsmooth convex constraints enforcing banded sparsity. This relaxation serves as a provable upper bound for the nonconvex likelihood maximization and extends to cases where the covariance matrix dimension exceeds the number of samples. We derive a variational inequality-based bound to assess its quality. We introduce an algorithm to jointly estimate the banded clutter covariance and noise power. In addition, we establish conditions ensuring the estimated covariance matrix remains positive definite and the bandsize is accurately recovered. Numerical results using the high-fidelity RFView radar simulation environment demonstrate that our algorithm achieves a higher signal-to-clutterplus-noise ratio than state-of-the-art methods, including tapered or banded shrinkage covariance, spiked covariance Stein shrinkage, and diagonal loading, particularly when the covariance matrix dimension exceeds the number of samples. 

   more » « less
2. Estimation of the Number of Spiked Eigenvalues in a Covariance Matrix by Bulk Eigenvalue Matching Analysis

   https://doi.org/10.1080/01621459.2021.1933497  

   Ke, Zheng Tracy; Ma, Yucong; Lin, Xihong (July 2021, Journal of the American Statistical Association)

   null (Ed.)

   The spiked covariance model has gained increasing popularity in high-dimensional data analysis. A fundamental problem is determination of the number of spiked eigenvalues, K. For estimation of K, most attention has focused on the use of top eigenvalues of sample covariance matrix, and there is little investigation into proper ways of using bulk eigenvalues to estimate K. We propose a principled approach to incorporating bulk eigenvalues in the estimation of K. Our method imposes a working model on the residual covariance matrix, which is assumed to be a diagonal matrix whose entries are drawn from a gamma distribution. Under this model, the bulk eigenvalues are asymptotically close to the quantiles of a fixed parametric distribution. This motivates us to propose a two-step method: the first step uses bulk eigenvalues to estimate parameters of this distribution, and the second step leverages these parameters to assist the estimation of K. The resulting estimator Kˆ aggregates information in a large number of bulk eigenvalues. We show the consistency of Kˆ under a standard spiked covariance model. We also propose a confidence interval estimate for K. Our extensive simulation studies show that the proposed method is robust and outperforms the existing methods in a range of scenarios. We apply the proposed method to analysis of a lung cancer microarray dataset and the 1000 Genomes dataset. 

   more » « less
3. Matrix denoising with partial noise statistics: optimal singular value shrinkage of spiked F-matrices

   https://doi.org/10.1093/imaiai/iaad028  

   Gavish, Matan; Leeb, William; Romanov, Elad (July 2023, Information and Inference: A Journal of the IMA)

   We study the problem of estimating a large, low-rank matrix corrupted by additive noise of unknown covariance, assuming one has access to additional side information in the form of noise-only measurements. We study the Whiten-Shrink-reColour (WSC) workflow, where a ‘noise covariance whitening’ transformation is applied to the observations, followed by appropriate singular value shrinkage and a ‘noise covariance re-colouring’ transformation. We show that under the mean square error loss, a unique, asymptotically optimal shrinkage nonlinearity exists for the WSC denoising workflow, and calculate it in closed form. To this end, we calculate the asymptotic eigenvector rotation of the random spiked F-matrix ensemble, a result which may be of independent interest. With sufficiently many pure-noise measurements, our optimally tuned WSC denoising workflow outperforms, in mean square error, matrix denoising algorithms based on optimal singular value shrinkage that do not make similar use of noise-only side information; numerical experiments show that our procedure’s relative performance is particularly strong in challenging statistical settings with high dimensionality and large degree of heteroscedasticity. 

   more » « less
4. Improved Shrinkage Prediction under a Spiked Covariance Structure

   Trambak Banerjee, Gourab Mukherjee (July 2021, Journal of machine learning research)

   We develop a novel shrinkage rule for prediction in a high-dimensional non-exchangeable hierarchical Gaussian model with an unknown spiked covariance structure. We propose a family of priors for the mean parameter, governed by a power hyper-parameter, which encompasses independent to highly dependent scenarios. Corresponding to popular loss functions such as quadratic, generalized absolute, and Linex losses, these prior models induce a wide class of shrinkage predictors that involve quadratic forms of smooth functions of the unknown covariance. By using uniformly consistent estimators of these quadratic forms, we propose an efficient procedure for evaluating these predictors which outperforms factor model based direct plug-in approaches. We further improve our predictors by considering possible reduction in their variability through a novel coordinate-wise shrinkage policy that only uses covariance level information and can be adaptively tuned using the sample eigen structure. Finally, we extend our disaggregate model based methodology to prediction in aggregate models. We propose an easy-to-implement functional substitution method for predicting linearly aggregated targets and establish asymptotic optimality of our proposed procedure. We present simulation experiments as well as real data examples illustrating the efficacy of the proposed method. 

   more » « less
5. Spectral diagonal ensemble Kalman filters

   https://doi.org/10.5194/npg-22-485-2015  

   Kasanický, I.; Mandel, J.; Vejmelka, M. (January 2015, Nonlinear Processes in Geophysics)

   A new type of ensemble Kalman filter is developed, which is based on replacing the sample covariance in the analysis step by its diagonal in a spectral basis. It is proved that this technique improves the approximation of the covariance when the covariance itself is diagonal in the spectral basis, as is the case, e.g., for a second-order stationary random field and the Fourier basis. The method is extended by wavelets to the case when the state variables are random fields which are not spatially homogeneous. Efficient implementations by the fast Fourier transform (FFT) and discrete wavelet transform (DWT) are presented for several types of observations, including high-dimensional data given on a part of the domain, such as radar and satellite images. Computational experiments confirm that the method performs well on the Lorenz 96 problem and the shallow water equations with very small ensembles and over multiple analysis cycles. 

   more » « less