An official website of the United States government
Abstract Observational data inherently contain noise which manifests as uncertainties in the measured parameters and creates positive biases or noise floors in second‐order products like variances, fluxes, and spectra. Historical methods estimate and subsequently subtract noise floors, but struggle with accuracy. Gardner and Chu (2020,doi.org/10.1364/AO.400375) proposed an interleaved data processing method, which inherently eliminates biases from variances and fluxes, and suggested that the method could also eliminate noise floors of power spectra. We investigate the interleaved method for spectral analysis of atmospheric waves through theoretical studies, forward modeling, and demonstration with lidar data. Our work shows that calculating the cross‐power spectral density (CPSD) from two interleaved subsamples does reduce the spectral noise floor significantly. However, only the Co‐PSD (the real part of CPSD) eliminates the noise floor completely, while taking the absolute magnitude of CPSD adds a reduced noise floor back to the spectrum when the sample number is finite. This reduced noise floor can be further minimized through averaging over more observations, completely different from traditional spectrum calculations whose noise floor cannot be reduced by incorporating more samples. We demonstrate the first application of the interleaved method to spectral data, successfully eliminating the noise floor using the Co‐PSD in a forward model and in lidar observations of the vertical wavenumber of gravity waves at McMurdo, Antarctica. This high accuracy is gained by sacrificing precision due to photon‐count splitting, requiring additional observations to counter this effect. We provide quantitative assessment of accuracy and precision as well as application recommendations.
more »
« less
Eliminating photon noise biases in the computation of second-order statistics of lidar temperature, wind, and species measurements
The precision of lidar measurements is limited by noise associated with the optical detection process. Photon noise also introduces biases in the second-order statistics of the data, such as the variances and fluxes of the measured temperature, wind, and species variations, and establishes noise floors in the computed fluctuation spectra. When the signal-to-noise ratio is low, these biases and noise floors can completely obscure the atmospheric processes being observed. We describe a novel data processing technique for eliminating the biases and noise floors. The technique involves acquiring two statistically independent datasets, covering the same altitude range and time period, from which the various second-order statistics are computed. The efficacy of the technique is demonstrated using Na Doppler lidar observations of temperature in the upper mesosphere and lower thermosphere acquired recently at McMurdo Station, Antarctica. The results show that this new technique enables observations of key atmospheric parameters in regions where the signal-to-noise ratio is far too low to apply conventional processing approaches.
more »
« less
- PAR ID:
- 10191865
- Publisher / Repository:
- Optical Society of America
- Date Published:
- Journal Name:
- Applied Optics
- Volume:
- 59
- Issue:
- 27
- ISSN:
- 1559-128X; APOPAI
- Format(s):
- Medium: X Size: Article No. 8259
- Size(s):
- Article No. 8259
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Random‐noise‐induced biases are inherent issues to the accurate derivation of second‐order statistical parameters (e.g., variances, fluxes, energy densities, and power spectra) from lidar and radar measurements. We demonstrate here for the first time an altitude‐interleaved method for eliminating such biases, following the original proposals by Gardner and Chu (2020,https://doi.org/10.1364/ao.400375) who demonstrated a time‐interleaved method. Interleaving in altitude bins provides two statistically independent samples over the same time period and nearly the same altitude range, thus enabling the replacement of variances that include the noise‐induced biases with covariances that are intrinsically free of such biases. Comparing the interleaved method with previous variance subtraction (VS) and spectral proportion (SP) methods using gravity wave potential energy density calculated from Antarctic lidar data and from a forward model, this study finds the accuracy and precision of each method differing in various conditions, each with its own strengths and weakness. VS performs well in high‐SNR, yet its accuracy fails at lower‐SNR as it often yields negative values. SP is accurate and precise under high‐SNR, remaining accurate in worse conditions than VS would, yet develops a positive bias under low‐SNR. The interleaved method is accurate in all SNRs but requires a large number of samples to drive random‐noise terms in covariances toward zero and to compensate for the reduced precision due to the splitting of return signals. Therefore, selecting the proper bias removal/elimination method for actual signal and sample conditions is crucial in utilizing lidar/radar data, as neglecting this can conceal trends or overstate atmospheric variability.more » « less
-
null (Ed.)Continuous advancements in LiDAR technology have enabled compelling wind turbulence measurements within the atmospheric boundary layer with range gates shorter than 20 m and sampling frequency of the order of 10 Hz. However, estimates of the radial velocity from the back-scattered laser beam are inevitably affected by an averaging process within each range gate, generally modeled as a convolution between the actual velocity projected along the LiDAR line-of-sight and a weighting function representing the energy distribution of the laser pulse along the range gate. As a result, the spectral energy of the turbulent velocity fluctuations is damped within the inertial sub-range with respective reduction of the velocity variance, and, thus, not allowing to take advantage of the achieved spatio-temporal resolution of the LiDAR technology. In this article, we propose to correct this turbulent energy damping on the LiDAR measurements by reversing the effect of a low-pass filter, which can be estimated directly from the LiDAR measurements. LiDAR data acquired from three different field campaigns are analyzed to describe the proposed technique, investigate the variability of the filter parameters and, for one dataset, assess the procedure for spectral LiDAR correction against sonic anemometer data. It is found that the order of the low-pass filter used for modeling the energy damping on the LiDAR velocity measurements has negligible effects on the correction of the second-order statistics of the wind velocity. In contrast, its cutoff frequency plays a significant role in the spectral correction encompassing the smoothing effects connected with the LiDAR gate length.more » « less
-
Since its development, the minimax framework has been one of the cornerstones of theoretical statistics, and has contributed to the popularity of many well-known estimators, such as the regularized M-estimators for high-dimensional problems. In this paper, we will first show through the example of sparse Gaussian sequence model, that the theoretical results under the classical minimax framework are insufficient for explaining empirical observations. In particular, both hard and soft thresholding estimators are (asymptotically) minimax, however, in practice they often exhibit sub-optimal performances at various signal-to-noise ratio (SNR) levels. The first contribution of this paper is to demonstrate that this issue can be resolved if the signal-to-noise ratio is taken into account in the construction of the parameter space. We call the resulting minimax framework the signal-to-noise ratio aware minimaxity. The second contribution of this paper is to showcase how one can use higher-order asymptotics to obtain accurate approximations of the SNR-aware minimax risk and discover minimax estimators. The theoretical findings obtained from this refined minimax framework provide new insights and practical guidance for the estimation of sparse signals.more » « less
-
Massive multiple-input multiple-output (MIMO) communications using low-resolution analog-to-digital converters (ADCs) is a promising technology for providing high spectral and energy efficiency with affordable hardware cost and power consumption. However, the use of low-resolution ADCs requires special signal processing methods for channel estimation and data detection since the resulting system is severely non-linear. This paper proposes joint channel estimation and data detection methods for massive MIMO systems with low-resolution ADCs based on the variational Bayes (VB) inference framework. We first derive matched-filter quantized VB (MF-QVB) and linear minimum mean-squared error quantized VB (LMMSE-QVB) detection methods assuming the channel state information (CSI) is available. Then we extend these methods to the joint channel estimation and data detection (JED) problem and propose two methods we refer to as MF-QVB-JED and LMMSE-QVB-JED. Unlike conventional VB-based detection methods that assume knowledge of the second-order statistics of the additive noise, we propose to float the elements of the noise covariance matrix as unknown random variables that are used to account for both the noise and the residual inter-user interference. We also present practical aspects of the QVB framework to improve its implementation stability. Finally, we show via numerical results that the proposed VB-based methods provide robust performance and also significantly outperform existing methods.more » « less
