An official website of the United States government
ABSTRACT We present posterior sample redshift distributions for the Hyper Suprime-Cam Subaru Strategic Program Weak Lensing three-year (HSC Y3) analysis. Using the galaxies’ photometry and spatial cross-correlations, we conduct a combined Bayesian Hierarchical Inference of the sample redshift distributions. The spatial cross-correlations are derived using a subsample of Luminous Red Galaxies (LRGs) with accurate redshift information available up to a photometric redshift of z < 1.2. We derive the photometry-based constraints using a combination of two empirical techniques calibrated on spectroscopic and multiband photometric data that cover a spatial subset of the shear catalogue. The limited spatial coverage induces a cosmic variance error budget that we include in the inference. Our cross-correlation analysis models the photometric redshift error of the LRGs to correct for systematic biases and statistical uncertainties. We demonstrate consistency between the sample redshift distributions derived using the spatial cross-correlations, the photometry, and the posterior of the combined analysis. Based on this assessment, we recommend conservative priors for sample redshift distributions of tomographic bins used in the three-year cosmological Weak Lensing analyses.
more »
« less
Spatial Correlation Robust Inference
We propose a method for constructing confidence intervals that account for many forms of spatial correlation. The interval has the familiar “estimator plus and minus a standard error times a critical value” form, but we propose new methods for constructing the standard error and the critical value. The standard error is constructed using population principal components from a given “worst‐case” spatial correlation model. The critical value is chosen to ensure coverage in a benchmark parametric model for the spatial correlations. The method is shown to control coverage in finite sample Gaussian settings in a restricted but nonparametric class of models and in large samples whenever the spatial correlation is weak, that is, with average pairwise correlations that vanish as the sample size gets large. We also provide results on the efficiency of the method.
more »
« less
- Award ID(s):
- 1919336
- PAR ID:
- 10436316
- Date Published:
- Journal Name:
- Econometrica
- Volume:
- 90
- Issue:
- 6
- ISSN:
- 0012-9682
- Page Range / eLocation ID:
- 2901 to 2935
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Summary Cross‐validation (CV) is one of the most widely used techniques in statistical learning for estimating the test error of a model, but its behavior is not yet fully understood. It has been shown that standard confidence intervals for test error using estimates from CV may have coverage below nominal levels. This phenomenon occurs because each sample is used in both the training and testing procedures during CV and as a result, the CV estimates of the errors become correlated. Without accounting for this correlation, the estimate of the variance is smaller than it should be. One way to mitigate this issue is by estimating the mean squared error of the prediction error instead using nested CV. This approach has been shown to achieve superior coverage compared to intervals derived from standard CV. In this work, we generalize the nested CV idea to the Cox proportional hazards model and explore various choices of test error for this setting.more » « less
-
null (Ed.)Understanding spatial relations (e.g., laptop on table) in visual input is important for both humans and robots. Existing datasets are insufficient as they lack large-scale, high-quality 3D ground truth information, which is critical for learning spatial relations. In this paper, we fill this gap by constructing Rel3D: the first large-scale, human-annotated dataset for grounding spatial relations in 3D. Rel3D enables quantifying the effectiveness of 3D information in predicting spatial relations on large-scale human data. Moreover, we propose minimally contrastive data collection---a novel crowdsourcing method for reducing dataset bias. The 3D scenes in our dataset come in minimally contrastive pairs: two scenes in a pair are almost identical, but a spatial relation holds in one and fails in the other. We empirically validate that minimally contrastive examples can diagnose issues with current relation detection models as well as lead to sample-efficient training. Code and data are available at https://github.com/princeton-vl/Rel3D.more » « less
-
t is well-known that the statistical performance of Lasso can suffer significantly when the covariates of interest have strong correlations. In particular, the prediction error of Lasso becomes much worse than computationally inefficient alternatives like Best Subset Selection. Due to a large conjectured computational-statistical tradeoff in the problem of sparse linear regression, it may be impossible to close this gap in general. In this work, we propose a natural sparse linear regression setting where strong correlations between covariates arise from unobserved latent variables. In this setting, we analyze the problem caused by strong correlations and design a surprisingly simple fix. While Lasso with standard normalization of covariates fails, there exists a heterogeneous scaling of the covariates with which Lasso will suddenly obtain strong provable guarantees for estimation. Moreover, we design a simple, efficient procedure for computing such a "smart scaling." The sample complexity of the resulting "rescaled Lasso" algorithm incurs (in the worst case) quadratic dependence on the sparsity of the underlying signal. While this dependence is not information-theoretically necessary, we give evidence that it is optimal among the class of polynomial-time algorithms, via the method of low-degree polynomials. This argument reveals a new connection between sparse linear regression and a special version of sparse PCA with a near-critical negative spike. The latter problem can be thought of as a real-valued analogue of learning a sparse parity. Using it, we also establish the first computational-statistical gap for the closely related problem of learning a Gaussian Graphical Model.more » « less
-
We present a novel method, to our knowledge, to synthesize non-trivial speckle patterns that can enable sub-Rayleigh second-order correlation imaging. The speckle patterns acquire a unique anti-correlation in the spatial intensity fluctuation by introducing the blue noise distribution on spatial Fourier power spectrum to the input light fields through amplitude modulation. Illuminating objects with the blue noise speckle patterns can lead to a sub-diffraction limit imaging system with a resolution more than three times higher than first-order imaging, which is comparable to the resolving power of ninth order correlation imaging with thermal light. Our method opens a new route towards non-trivial speckle pattern generation by tailoring amplitudes in spatial Fourier power spectrum of the input light fields and provides a versatile scheme for constructing sub-Rayleigh imaging and microscopy systems without invoking complicated higher-order correlations.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3982/ECTA19465
