More Like this

1. Multi-scale Fisher’s independence test for multivariate dependence

   https://doi.org/10.1093/biomet/asac013  

   Gorsky, S; Ma, L (February 2022, Biometrika)

   Summary Identifying dependency in multivariate data is a common inference task that arises in numerous applications. However, existing nonparametric independence tests typically require computation that scales at least quadratically with the sample size, making it difficult to apply them in the presence of massive sample sizes. Moreover, resampling is usually necessary to evaluate the statistical significance of the resulting test statistics at finite sample sizes, further worsening the computational burden. We introduce a scalable, resampling-free approach to testing the independence between two random vectors by breaking down the task into simple univariate tests of independence on a collection of $$2\times 2$$ contingency tables constructed through sequential coarse-to-fine discretization of the sample , transforming the inference task into a multiple testing problem that can be completed with almost linear complexity with respect to the sample size. To address increasing dimensionality, we introduce a coarse-to-fine sequential adaptive procedure that exploits the spatial features of dependency structures. We derive a finite-sample theory that guarantees the inferential validity of our adaptive procedure at any given sample size. We show that our approach can achieve strong control of the level of the testing procedure at any sample size without resampling or asymptotic approximation and establish its large-sample consistency. We demonstrate through an extensive simulation study its substantial computational advantage in comparison to existing approaches while achieving robust statistical power under various dependency scenarios, and illustrate how its divide-and-conquer nature can be exploited to not just test independence, but to learn the nature of the underlying dependency. Finally, we demonstrate the use of our method through analysing a dataset from a flow cytometry experiment. 

   more » « less
2. A new ranking scheme for modern data and its application to two-sample hypothesis testing

   Zhou, Doudou; Chen, Hao (July 2023, Proceedings of Machine Learning Research)

   Rank-based approaches are among the most popular nonparametric methods for univariate data in tackling statistical problems such as hypothesis testing due to their robustness and effectiveness. However, they are unsatisfactory for more complex data. In the era of big data, high-dimensional and non-Euclidean data, such as networks and images, are ubiquitous and pose challenges for statistical analysis. Existing multivariate ranks such as component-wise, spatial, and depth-based ranks do not apply to non-Euclidean data and have limited performance for high-dimensional data. Instead of dealing with the ranks of observations, we propose two types of ranks applicable to complex data based on a similarity graph constructed on observations: a graph-induced rank defined by the inductive nature of the graph and an overall rank defined by the weight of edges in the graph. To illustrate their utilization, both the new ranks are used to construct test statistics for the two-sample hypothesis testing, which converge to the $$\chi_2^2$$ distribution under the permutation null distribution and some mild conditions of the ranks, enabling an easy type-I error control. Simulation studies show that the new method exhibits good power under a wide range of alternatives compared to existing methods. The new test is illustrated on the New York City taxi data for comparing travel patterns in consecutive months and a brain network dataset comparing male and female subjects. 

   more » « less
3. Pitman efficiency lower bounds for multivariate distribution-free tests based on optimal transport

   https://doi.org/10.1093/jrsssb/qkaf072  

   Deb, Nabarun; Bhattacharya, Bhaswar B; Sen, Bodhisattva (February 2026, Journal of the Royal Statistical Society Series B: Statistical Methodology)

   Abstract The Wilcoxon rank sum test is one of the most popular distribution-free two-sample tests for univariate data. Among the important reasons for their popularity are the striking results of Hodges–Lehmann and Chernoff–Savage, where the authors show that the asymptotic (Pitman) relative efficiency of Wilcoxon’s test compared to Student’s t-test, never falls below 0.864 (with identity score) and 1 (with Gaussian score), respectively. Motivated by these results, we propose and study a large family of exactly distribution-free multivariate rank-based two-sample tests by leveraging the theory of optimal transport. First, we propose distribution-free analogues of the Hotelling T2 test and show that they satisfy Hodges–Lehmann and Chernoff–Savage-type efficiency lower bounds over natural sub-families of multivariate distributions—making them the first multivariate, nonparametric, finite-sample distribution-free tests that provably achieve such efficiency lower bounds. Next, we propose exactly distribution-free versions of the celebrated kernel maximum mean discrepancy test. In addition to being distribution-free in finite-samples, these tests are universally consistent under no moment assumptions and have nontrivial Pitman efficiency. To the best of our knowledge, these are the first class of tests to have this trifecta of properties. 

   more » « less
4. Reconstructing SNP allele and genotype frequencies from GWAS summary statistics

   https://doi.org/10.1038/s41598-022-12185-6  

   Yang, Zhiyu; Paschou, Peristera; Drineas, Petros (December 2022, Scientific Reports)

   Abstract The emergence of genome-wide association studies (GWAS) has led to the creation of large repositories of human genetic variation, creating enormous opportunities for genetic research and worldwide collaboration. Methods that are based on GWAS summary statistics seek to leverage such records, overcoming barriers that often exist in individual-level data access while also offering significant computational savings. Such summary-statistics-based applications include GWAS meta-analysis, with and without sample overlap, and case-case GWAS. We compare performance of leading methods for summary-statistics-based genomic analysis and also introduce a novel framework that can unify usual summary-statistics-based implementations via the reconstruction of allelic and genotypic frequencies and counts (ReACt). First, we evaluate ASSET, METAL, and ReACt using both synthetic and real data for GWAS meta-analysis (with and without sample overlap) and find that, while all three methods are comparable in terms of power and error control, ReACt and METAL are faster than ASSET by a factor of at least hundred. We then proceed to evaluate performance of ReACt vs an existing method for case-case GWAS and show comparable performance, with ReACt requiring minimal underlying assumptions and being more user-friendly. Finally, ReACt allows us to evaluate, for the first time, an implementation for calculating polygenic risk score (PRS) for groups of cases and controls based on summary statistics. Our work demonstrates the power of GWAS summary-statistics-based methodologies and the proposed novel method provides a unifying framework and allows further extension of possibilities for researchers seeking to understand the genetics of complex disease. 

   more » « less
5. High-dimensional analysis of variance in multivariate linear regression

   https://doi.org/10.1093/biomet/asad001  

   Lou, Zhipeng; Zhang, Xianyang; Wu, Wei Biao (January 2023, Biometrika)

   Summary In this paper, we develop a systematic theory for high-dimensional analysis of variance in multivariate linear regression, where the dimension and the number of coefficients can both grow with the sample size. We propose a new U-type statistic to test linear hypotheses and establish a high-dimensional Gaussian approximation result under fairly mild moment assumptions. Our general framework and theory can be used to deal with the classical one-way multivariate analysis of variance, and the nonparametric one-way multivariate analysis of variance in high dimensions. To implement the test procedure, we introduce a sample-splitting-based estimator of the second moment of the error covariance and discuss its properties. A simulation study shows that our proposed test outperforms some existing tests in various settings. 

   more » « less