More Like this

1. On relationships between Chatterjee’s and Spearman’s correlation coefficients

   https://doi.org/10.1080/03610926.2024.2309971  

   Zhang, Qingyang (February 2024, Communications in Statistics - Theory and Methods)

   In his seminal work, Chatterjee (Citation2021) introduced a novel correlation measure that is distribution-free, asymptotically normal, and consistent against all alternatives. In this article, we study the probabilistic relationships between Chatterjee’s correlation and the widely used Spearman’s correlation. We show that, under independence, the two sample-based correlations are asymptotically joint normal and asymptotically independent. Under dependence, the magnitudes of two correlations can be substantially different. We establish some extreme cases featuring large differences between these two correlations. Motivated by these findings, a new independence test is proposed by combining Chatterjee’s and Spearman’s correlations into a maximal strength measure of variable association. Our simulation study and real-data application show the good sensitivity of the new test to different correlation patterns. 

   more » « less
2. On the power of Chatterjee’s rank correlation

   https://doi.org/10.1093/biomet/asab028  

   Shi, H; Drton, M; Han, F (October 2021, Biometrika)

   Summary Chatterjee (2021) introduced a simple new rank correlation coefficient that has attracted much attention recently. The coefficient has the unusual appeal that it not only estimates a population quantity first proposed by Dette et al. (2013) that is zero if and only if the underlying pair of random variables is independent, but also is asymptotically normal under independence. This paper compares Chatterjee’s new correlation coefficient with three established rank correlations that also facilitate consistent tests of independence, namely Hoeffding’s $$D$$, Blum–Kiefer–Rosenblatt’s $$R$$, and Bergsma–Dassios–Yanagimoto’s $$\tau^*$$. We compare the computational efficiency of these rank correlation coefficients in light of recent advances, and investigate their power against local rotation and mixture alternatives. Our main results show that Chatterjee’s coefficient is unfortunately rate-suboptimal compared to $$D$$, $$R$$ and $$\tau^*$$. The situation is more subtle for a related earlier estimator of Dette et al. (2013). These results favour $$D$$, $$R$$ and $$\tau^*$$ over Chatterjee’s new correlation coefficient for the purpose of testing independence. 

   more » « less
3. Empirical likelihood method for complete independence test on high-dimensional data

   https://doi.org/10.1080/00949655.2022.2029860  

   Qi, Yongcheng; Zhou, Yingchao (August 2022, Journal of statistical computation and simulation)

   Given a random sample of size n from a p dimensional random vector, we are interested in testing whether the p components of the random vector are mutually independent. This is the so-called complete independence test. In the multivariate normal case, it is equivalent to testing whether the correlation matrix is an identity matrix. In this paper, we propose a one-sided empirical likelihood method for the complete independence test based on squared sample correlation coefficients. The limiting distribution for our one-sided empirical likelihood test statistic is proved to be Z^2I(Z > 0) when both n and p tend to infinity, where Z is a standard normal random variable. In order to improve the power of the empirical likelihood test statistic, we also introduce a rescaled empirical likelihood test statistic. We carry out an extensive simulation study to compare the performance of the rescaled empirical likelihood method and two other statistics. 

   more » « less
4. Correlation Improves Group Testing: Modeling Concentration-Dependent Test Errors

   https://doi.org/10.1287/mnsc.2021.04217  

   Wan, Jiayue; Zhang, Yujia; Frazier, Peter I (July 2026, Management Science)

   Population-wide screening is a powerful tool for controlling infectious diseases. Group testing can enable such screening despite limited resources. Viral concentration of pooled samples are often positively correlated, either because prevalence and sample collection are influenced by location, or through intentional enhancement via pooling samples according to risk or household. Such correlation is known to improve efficiency when test sensitivity is fixed. However, in reality, a test’s sensitivity depends on the concentration of the analyte (e.g., viral RNA), as in the so-called dilution effect, where sensitivity decreases for larger pools. We show that concentration-dependent test error alters correlation’s effect under the most widely used group testing procedure, the two-stage Dorfman procedure. We prove that when test sensitivity increases with concentration: pooling correlated samples together (correlated pooling) achieves asymptotically higher sensitivity than independently pooling the samples (naive pooling). In contrast, in the concentration-independent case, correlation does not affect sensitivity. Moreover, with concentration-dependent errors, correlation can degrade test efficiency compared with naive pooling, whereas under concentration-independent errors, correlation always improves efficiency. We propose an alternative measure of test resource usage, the number of positives found per test consumed, which we argue is better aligned with infection control, and show that correlated pooling outperforms naive pooling on this measure. In simulation, we show that the effect of correlation under realistic concentration-dependent test error is meaningfully different from correlation’s effect assuming fixed sensitivity. Our findings underscore the importance for policy makers of using models that incorporate naturally occurring correlation and of considering ways of strengthening this correlation. This paper was accepted by Carri Chan, healthcare management. Funding: This work was supported by the Provost’s Office of Cornell University, the Air Force Office of Scientific Research [Grant FA9550-19-1-0283], and the National Science Foundation Division of Mathematical Sciences [Grant DMS2230023]. Supplemental Material: The online appendices and data files are available at https://doi.org/10.1287/mnsc.2021.04217 . 

   more » « less
5. On the failure of the bootstrap for Chatterjee’s rank correlation

   https://doi.org/10.1093/biomet/asae004  

   Lin, Zhexiao; Han, Fang (February 2024, Biometrika)

   Abstract While researchers commonly use the bootstrap to quantify the uncertainty of an estimator, it has been noticed that the standard bootstrap, in general, does not work for Chatterjee’s rank correlation. In this paper, we provide proof of this issue under an additional independence assumption, and complement our theory with simulation evidence for general settings. Chatterjee’s rank correlation thus falls into a category of statistics that are asymptotically normal, but bootstrap inconsistent. Valid inferential methods in this case are Chatterjee’s original proposal for testing independence and the analytic asymptotic variance estimator of Lin & Han (2022) for more general purposes. [Received on 5 April 2023. Editorial decision on 10 January 2024] 

   more » « less