<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>On relationships between Chatterjee’s and Spearman’s correlation coefficients</title></titleStmt>
			<publicationStmt>
				<publisher>Taylor and Francis</publisher>
				<date>02/01/2024</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10511234</idno>
					<idno type="doi">10.1080/03610926.2024.2309971</idno>
					<title level='j'>Communications in Statistics - Theory and Methods</title>
<idno>0361-0926</idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Qingyang Zhang</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[In his seminal work, Chatterjee (Citation2021) introduced a novel correlation measure that is distribution-free, asymptotically normal, and consistent against all alternatives. In this article, we study the probabilistic relationships between Chatterjee’s correlation and the widely used Spearman’s correlation. We show that, under independence, the two sample-based correlations are asymptotically joint normal and asymptotically independent. Under dependence, the magnitudes of two correlations can be substantially different. We establish some extreme cases featuring large differences between these two correlations. Motivated by these findings, a new independence test is proposed by combining Chatterjee’s and Spearman’s correlations into a maximal strength measure of variable association. Our simulation study and real-data application show the good sensitivity of the new test to different correlation patterns.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><p>The limiting quantity &#958;(X, Y) is also known as Dette-Siburg-Stoimenov's dependence measure, <ref type="bibr">Dette, Siburg, and Stoimenov (2013)</ref>,whichisbetween0and1(0ifandonlyifX and Y are independent, 1 if and only if Y is a measurable function of X). Chatterjee (2021)alsoestablished the asymptotic normality of &#958; n (X, Y) under independence. Precisely, &#8730; n&#958; n (X, Y) d -&#8594; N(0, 2/5),a sn &#8594;&#8734;.TheCentralLimitTheoremof&#958; n (X, Y) under dependence (as long as Y is not a measurable function of X)isrecentlyprovedbyLinandHan(2022). In addition, <ref type="bibr">Auddy, Deb, and Nandy (2021)</ref> investigated the limiting power of &#958; n (X, Y) under local alternatives and obtained the exact detection threshold <ref type="bibr">Auddy, Deb, and Nandy (2021)</ref>. The fast-growing literature on Chatterjee's correlation also includes <ref type="bibr">Shi, Drton, and Han (2022)</ref>, <ref type="bibr">Shi, Drton, and Han (2021)</ref>, <ref type="bibr">Lin and Han (2021)</ref>, <ref type="bibr">Cao and Bickel (2020)</ref>, <ref type="bibr">Deb, Ghosal, and Sen (2020)</ref>, <ref type="bibr">Han and Huang (2022)</ref>, <ref type="bibr">Azadkia and Chatterjee (2021)</ref>, <ref type="bibr">Zhang (2023)</ref>, <ref type="bibr">Chatterjee and Vidyasagar (2022)</ref>, among many others.</p><p>In addition to the simple asymptotic theory, as an empirical finding, Chatterjee's test is powerful to detect non monotonic associations, especially those with an oscillating nature such as the W-shaped scatterplot and the sinusoid <ref type="bibr">Chatterjee (2021)</ref>. The only disadvantage of Chatterjee's test is that it may have less statistical power for smoother alternatives (such as linear or other monotonic relationships) compared to other popular tests, including distance correlation test and Bergsma-Dassios test. For instance, as shown in Figure <ref type="figure">5</ref> of <ref type="bibr">Chatterjee (2021)</ref>, the power of &#958; n (X, Y) quickly deteriorates as the noise level increases, which could be a matter of concern in practice. Motivated by these facts, we propose a versatile test by taking the maximum of Chatterjee's correlation and Spearman's correlation, where the latter one is powerful to detect monotonic and smoother associations. Two questions arising from this proposal are (1) What is the asymptotic joint distribution of the two correlations under independence?</p><p>(2) How much can they differ as a measure of dependence?</p><p>The first question is about the analytical calculation of p-values. The second question investigates if the two correlations to be combined are complementary in the sense that they measure different dependencies. In this article, we give a complete answer to the first question. For the second question, we provide two extreme examples featuring large differences between the two correlations. The idea of combining two complementary correlation metrics is not new. For instance, <ref type="bibr">Zhang, Qi, and Ma (2011)</ref> showed the asymptotic independence between Pearson's correlation and a quotient-type correlation and proposed a new test by combining them into a maximal type measure <ref type="bibr">Zhang, Qi, and Ma (2011)</ref>.</p><p>The remainder of this article is structured as follows: Section 2 derives the asymptotic joint distribution of S n (X, Y) and &#958; n (X, Y) under independence. Section 3 investigates how much the two correlations can differ under dependence. Section 4 proposes the new test of independence, validated by both synthetic data and a real-world dataset. Section 5 discusses the paper with some future perspectives.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.">Asymptotic joint distribution under independence</head><p>With the same notations in previous section, Spearman's rank correlation can be written as</p><p>where R i represents the rank of Y (i) , i = 1, ..., n. Under the hypothesis of independence, it</p><p>-&#8594; N(0, 1),a sn &#8594;&#8734;.Though&#958; n (X, Y) and S n (X, Y) are both asymptotically normal, their joint behavior remains unexplored. In this section, we derive the asymptotic joint distribution of &#958; n (X, Y) and S n (X, Y) under independence.</p><p>Let [n] :={1, 2, ..., n} be the sample indices. Under independence, {R 1 , ..., R n } is a random permutation of <ref type="bibr">[n]</ref>.W efi r s ts h o wt h a t&#958; n (X, Y) and S n (X, Y) are uncorrelated for a finite sample, as stated in the following lemma:</p><p>Proof. Spearman's correlation can be rewritten as</p><p>.</p><p>For the covariance between &#958; n (X, Y) and S n (X, Y),wehave</p><p>The following results <ref type="bibr">(Lin and Han (2021)</ref>, Lemma 6.1, page 13) are needed for our derivations</p><p>For the first term in Equation (2), we have</p><p>Similarly for the second term, we have</p><p>For the third term, we have </p><p>N e x t ,w ep r e s e n tal e m m at h a te s t a b l i s h e st h eC e n t r a lL i m i tT h e o r e mf o r {S n (X, Y), &#958; n (X, Y)}. The key steps to prove Lemma 2 include (1) the coupling method for permutation oscillation proposed by <ref type="bibr">Angus (1995)</ref> (2) the Central Limit Theorem for m-dependent sequence, and (3) Cramer-Wold device. The detailed proof is a bit lengthy, and we provide it in Appendix.</p><p>Lemma 2. If X and Y are independent, &#8730; nS n (X, Y) and &#8730; n&#958; n (X, Y) are asymptotically joint normal.</p><p>By Lemmas 1 and 2, our main theorem follows immediately.</p><p>Theorem 1. If X and Y are independent, &#8730; nS n (X, Y) and &#8730; n&#958; n (X, Y) are asymptotically joint normal and asymptotically independent. To be specific,</p><p>as n &#8594;&#8734;.</p><p>Theorem 1 answers the first question that we asked in Section 1,w h i c he n a b l e st h e analytical calculation of p-values for the proposed integrated test (to be further discussed in Section 4). Theorem 1, together with Lemma 1 and Remark 1, give a complete characterization for the joint behavior of S n (X, Y) and &#958; n (X, Y) under independence. The convergence of {S n (X, Y), &#958; n (X, Y)} t oj o i n tn o r m a l i t y ,a ss a m p l es i z en increases, is illustrated in the Figure <ref type="figure">1</ref> below. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.">Chatterjee's and Spearman's correlations -how much can they differ?</head><p>In this section, we explore the second question outlined in Section 1, that is, how much the two correlations can differ as a metric of dependence. We focus on some extremal cases where the magnitudes of &#958; n (X, Y) and S n (X, Y) are largely different. For dependent variables X and Y, &#958; n (X, Y) is generally, though not always, between 0 and 1, while S n (X, Y) is between -1and1, therefore we take the absolute value of S n (X, Y),andcompare&#958; n (X, Y) and |S n (X, Y)| instead. We first provide an extremal case where the absolute Spearman's correlation is small but Chatterjee's correlation is large. This extremal case is easy to construct using simple symmetric patterns such as</p><p>Proof. Without loss of generality, suppose n is odd. We construct the following ranks</p><p>It is straightforward to show that</p><p>For any given &gt;0, we can find an odd number n,s u c ht h a t(6n -9)/(n 2 -1)&lt; and 3/(2n)&lt; , therefore</p><p>N e x t ,w es e e ka no p p o s i t ee x t r e m a lc a s ew h e r e|S n (X,</p><p>the minimum possible value of &#958; n (X, Y) may not be close to 0. Mathematically, this can be formulated as the following optimization problem Foragiven0&lt; &lt;1, find max</p><p>Unless n is small enough to enumerate all permutations of {R 1 , ..., R n }, the optimization problem above is difficult because of the complicated constraints. It may require advanced i n t e g e rp r o g r a m m i n gt e c h n i q u e s ,w h i c ha r eb e y o n dt h es c o p eo ft h i sw o r k .W el e a v et h e optimization problem as an open question and try to give a simple example instead, where &#958; n (X, Y) is relatively small but |S n (X, Y)| is substantially larger. Our intuition is that Spearman's correlation measures the overall monotonic relations, while Chatterjee's correlation is sensitive to local changes. Accordingly, we construct a case that is overall monotonic but has wiggly local patterns.</p><p>Proof. We const r uc t n = 2m + p ranks which can be partitioned into two parts: the part of 1 &#8804; i &#8804; 2m has an oscillating pattern, while the part of 2m + 1 &#8804; i &#8804; n is monotonically increasing</p><p>For any 1 &gt; &gt;0, there exists</p><p>We give two examples for this extremal case (1) when n = 100, m = 40 and p = 20, &#958; n (X, Y) &#8776; 0.058 while S n (X, Y) &#8776; 0.753 (2) when n = 60, m = 23 and p = 14, &#958; n (X, Y) &#8776; 0.144 while S n (X, Y) &#8776; 0.789, both show substantial difference between the two metrics.</p><p>Beyond extremal cases 1 and 2, we explore the magnitude of the difference when both coefficients clearly show dependence, that is, when both |S n | and &#958; n exceed a certain threshold, for example 0.4. Case 3 is constructed such that |S n | isclosetoanygiventhreshold&#951; while &#958; n approaches 1.</p><p>Proof. Without loss of generality, suppose n = m + p,w h e r em is odd. We construct the following ranks</p><p>and</p><p>Foragiven1&gt;&#951;&gt;0, one can choose n and m,suchthatm/n = (1&#951;) 1/3 . As an example, for &#951; = 0.4, n = 94 and m = 79, we have S n (X, Y) = 0.4 and &#958; n (X, Y) = 0.94.</p><p>In Case 4, &#958; n (X, Y) is close to the given threshold &#951; while |S n (X, Y)| is significantly larger. This can be easily achieved by adapting Case 2. For instance, when n = 60, m = 38, we obtain &#958; n (X, Y) = 0.41 and S n (X, Y) = 0.88. Figure <ref type="figure">2</ref> below presents examples for all four cases discussed here. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.">A new test for independence</head><p>Motivated by the findings in Sections 2 and 3, we propose the following new metric</p><p>As I n (X, Y) takes advantage of both S n (X, Y) and &#958; n (X, Y),itcanbeusedasaversatiletest for detecting both monotonic and non monotonic associations. Moreover, by Theorem 1,one can calculate the asymptotic p-value as follows</p><p>where z &#8805; 0a n d (&#8226;) represents the standard normal distribution function. For a given significance level of &#945;, we reject the null hypothesis if</p><p>The consistency of the new test under fixed alternatives can be established using <ref type="bibr">Shi, Drton, and Han (2022)</ref>. Precisely, the testing power satisfies</p><p>By Proposition 5 of Shi, Drton, and Han (2022), Chatterjee's independence test is consistent for any given &#945;,thatis, n are made by R package independence Even-Zohar (2020). The following six alternatives were considered, where Z &#8764; N(0, 1) and Z &#8869; X 1. Linear:</p><p>Figure <ref type="figure">3</ref> summarizes the empirical power over 5, 000 simulation runs (at the significance level of 0.05). As expected, Spearman's test has the highest power for the monotonic settings, thatis,"Linear"and"Stepwise1" ,butextremelylowpowerforsomenonmonotonicsettings including "Sinusoid" and "Quadratic 1". Chatterjee's test is most powerful for two non Inthesecondstudy ,weexaminedthep-value bias. The exact p-value was approximated using 5, 000 permutations and the bias was computed as the asymptotic p-value minus the exact p-value. In each simulation run, we generated X from Uniform[-1, 1] and Y from N(0, 1) independently with sample size {20, 40, 60, 80, 100}. Figure <ref type="figure">4</ref> summarizes the bias over 1, 000 simulations runs. It can be seen that the asymptotic p-values are overall close to the exact p-values. However, for a relatively small sample size, for example, n = 20, the asymptotic p-valuesispositivelybiased,indicatingtheconservativenessofthetest.Thebias vanishes as sample size increases. In practice, if the sample size is small, for example, n &lt; 30, we recommend a permutation test based on I n (X, Y) to avoid power loss.</p><p>The proposed method was also tested on a transcriptomics dataset by Spellman et al. <ref type="bibr">(1998)</ref>, which contains the expression levels of 6,223 yeast genes over 23 successive time points during the cell cycle <ref type="bibr">Spellman et al. (1998)</ref>. This dataset was processed by <ref type="bibr">Reshef et al. (2011)</ref>, where genes with missing observations were excluded. The processed dataset has 4,381 genes, which are available through the R package minerva. There have been many papers testing different correlation measures using this particular dataset including <ref type="bibr">Chatterjee (2021)</ref>.</p><p>We analyzed this data using S n , &#958; n and I n .Forallthreemethods,p-values were calculated using asymptotic formulas and then adjusted by the Benjamini-Hochberg procedure to control the false discovery rate (FDR) at the level of 0.05. Figure <ref type="figure">5</ref> summarizes the number of significant genes identified by three tests. Out of a total of 4,381 genes, the new test selected 734 genes whose expression levels change during the cell cycle, while the other two tests selected 619 and 385 genes, respectively. This is due to the existence of different expression patterns in the data, that is, some genes have smoother expression change while others have non monotonic such as oscillating expression change. Figure <ref type="figure">6</ref> presents a random sample of four genes that were identified by the new test but missed by Spearman's test. It can be seen that the expression levels of all four genes exhibit certain oscillating patterns. Figure <ref type="figure">7</ref> shows  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.">Discussion and conclusions</head><p>Chatterjee's rank correlation has attracted a lot of attention during the past two years due to its simplicity and nice statistical properties. However, the cost we pay for this simple method is its inferior performance in detecting smoother correlation patterns, such as linear relationships.  To boost the power of this ingenious measure, in this article, we proposed a max-type test by combining Chatterjee's correlation with Spearman's correlation, as the latter one is also rank basedbutsensitivetosmoothcorrelationpatterns.Wederivetheasymptoticjointdistribution of these two correlations under independence, which enables the analytical calculation of pvalues. Our simulation study and the transcriptomics application illustrated the promise of the new test. Due to the simple calculation and satisfactory performance, the test is readily applicable to many correlative analyses, for example, the gene-gene interaction and proteinprotein interaction network construction.</p><p>There are several possible extensions of this work. First, the new test statistic I n (X, Y) is generally asymmetric because &#958; n (X, Y) is asymmetric, that is, &#958; n (X, Y) = &#958; n (Y, X).Whena symmetric measure is more suitable, one can consider the following modification</p><p>In previous work, <ref type="bibr">Zhang (2023)</ref>, we established the asymptotic joint normality of &#958; n (X, Y) and &#958; n (Y, X) a n ds h o w e dt h a tt h es y m m e t r i z e dm e t r i c ,t h a ti s ,m a x {&#958; n (X, Y), &#958; n (Y, X)}, converges to a skew normal distribution under independence. The proof is based on Chatterjee's Central Limit Theorem <ref type="bibr">Chatterjee (2008)</ref>. The joint asymptotic behavior of</p><p>} could be studied in a similar way, and the first and most important step is to construct a valid interaction rule for I sym n (X, Y) Auddy, Deb, and Nandy (2021); <ref type="bibr">Zhang (2023)</ref>. Although the asymptotic theory of I sym n remains unknown, it is easy to carry out a permutation test for independence using this statistic. Figure <ref type="figure">8</ref> shows the empirical power of I sym n and I n under the same simulation settings as those described in Section 4,whereI sym n exhibits satisfactory performance in all settings, albeit with slightly lower power than I n .</p><p>Second, one can consider generalizing our test by replacing &#958; n with its modified version, L i na n dH a n( 2021). The modified statistic &#958; n,M incorporates M right nearest neighbors, which is also asymptotically normal but generally more powerful than &#958; n .N otably ,Linand <ref type="bibr">Han (2021)</ref> showed that the modified test achieves near-parametric efficiency in testing against Gaussian rotation alternatives. The simulations using permutation tests (Figure <ref type="figure">9</ref>) confirm that replacing the traditional Chatterjee's coefficient &#958; n with &#958; n,M results in consistent power improvement. Notably, the generalized test (I n,M = &#8730; n max(|S n |, &#8730; 5M/2&#958; n,M ),where M = 10, 20) outperforms the original test across all six settings with different sample sizes, albeit with slight but consistent gains.</p><p>Theoretically, it would be important to study the joint behavior of S n and &#958; n,M and derive theasymptoticnulldistributionofI n,M . One possible approach involves leveraging the H&#225;jek representation from <ref type="bibr">Lin and Han (2021)</ref>. However, a significant obstacle exists concerning the application of Chatterjee's CLT to locally dependent sequences. Recall that &#958; n,M is defined as</p><p>where m(i) is the index of the mth right nearest neighbor of X i . <ref type="bibr">Lin and Han (2021)</ref>established its asymptotic normality under independence by leveraging the H&#225;jek representation of &#958; n,M (see &#958;n,M in Section A.2.1, proof of Theorem 3.2). Their proof hinges on demonstrating the asymptotic equivalence in distribution between &#958; n,M and its H&#225;jek representation. This allows them to focus on the asymptotic normality of &#958;n,M , which can be further established using Chatterjee's CLT based on a well-defined interaction rule. Following this approach, we define the H&#225;jek representation of a&#958; n,M + bS n for constants a and b as follows</p><p>Then an asymptotically equivalent version with local dependence is</p><p>It suffices to show the asymptotic normality of a &#958;n,M + b &#352;n . However, applying Chatterjee's CLT necessitates a valid interaction rule G that satisfies all conditions outlined in Theorem 2.5 of <ref type="bibr">Chatterjee (2008)</ref>. The interaction rule employed by <ref type="bibr">Lin and Han (2021)</ref>, unfortunately, does not apply in this regard. Specifically, the function f i (Z) in their proof (see Section A.2.1,</p><p>Step III-1, Lin and Han (2021)) becomes</p><p>The partial rank correlation, defined as</p><p>) is an adaptation of Pearson's partial correlation that replaces the actual values with their ranks. However, as noted in <ref type="bibr">Kendall (1942)</ref>, this definition lacks justification, and its interpretation is unclear (e.g., whether it truly measures conditional linear relationships). <ref type="bibr">Liu et al. (2018)</ref> proposed a sound definition for conditional Spearman's correlation based on a kernel-based estimator involving residual calculations (see Sections 3.2-3.3 in <ref type="bibr">Liu et al. (2018)</ref>). This definition, however, is not rank-based, making it challenging to analyze its joint behavior with T n (X, Y|Z).</p><p>Deriving the asymptotic distribution of the integrated statistic presents another significant challenge. Even for the simple rank-based estimator S n (X, Y|Z),o b t a i n i n gi t sa s y m p t o t i c null distribution is difficult (although simulations suggest normality). For instance, applying LinandHan(2021)' smethodprovedunfeasibleduetothedifficultyinestablishingaH&#225;jek representation of S n (X, Y|Z) that readily transforms into a locally dependent sequence. The H&#225;jek representation of T n (X, Y|Z) is more straightforward, which can be written as a linear combination (see the representation &#958; # n in A.2.2, proof of Theorem 3.1, Shi, Drton, and Han (2021)). Additionally, even under conditional independence, the asymptotic distributions of S 2 n (X, Y), S 2 n (X, Z) and S 2 n (Y, Z) remain elusive due to potential pairwise dependencies among X, Y and Z.</p><p>Finally, as we discussed in the simulation study, the asymptotic p-value is generally close to the true p-value,butittendstobepositivelybiasedforsmallsample,forexample,n &lt; 30, r e s ul ti n gi nc e rta i npo w e rl o s s .I nth eca seo fs m allsa m p l e ,w er ec o m m e n dape rm u ta ti o n test for better testing performance. Another way to reduce the potential p-value bias is to use asymptotic expansion method, for example, Edgeworth expansion, Cornish-Fisher expansion, or saddle point approximation, which may improve p-value approximation by incorporating higher-order moments such as skewness and kurtosis.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Appendix: Proof of Lemma 2</head><p>Proof. We first define F(y) = P(Y &lt; y), U i = F(Y (i) ), F n (y) = n i=1 1{Y (i) &#8804; y}/n,a n d H n (x) = n i=1 1{U i &#8804; x}/n. For Chatterjee's correlation, using Equations ( <ref type="formula">5</ref>)-(8) in Angus (1995), we have</p><p>where Z P -&#8594; 0. For Spearman's correlation, we define the following function </p><p>where the second term can be rewritten as</p><p>ThefirstterminEquation(A.1),usingcontinuousmappingtheorem,hasthesamelimiting distribution as For any two constants, a and b,define</p><p>where</p><p>Itcanbeseenthatforanyj &#8805; 1, Z i+j is independent of [Z 1 , ..., Z i ], therefore the sequence {Z i } is 1-dependent sequence. Similar to Equations ( <ref type="formula">11</ref>)-( <ref type="formula">14</ref>) in <ref type="bibr">Angus (1995)</ref>, we have</p><p>where Var[a|U i+1 -U i |] = a 2 18 ,</p></div></body>
		</text>
</TEI>
