More Like this

1. On the relative efficiency of the intent-to-treat Wilcoxon–Mann–Whitney test in the presence of noncompliance

   https://doi.org/10.1093/biomet/asab053  

   Mao, Lu (October 2021, Biometrika)

   Summary A general framework is set up to study the asymptotic properties of the intent-to-treat Wilcoxon–Mann–Whitney test in randomized experiments with nonignorable noncompliance. Under location-shift alternatives, the Pitman efficiencies of the intent-to-treat Wilcoxon–Mann–Whitney and $$t$$ tests are derived. It is shown that the former is superior if the compliers are more likely to be found in high-density regions of the outcome distribution or, equivalently, if the noncompliers tend to reside in the tails. By logical extension, the relative efficiency of the two tests is sharply bounded by least and most favourable scenarios in which the compliers are segregated into regions of lowest and highest density, respectively. Such bounds can be derived analytically as a function of the compliance rate for common location families such as Gaussian, Laplace, logistic and $$t$$ distributions. These results can help empirical researchers choose the more efficient test for existing data, and calculate sample size for future trials in anticipation of noncompliance. Results for nonadditive alternatives and other tests follow along similar lines. 

   more » « less
2. Inference at Scale: Significance Testing for Large Search and Recommendation Experiments

   Ihemelandu, Ngozi; Ekstrand, Michael D. (July 2023, Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23))

   A number of information retrieval studies have been done to assess which statistical techniques are appropriate for comparing systems. However, these studies are focused on TREC-style experiments, which typically have fewer than 100 topics. There is no similar line of work for large search and recommendation experiments; such studies typically have thousands of topics or users and much sparser relevance judgements, so it is not clear if recommendations for analyzing traditional TREC experiments apply to these settings. In this paper, we empirically study the behavior of significance tests with large search and recommendation evaluation data. Our results show that the Wilcoxon and Sign tests show significantly higher Type-1 error rates for large sample sizes than the bootstrap, randomization and t-tests, which were more consistent with the expected error rate. While the statistical tests displayed differences in their power for smaller sample sizes, they showed no difference in their power for large sample sizes. We recommend the sign and Wilcoxon tests should not be used to analyze large scale evaluation results. Our result demonstrate that with Top-\(N\) recommendation and large search evaluation data, most tests would have a 100% chance of finding statistically significant results. Therefore, the effect size should be used to determine practical or scientific significance. 

   more » « less
3. Differentially Private Nonparametric Hypothesis Testing

   https://doi.org/10.1145/3319535.3339821  

   Couch, Simon; Kazan, Zeki; Shi, Kaiyan; Bray, Andrew; Groce, Adam (November 2019, Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS))

   Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we study differentially private tests of independence between a categorical and a continuous variable. We take as our starting point traditional nonparametric tests, which require no distributional assumption (e.g., normality) about the data distribution. We present private analogues of the Kruskal-Wallis, Mann-Whitney, and Wilcoxon signed-rank tests, as well as the parametric one-sample t-test. These tests use novel test statistics developed specifically for the private setting. We compare our tests to prior work, both on parametric and nonparametric tests. We find that in all cases our new nonparametric tests achieve large improvements in statistical power, even when the assumptions of parametric tests are met. 

   more » « less
4. Spatial Variation of Changes in Extreme Discharge Seasonality Across the Northeastern United States

   https://doi.org/10.1002/hyp.15317  

   Richardson, Owen H; Renshaw, Carl E; Magilligan, Francis J (October 2024, Hydrological Processes)

   ABSTRACT The Northeast United States exhibits significant spatial heterogeneity in flood seasonality, with spring snowmelt‐driven floods historically dominating northern areas, while other regions show more varied flood seasonality. While it is well documented that since 1996 there has been a marked increase in extreme precipitation across this region, the response of flood seasonality to these changes in extreme precipitation and the spatial distribution of these effects remain uncertain. Here we show that, historically, snowmelt‐dominated northern regions were relatively insensitive to changes in extreme precipitation. However, with climate warming, the dominance of snowmelt floods is decreasing and thus the extreme flood regimes in northern regions are increasingly susceptible to changes in extreme precipitation. While extreme precipitation increased everywhere in the Northeastern United States in 1996, it has since returned to near pre‐1996 levels in the coastal north while remaining elevated in the inland north. Thus, the inland north region has and continues to experience the greatest changes in extreme flooding seasonality, including a substantial rise in floods outside the historical spring flood season, particularly in smaller watersheds. Further analysis reveals that while early winter floods are increasingly common, the magnitude of cold season floods (Nov‐May) have remained unchanged over time. In contrast, warm season floods (June‐Oct), historically less significant, are now increasing in both frequency and magnitude in the inland north. Our results highlight that treating the entire Northeast as a uniform hydroclimatic region conceals significant regional variations in extreme discharge trends and, more generally, climate warming will likely increase the sensitivity of historically snowmelt dominated watersheds to extreme precipitation. Understanding this spatial variability in increased extreme precipitation and increased sensitivity to extreme precipitation is crucial for enhancing disaster preparedness and refining water management strategies in affected regions. 

   more » « less
5. Evaluating the Ability of FARSITE to Simulate Wildfires Influenced by Extreme, Downslope Winds in Santa Barbara, California

   https://doi.org/10.3390/fire3030029  

   Zigner, Katelyn; Carvalho, Leila M.; Peterson, Seth; Fujioka, Francis; Duine, Gert-Jan; Jones, Charles; Roberts, Dar; Moritz, Max (September 2020, Fire)

   Extreme, downslope mountain winds often generate dangerous wildfire conditions. We used the wildfire spread model Fire Area Simulator (FARSITE) to simulate two wildfires influenced by strong wind events in Santa Barbara, CA. High spatial-resolution imagery for fuel maps and hourly wind downscaled to 100 m were used as model inputs, and sensitivity tests were performed to evaluate the effects of ignition timing and location on fire spread. Additionally, burn area rasters from FARSITE simulations were compared to minimum travel time rasters from FlamMap simulations, a wildfire model similar to FARSITE that holds environmental variables constant. Utilization of two case studies during strong winds revealed that FARSITE was able to successfully reconstruct the spread rate and size of wildfires when spotting was minimal. However, in situations when spotting was an important factor in rapid downslope wildfire spread, both FARSITE and FlamMap were unable to simulate realistic fire perimeters. We show that this is due to inherent limitations in the models themselves, related to the slope-orientation relative to the simulated fire spread, and the dependence of ember launch and land locations. This finding has widespread implications, given the role of spotting in fire progression during extreme wind events. 

   more » « less