More Like this

1. Vis Repligogy: Towards a Culture of Facilitating Replication Studies in Visualization Pedagogy and Research

   https://doi.org/10.2312/eved.20241054  

   Syeda, Uzma Haque; South, Laura; Raynor, Justin; Panavas, Liudas; Saffo, David; Morriss, Tommy; Dunne, Cody; Borkin, Michelle A (January 2024, The Eurographics Association)

   In this paper, we present the Vis Repligogy framework that enables conducting replication studies in the class. Replication studies are crucial to strengthening the data visualization field and ensuring its foundations are solid and methods accurate. Although visualization researchers acknowledge the epistemological significance of replications and their necessity to establish trust and reliability, the field has made little progress to support the publication of such studies and, importantly, provide methods to the community to encourage replications. Therefore, we contribute Vis Repligogy, a novel framework to systematically incorporate replications within visualization course curricula that not only teaches students replication and evaluation methodologies but also results in executed replication studies to validate prior work. To validate the feasibility of the framework, we present case studies of two graduate data visualization courses that implemented it. These courses resulted in a total of five replication studies. Finally, we reflect on our experience implementing the Vis Repligogy framework to provide useful recommendations for future use. We envision this framework will encourage instructors to conduct replications in their courses, help facilitate more replications in visualization pedagogy and in research, and support a culture shift towards reproducible research. Supplemental materials of this paper are available at https://osf.io/ncb6d/. 

   more » « less
2. The Design of Replication Studies

   https://doi.org/10.1111/rssa.12688  

   Hedges, Larry V.; Schauer, Jacob M. (March 2021, Journal of the Royal Statistical Society Series A: Statistics in Society)

   Abstract Empirical evaluations of replication have become increasingly common, but there has been no unified approach to doing so. Some evaluations conduct only a single replication study while others run several, usually across multiple laboratories. Designing such programs has largely contended with difficult issues about which experimental components are necessary for a set of studies to be considered replications. However, another important consideration is that replication studies be designed to support sufficiently sensitive analyses. For instance, if hypothesis tests are to be conducted about replication, studies should be designed to ensure these tests are well-powered; if not, it can be difficult to determine conclusively if replication attempts succeeded or failed. This paper describes methods for designing ensembles of replication studies to ensure that they are both adequately sensitive and cost-efficient. It describes two potential analyses of replication studies—hypothesis tests and variance component estimation—and approaches to obtaining optimal designs for them. Using these results, it assesses the statistical power, precision of point estimators and optimality of the design used by the Many Labs Project and finds that while it may have been sufficiently powered to detect some larger differences between studies, other designs would have been less costly and/or produced more precise estimates or higher-powered hypothesis tests. 

   more » « less
3. Scientific Visualization and Reproducibility for “Open” Environmental Science

   Judith Bayard Cushing, Denise Lach (December 2018, Proceedings IEEE Conference on Big Data)

   Practicing reproducible scientific research requires access to appropriate reproducibility methodology and software, as well as open data. Strict reproducibility in complex scientific domains such as environmental science, ecology and medicine, however, is difficult if not impossible. Here, we consider replication as a relaxed but bona fide substitution for strict reproducibility and propose using 3D terrain visualization for replication in environmental science studies that propose causal relationships between one or more driver variables and one or more response variables across complex ecosystem landscapes. We base our contention of the usefulness of visualization for replication on more than ten years observing environmental science modelers who use our 3D terrain visualization software to develop, calibrate, validate, and integrate predictive models. To establish the link between replication and model validation and corroboration, we consider replication as proposed by Munafò, i.e., triangulation. We enumerate features of visualization systems that would enable such triangulation and argue that such systems would render feasible domain-specific, open visualization software for use in replicating environmental science studies. 

   more » « less
4. Causal Evidence for Theories of Contagious Civil Unrest

   https://doi.org/10.1093/isq/sqae124  

   Fyfe, Rebekah; Desmarais, Bruce (October 2024, International Studies Quarterly)

   Abstract Many types of civil unrest, including protest, violent conflict, and rebellion, have been found to be subject to both inter- and intra-state contagion. These spillover effects are conventionally tested through the application of parametric structural models that are estimated using observational data. Drawing on research in methods for network analysis, we note important challenges in conducting causal inference on contagion effects in observational data. We review a recently developed non-parametric test—the “split-halves test”—that is robust to confounding and apply the test to replication data from several recent studies in which researchers tested for contagion in civil unrest. We find that about half the time findings in the published literature fail to replicate with the split-halves test. Across ten total replications, we do not see strong patterns in terms of which results do and do not replicate. We do, however, find evidence for general contagion in six of the replications, indicating that contagion is a prevalent phenomenon in civil unrest. As such, we recommend that researchers (1) use the split-halves test as a general-purpose robustness check for parametric models of contagion in the study of civil unrest, and (2) consider modeling contagion in research on civil unrest. 

   more » « less
5. Introduction: Caring for Equitable Relations in Interdisciplinary Collaborations

   https://doi.org/10.28968/cftt.v9i2.41070  

   Carrigan, Coleen; Wylie, Caitlin D (November 2023, Catalyst: Feminism, Theory, Technoscience)

   Collaborative research between scholars of science and technology studies (STS)and scholars of science, technology, engineering, and math (STEM) is a growing trend. The papers assembled in thisSpecial Section offer both embodied and empirical knowledge on how ethnographers negotiate our roles in integrative research when constrained by what our technoscientific collaborators value, what funders demand, what our home institutions expect, what we want to learn from the worlds we study, and the social transformations we envision in science and society. We grapple with how we as ethnographers can best balance caring for the communities we study, the ones we serve, and the ones we identify with. We take care that knowledge making is political. Race, gender, class, and ability status of scholars intersect with the organizational, institutional, and cultural contexts in which we practice science to shape and be shaped by entrenched power relations.Through a feminist politics of care, this collection transforms tensions in interdisciplinary collaborations into resources that enlarge our understandings of what these collaborations are like for STS ethnographers, make visible certain labors within them and, crucially, enrich our vision for what we want these collaborations to be. 

   more » « less