More Like this

1. Reproducibility and Replicability in Science

   National Academies of Sciences, Engineering (January 2019, Publications listing National Academy of Sciences National Academy of Engineering Institute of Medicine National Research Council)

   null (Ed.)

   One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science. 

   more » « less
2. Data to Infinity and Beyond: Examining Data Sharing and Reuse Practices in the Computer Security Community

   https://doi.org/10.1109/SP61157.2025.00180  

   Crowder, Anna; Lu, Allison; Childs, Kevin; Stillman, Carson; Traynor, Patrick; Butler, Kevin RB (May 2025, IEEE)

   Sharing high-quality research data specifically for reuse in future work helps the scientific community progress by enabling researchers to build upon existing work and explore new research questions without duplicating data collection efforts. Because current discussions about research artifacts in Computer Security focus on reproducibility and availability of source code, the reusability of data is unclear. We examine data sharing practices in Computer Security and Measurement to provide resources and recommendations for sharing reusable data. Our study covers five years (2019–2023) and seven conferences in Computer Security and Measurement, identifying 948 papers that create a dataset as one of their contributions. We analyze the 265 accessible datasets, evaluating their under-standability and level of reuse. Our findings reveal inconsistent practices in data sharing structure and documentation, causing some datasets to not be shared effectively. Additionally, reuse of datasets is low, especially in fields where the nature of the data does not lend itself to reuse. Based on our findings, we offer data-driven recommendations and resources for improving data sharing practices in our community. Furthermore, we encourage authors to be intentional about their data sharing goals and align their sharing strategies with those goals. 

   more » « less
3. Building a Vision for Reproducibility in the Cyberinfrastructure Ecosystem: Leveraging Community Efforts

   https://doi.org/10.14529/jsfi200106  

   Chapp, D.; Stodden, V.; Taufer, M. (March 2020, Supercomputing Frontiers and Innovations)

   The scientific computing community has long taken a leadership role in understanding and assessing the relationship of reproducibility to cyberinfrastructure, ensuring that computational results - such as those from simulations - are "reproducible", that is, the same results are obtained when one re-uses the same input data, methods, software and analysis conditions. Starting almost a decade ago, the community has regularly published and advocated for advances in this area. In this article we trace this thinking and relate it to current national efforts, including the 2019 National Academies of Science, Engineering, and Medicine report on "Reproducibility and Replication in Science". To this end, this work considers high performance computing workflows that emphasize workflows combining traditional simulations (e.g. Molecular Dynamics simulations) with in situ analytics. We leverage an analysis of such workflows to (a) contextualize the 2019 National Academies of Science, Engineering, and Medicine report's recommendations in the HPC setting and (b) envision a path forward in the tradition of community driven approaches to reproducibility and the acceleration of science and discovery. The work also articulates avenues for future research at the intersection of transparency, reproducibility, and computational infrastructure that supports scientific discovery. 

   more » « less
4. Stories from the Field (of Networking): Lessons from Deploying Research Systems in the real World

   Heimerl, Kurtis; Jang, Esther; Ndubuisi-Obi_Jr, Innocent (September 2023, ACM)

   As computer science has grown in importance throughout the world, there’s been a similarly growing drive to apply foundational computer science to address core societal issues. For networking researchers, this effort has largely focused on bringing the benefits of Internet access to marginalized populations. In this work, we reflect on more than a decade of our own work deploying such research into the world in all of rural, urban, developed, and developing regions. We share key lessons and related stories that have influenced they way we do this type of work. We conclude with a call to other researchers to add their own stories to share back to the wider networking research community. 

   more » « less
5. Reproducibility in the Classroom

   https://doi.org/10.1146/annurev-statistics-112723-034436  

   Dogucu, Mine (March 2025, Annual Review of Statistics and Its Application)

   Difficulties in reproducing results from scientific studies have lately been referred to as a reproducibility crisis. Scientific practice depends heavily on scientific training. What gets taught in the classroom is often practiced in labs, fieldwork, and data analysis. The importance of reproducibility in the classroom has gained momentum in statistics education in recent years. In this article, we review the existing literature on reproducibility education. We delve into the relationship between computing tools and reproducibility through visiting historical developments in this area. We share examples for teaching reproducibility and reproducible teaching while discussing the pedagogical opportunities created by these examples as well as challenges that the instructors should be aware of. We detail the use of teaching reproducibility and reproducible teaching practices in an introductory data science course. Lastly, we provide recommendations on reproducibility education for instructors, administrators, and other members of the scientific community. 

   more » « less