More Like this

1. Reproducibility in Scientific Computing

   https://doi.org/10.1145/3186266  

   Ivie, Peter; Thain, Douglas (July 2018, ACM Computing Surveys)
2. Containers for computational reproducibility

   https://doi.org/10.1038/s43586-023-00236-9  

   Moreau, David; Wiebels, Kristina; Boettiger, Carl (December 2023, Nature Reviews Methods Primers)
3. Reproducibility in the Classroom

   https://doi.org/10.1146/annurev-statistics-112723-034436  

   Dogucu, Mine (March 2025, Annual Review of Statistics and Its Application)

   Difficulties in reproducing results from scientific studies have lately been referred to as a reproducibility crisis. Scientific practice depends heavily on scientific training. What gets taught in the classroom is often practiced in labs, fieldwork, and data analysis. The importance of reproducibility in the classroom has gained momentum in statistics education in recent years. In this article, we review the existing literature on reproducibility education. We delve into the relationship between computing tools and reproducibility through visiting historical developments in this area. We share examples for teaching reproducibility and reproducible teaching while discussing the pedagogical opportunities created by these examples as well as challenges that the instructors should be aware of. We detail the use of teaching reproducibility and reproducible teaching practices in an introductory data science course. Lastly, we provide recommendations on reproducibility education for instructors, administrators, and other members of the scientific community. 

   more » « less
4. Summer of Reproducibility: Building Global Capacity for Practical Reproducibility through Hands-On Mentorship

   Lieggi, Stephanie; Fund, Fraida; Keahey, Kate; Richardson, Marc (July 2025, ACM)

   Thanks to increasing awareness of the importance of reproducibility in computer science research, initiatives such as artifact review and badging have been introduced to encourage reproducible research in this field. However, making "practical reproducibility" truly widespread requires more than just incentives. It demands an increase in capacity for reproducible research among computer scientists - more tools, workflows, and exemplar artifacts, and more human resources trained in best practices for reproducibility. In this paper, we describe our experiences in the first two years of the Summer of Reproducibility (SoR), a mentoring program that seeks to build global capacity by enabling students around the world to work with expert mentors while producing reproducibility artifacts, tools, and education materials. We give an overview of the program, report preliminary outcomes, and discuss plans to evolve this program. 

   more » « less
5. Three Pillars of Practical Reproducibility

   Keahey, K; Anderson, J; Powers, M; Cooper, A (October 2023, ReWorDS 23 workshop at eScience'23)

   Practical reproducibility is the ability to reproduce results is a manner that is cost-effective enough to become a vehicle of mainstream scientific exploration. Since computational research artifacts usually require some form of computing to interpret, open and programmable infrastructure, such as a range of NSF-supported testbeds spanning infrastructure from datacen- ter through networks to wireless systems, is a necessary – but not sufficient – requirement for reproducibility. The question arises what other services and tools should build on the availability of such programmable infrastructure to foster the development and sharing of findable, accessible, integrated, and reusable (FAIR) experiments that underpin practical reproducibility. In this paper, we propose three such services addressing the problems of packaging for reuse, findability, and accessibility, respectively. We describe how we developed these services in Chameleon, an NSF-funded testbed for computer science research which has supported the research of a community of 8,000+ users, and discuss their strengths and limitations. 

   more » « less