An official website of the United States government
With increasing recognition of the importance of reproducibility in computer science research, a wide range of efforts to promote reproducible research have been implemented across various sub-disciplines of computer science. These include artifact review and badging processes, and dedicated reproducibility tracks at conferences. However, these initiatives primarily engage active researchers and students already involved in research in their respective areas. In this paper, we present an argument for expanding the scope of these efforts to include a much larger audience, by introducing more reproducibility content into computer science courses. We describe various ways to integrate reproducibility content into the curriculum, drawing on our own experiences, as well as published experience reports from several sub-disciplines of computer science and computational science.
more »
« less
This content will become publicly available on April 11, 2026
What Do Machine Learning Researchers Mean by “Reproducible”?
The concern that Artificial Intelligence (AI) and Machine Learning (ML) are entering a reproducibility crisis has spurred significant research in the past few years. Yet with each paper, it is often unclear what someone means by reproducibility. Our work attempts to clarify the scope of reproducibility as displayed by the community at large. In doing so, we propose to refine the research to eight general topic areas. In this light, we see that each of these areas contains many works that do not advertise themselves as being about reproducibility, in part because they go back decades before the matter came to broader attention.
more »
« less
- PAR ID:
- 10611557
- Publisher / Repository:
- AAAI
- Date Published:
- Journal Name:
- Proceedings of the AAAI Conference on Artificial Intelligence
- Volume:
- 39
- Issue:
- 27
- ISSN:
- 2159-5399
- Page Range / eLocation ID:
- 28671 to 28683
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The number of reproduction and replication studies undertaken across the sciences continues to rise, but such studies have not yet become commonplace in geography. Existing attempts to reproduce geographic research suggest that many studies cannot be fully reproduced, or are simply missing components needed to attempt a reproduction. Despite this suggestive evidence, a systematic assessment of geographers’ perceptions of reproducibility and use of reproducible research practices remains absent from the literature, as does an identification of the factors that keep geographers from conducting reproduction studies. We address each of these needs by surveying active geographic researchers selected using probability sampling techniques from a rigorously constructed sampling frame. We identify a clear division in perceptions of reproducibility among geographic subfields. We also find varying levels of familiarity with reproducible research practices and a perceived lack of incentives to attempt and publish reproduction studies. Despite many barriers to reproducibility and divisions between subfields, we also find common foundations for examining and expanding reproducibility in the field. These include interest in publishing transparent and reproducible methods, and in reproducing other researchers’ studies for a variety of motivations including learning, assessing the internal validity of a study, or extending prior work.more » « less
-
The ability to repeat research is vital in confirming the validity of scientific discovery and is relevant to ubiquitous sensor research. Investigation of novel sensors and sensing mechanisms intersect several Federal and non-Federal agencies. Despite numerous studies on sensors at different stages of development, the absence of new field-ready or commercial sensors seems limited by reproducibility. Current research practices in sensors needs sustainable transformations. The scientific community seeks ways to incorporate reproducibility and repeatability to validate published results. A case study on the reproducibility of low-cost air quality sensors is presented. In this context, the article discusses (a) open source data management frameworks in alignment with findability, accessibility, interoperability, and reuse (FAIR) principles to facilitate sensor reproducibility; (b) suggestions for journals focused on sensors to incorporate a reproducibility editorial board and incentivization for data sharing; (c) practice of reproducibility by targeted focus issues; and (d) education of current and the next generation of diverse student and faculty community on FAIR principles. The existence of different types of sensors such as physical, chemical, biological, and magnetic (to name a few) and the fact that the sensing field spans multiple disciplines (electrical engineering, mechanical engineering, physics, chemistry, and electrochemistry) call for a generic model for reproducibility. Considering the available metrics, the authors propose eight FAIR metric standards to that transcend disciplines: citation standards, design and analysis transparency, data transparency, analytical methods transparency, research materials transparency, hardware transparency, preregistration of studies, and replication.more » « less
-
Abstract In this paper, we explore the crucial role and challenges of computational reproducibility in geosciences, drawing insights from the Climate Informatics Reproducibility Challenge (CICR) in 2023. The competition aimed at (1) identifying common hurdles to reproduce computational climate science; and (2) creating interactive reproducible publications for selected papers of the Environmental Data Science journal. Based on lessons learned from the challenge, we emphasize the significance of open research practices, mentorship, transparency guidelines, as well as the use of technologies such as executable research objects for the reproduction of geoscientific published research. We propose a supportive framework of tools and infrastructure for evaluating reproducibility in geoscientific publications, with a case study for the climate informatics community. While the recommendations focus on future CIRCs, we expect they would be beneficial for wider umbrella of reproducibility initiatives in geosciences.more » « less
-
Abstract MRI acquisition and reconstruction research has transformed into a computation-driven field. As methods become more sophisticated, compute-heavy, and data-hungry, efforts to reproduce them become more difficult. While the computational MRI research community has made great leaps toward reproducible computational science, there are few tailored guidelines or standards for users to follow. In this review article, we develop a cookbook to facilitate reproducible research for MRI acquisition and reconstruction. Like any good cookbook, we list several recipes, each providing a basic standard on how to make computational MRI research reproducible. And like cooking, we show example flavours where reproducibility may fail due to under-specification. We structure the article, so that the cookbook itself serves as an example of reproducible research by providing sequence and reconstruction definitions as well as data to reproduce the experimental results in the figures. We also propose a community-driven effort to compile an evolving list of best practices for making computational MRI research reproducible.more » « less
