An official website of the United States government
Why do bad methods persist in some academic disciplines, even when they have been widely rejected in others? What factors allow good methodological advances to spread across disciplines? In this paper, we investigate some key features determining the success and failure of methodological spread between the sciences. We introduce a formal model that considers factors like methodological competence and reviewer bias toward one’s own methods. We show how these self-preferential biases can protect poor methodology within scientific communities, and lack of reviewer competence can contribute to failures to adopt better methods. We then use a second model to argue that input from outside disciplines can help break down barriers to methodological improvement. In doing so, we illustrate an underappreciated benefit of interdisciplinarity.
more »
« less
The case for formal methodology in scientific reform
Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological reform attempts suffer from similar mistakes and over-generalizations to the ones they aim to address. We argue that this can be attributed in part to lack of formalism and first principles. Considering the costs of allowing false claims to become canonized, we argue for formal statistical rigor and scientific nuance in methodological reform. To attain this rigor and nuance, we propose a five-step formal approach for solving methodological problems. To illustrate the use and benefits of such formalism, we present a formal statistical analysis of three popular claims in the metascientific literature: (i) that reproducibility is the cornerstone of science; (ii) that data must not be used twice in any analysis; and (iii) that exploratory projects imply poor statistical practice. We show how our formal approach can inform and shape debates about such methodological claims.
more »
« less
- Award ID(s):
- 1850849
- PAR ID:
- 10274448
- Date Published:
- Journal Name:
- Royal Society Open Science
- Volume:
- 8
- Issue:
- 3
- ISSN:
- 2054-5703
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.more » « less
-
null (Ed.)Abstract This position paper is motivated by recent educational reform efforts that urge the integration of engineering in science education. We argue that it is plausible and beneficial to integrate engineering into formal K-12 science education. We illustrate how current literature, though often implicitly, discusses this integration from a pedagogical, epistemological, or methodological argumentative stance. From a pedagogical perspective, a historically dominant argument emphasizes how engineering helps make abstract science concepts more concrete. The epistemological argument is centered on how engineering is inherently interdisciplinary and hence its integrative role in support of scientific literacy and more broadly STEM literacy is natural. From a methodological perspective, arguments focus on the engineering design process, which is compatible with scientific inquiry and adaptable to answering different types of engineering questions. We call for the necessity of spelling out these arguments and call for common language as science and engineering educators form a research-base on the integration of science and engineering. We specifically provide and discuss specific terminology associated with four different models, each effectively used to integrate engineering into school science. We caution educators against a possible direction towards a convergence approach for a specific type of integrating engineering and science. Diversity in teaching models, more accurately represents the nature of engineering but also allows adaptations based on available school resources. Future synthesis can then examine student learning outcomes associated with different teaching models.more » « less
-
Despite recent calls to make geographical analyses more reproducible, formal attempts to reproduce or replicate published work remain largely absent from the geographic literature. The reproductions of geographic research that do exist typically focus on computational reproducibility—whether results can be recreated using data and code provided by the authors—rather than on evaluating the conclusion and internal validity and evidential value of the original analysis. However, knowing if a study is computationally reproducible is insufficient if the goal of a reproduction is to identify and correct errors in our knowledge. We argue that reproductions of geographic work should focus on assessing whether the findings and claims made in existing empirical studies are well supported by the evidence presented. We aim to facilitate this transition by introducing a model framework for conducting reproduction studies, demonstrating its use, and reporting the findings of three exemplar studies. We present three model reproductions of geographical analyses of COVID‐19 based on a common, open access template. Each reproduction attempt is published as an open access repository, complete with pre‐analysis plan, data, code, and final report. We find each study to be partially reproducible, but moving past computational reproducibility, our assessments reveal conceptual and methodological concerns that raise questions about the predictive value and the magnitude of the associations presented in each study. Collectively, these reproductions and our template materials offer a practical framework others can use to reproduce and replicate empirical spatial analyses and ultimately facilitate the identification and correction of errors in the geographic literature.more » « less
-
This paper reviews trends in GeoAI research and discusses cutting-edge advances in GeoAI and its roles in accelerating environmental and social sciences. It addresses ongoing attempts to improve the predictability of GeoAI models and recent research aimed at increasing model explainability and reproducibility to ensure trustworthy geospatial findings. The paper also provides reflections on the importance of defining the science of GeoAI in terms of its fundamental principles, theories, and methods to ensure scientific rigor, social responsibility, and lasting impacts.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1098/rsos.200805
