Why are some research studies easy to reproduce while others are difficult? Casting doubt on the accuracy of scientific work is not fruitful, especially when an individual researcher cannot reproduce the claims made in the paper. There could be many subjective reasons behind the inability to reproduce a scientific paper. The field of Machine Learning (ML) faces a reproducibility crisis, and surveying a portion of published articles has resulted in a group realization that although sharing code repositories would be appreciable, code bases are not the end all be all for determining the reproducibility of an article. Various parties involved in the publication process have come forward to address the reproducibility crisis and solutions such as badging articles as reproducible, reproducibility checklists at conferences (NeurIPS, ICML, ICLR, etc.), and sharing artifacts on OpenReview come across as promising solutions to the core problem. The breadth of literature on reproducibility focuses on measures required to avoid ir-reproducibility, and there is not much research into the effort behind reproducing these articles. In this paper, we investigate the factors that contribute to the easiness and difficulty of reproducing previously published studies and report on the foundational framework to quantify effort of reproducibility.
more »
« less
When are hypotheses useful in ecology and evolution?
Research hypotheses have been a cornerstone of science since before Galileo. Many have argued that hypotheses (1) encourage discovery of mechanisms, and (2) reduce bias—both features that should increase transferability and reproducibility. However, we are entering a new era of big data and highly predictive models where some argue the hypothesis is outmoded. We hypothesized that hypothesis use has declined in ecology and evolution since the 1990s, given the substantial advancement of tools further facilitating descriptive, correlative research. Alternatively, hypothesis use may have become
- PAR ID:
- 10387159
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Ecology and Evolution
- Volume:
- 11
- Issue:
- 11
- ISSN:
- 2045-7758
- Page Range / eLocation ID:
- p. 5762-5776
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract The function of intergroup encounters (IGEs) may differ substantially among species of different group sizes and social organizations. Research in group‐living primates has shown that the behavioral responses during IGEs can vary widely from affiliative to neutral or aggressive interactions; still, little is known about IGEs in pair‐living taxa. We conducted a systematic literature review to find relevant studies on the functions of IGEs in pair‐living nonhuman primates that could inform analyses of IGE data (
n = 242 IGEs, 21 groups and 10 solitary individuals, 1997−2020) from wild owl monkeys, a pair‐living, monogamous primate with extensive biparental care. We identified 1315 studies published between 1965 and 2021; only 13 of them (n = 10 species) contained raw data on the number of IGEs. Our review of those studies showed that IGEs are common, but highly variable in their nature and characteristics in pair‐living primates. To examine the non‐mutually exclusive hypotheses of resource‐, and mate defense, and infanticide avoidance we analyzed data from the Owl Monkey Project 27‐year long database to build first an a priori model set. To incorporate prior knowledge from the literature review, we conducted our analyses as a consecutive series of binomial logistic regressions. All IGEs including all biologically relevant parameters (N = 156) were codified into three different behavioral categories (Reaction, Agonism, and Physical Aggression). The analysis showed that owl monkeys regularly engaged in IGEs, most of which were agonistic. They showed more reaction when infants were present, but reactions were less physically aggressive when infants and pregnant females were involved. Overall, our results lend more support for the infant and mate defense hypotheses than they do for the resource defense one. -
null (Ed.)One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.more » « less
-
null (Ed.)Abstract This article provides a systematic review of the network formation literature in the public sector. In particular, we code and categorize the theoretical mechanisms used in empirical network research to motivate collaboration and tie formation. Based on a review of the 107 articles on network formation found in 40 journals of public administration and policy from 1998 to 2019, we identify 15 distinct theoretical categories. For each category, we describe the theory, highlight its use in the literature, and identify limitations and concerns with current applications. Overall, we find that most studies rely on a similar set of general theories of network formation. More importantly, we find that most theoretical mechanisms are not well specified, and empirical tests are often unable to directly assess the specific underlying mechanism. The results of our review highlight the need for our field to embrace experimental designs, develop panel network datasets, and engage in more network-level research.more » « less
-
more » « less
Abstract Human gene research generates new biology insights with translational potential, yet few studies have considered the health of the human gene literature. The accessibility of human genes for targeted research, combined with unreasonable publication pressures and recent developments in scholarly publishing, may have created a market for low-quality or fraudulent human gene research articles, including articles produced by contract cheating organizations known as paper mills. This review summarises the evidence that paper mills contribute to the human gene research literature at scale and outlines why targeted gene research may be particularly vulnerable to systematic research fraud. To raise awareness of targeted gene research from paper mills, we highlight features of problematic manuscripts and publications that can be detected by gene researchers and/or journal staff. As improved awareness and detection could drive the further evolution of paper mill-supported publications, we also propose changes to academic publishing to more effectively deter and correct problematic publications at scale. In summary, the threat of paper mill-supported gene research highlights the need for all researchers to approach the literature with a more critical mindset, and demand publications that are underpinned by plausible research justifications, rigorous experiments and fully transparent reporting.
