An official website of the United States government
Peer review is an integral component of contemporary science. While peer review focuses attention on promising and interesting science, it also encourages scientists to pursue some questions at the expense of others. Here, we use ideas from forecasting assessment to examine how two modes of peer review—ex ante review of proposals for future work and ex post review of completed science—motivate scientists to favor some questions instead of others. Our main result is that ex ante and ex post peer review push investigators toward distinct sets of scientific questions. This tension arises because ex post review allows investigators to leverage their own scientific beliefs to generate results that others will find surprising, whereas ex ante review does not. Moreover, ex ante review will favor different research questions depending on whether reviewers rank proposals in anticipation of changes to their own personal beliefs or to the beliefs of their peers. The tension between ex ante and ex post review puts investigators in a bind because most researchers need to find projects that will survive both. By unpacking the tension between these two modes of review, we can understand how they shape the landscape of science and how changes to peer review might shift scientific activity in unforeseen directions.
more »
« less
Event Based Approach for Modeling Indoor Airflow Patterns
This paper is under review, the authors are currently working on the requested revisions by the review team
more »
« less
- Award ID(s):
- 1832971
- PAR ID:
- 10312646
- Date Published:
- Journal Name:
- Journal of building engineering
- ISSN:
- 2352-7102
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Although code review is an essential step for ensuring the quality of software, it is surprising that current code review systems do not have mechanisms to protect the integrity of the code review process. We uncover multiple attacks against the code review infrastructure which are easy to execute, stealthy in nature, and can have a significant impact, such as allowing malicious or buggy code to be merged and propagated to future releases. To improve this status quo, in this work we lay the foundations for securing the code review process. Towards this end, we first identify a set of key design principles necessary to secure the code review process. We then use these principles to propose SecureReview, a security mechanism that can be applied on top of a Git-based code review system to ensure the integrity of the code review process and provide verifiable guarantees that the code review process followed the intended review policy. We implement SecureReview as a Chrome browser extension for GitHub and Gerrit. Our security analysis shows that SecureReview is effective in mitigating the aforementioned attacks. An experimental evaluation shows that the SecureReview implementation only adds a slight storage overhead (i.e., less than 0.0006 of the repository size).more » « less
-
East, Martin; Slomp, David (Ed.)Studies examining peer review demonstrate that students can learn from giving feedback to and receiving feedback from their peers, especially when they utilize information gained from the review process to revise. However, much of the research on peer review is situated within the literature regarding how students learn to write. With an increasing use of writing-to-learn in STEM classrooms, it is important to study how students engage in peer review for these types of writing assignments. This study sought to better understand how peer review and revision can support student learning for writing-to-learn specifically, using the lenses of cognitive perspectives of writing and engagement with written corrective feedback. Using a case study approach, we provide a detailed analysis of six students’ written artifacts in response to a writing-to-learn assignment that incorporated peer review and revision implemented in an organic chemistry course. Students demonstrated a range in the types of revisions they made and the extent to which the peer review process informed their revisions. Additionally, students exhibited surface, midlevel, and active engagement with the peer review and revision process. Considering the different engagement levels can inform how we frame peer review to students when using it as an instructional practice.more » « less
-
In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper qualities from an author with multiple submissions to help improve the conference's decisions. However, the isotonic mechanism relies on the assumption that the author's utility is both an increasing and a convex function with respect to the review score, which is often violated in realistic settings (e.g.~when authors aim to maximize the number of accepted papers). In this paper, we propose a sequential review mechanism that can truthfully elicit the ranking information from authors while only assuming the agent's utility is increasing with respect to the true quality of her accepted papers. The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers. Advantages of the sequential review mechanism include: 1) eliciting truthful ranking information in a more realistic setting than prior work; 2) reducing the reviewing workload and increasing the average quality of papers being reviewed; 3) incentivizing authors to write fewer papers of higher quality.more » « less
-
Across a broad range of disciplines, research has found that inequity is systemic in the journal review process. Collectively, however, this study does not specifically examine racial inequity. Moreover, literature on the peer review process in science education, in particular, does not foreground equity as a subject of study. The present study aims to address this void by examining racial equity in the peer review process with a specific focus on journals in science education. Data are collected from lead editors of major science education journals through the form of interviews, focus groups, and critical arts-based methods. The two research questions driving data collection are (a) In what ways does the science education journal peer review process promote racial equity? and (b) How are science education journal editors’ perceptions of racial inequity reflected in the peer review process? McNair and colleagues’ racial equity framework informs the explorations of journal review in science education from the lead editors’ perspectives. From our findings, we offer four suggestions for moving toward greater racial equity in the science education peer review process.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
