skip to main content
Explore Scholarly Publications and Datasets in the NSF-PAR

Title: Peering into the Black Box of Peer Review: From the Perspective of Editors
ABSTRACT CONTEXT The peer review process plays a critical role in ensuring the quality of work published within a field and advancing the knowledge within the research community. However, for many members of the community, the process of peer review largely remains a black box to many scholars, especially those with less experience within the community. Therefore, there is a need to illuminate the peer review process for the research community. PURPOSE OR GOAL To more transparently reveal the contents of the black box around the peer review process, we interviewed editors (associate and deputy editors) for the Journal of Engineering Education (JEE) to provide editor perspectives on the overall peer review process. The goal of this paper is to clearly articulate the behind-the-scenes processes of peer review as well as the expectations and perceptions of the editors with respect to publishing within JEE. By bringing these processes to light, we hope that more members of the field will be aware of the overall process and the associated expectations for contributing to the field. APPROACH OR METHODOLOGY/METHODS To meet the goals of this study, we conducted semi-structured interviews with six editors of JEE who worked in the field of engineering education research (EER), as a part more » of a larger project exploring the boundaries of the field as expressed within the peer reviews process. The interviewer from the research team followed a protocol but also asked additional questions to elicit more details in some cases. The interviews were recorded, transcribed, and thematically coded using an open-coding process. ACTUAL OR ANTICIPATED OUTCOMES Based on the analysis of the editor interviews, we present three critical aspects of the peer review process: the types of editors, the process that editors typically conduct to identify reviewers, and the types of decisions through the process. Additionally, we highlight considerations and advice from the editors to help members of the EER community develop. CONCLUSIONS/RECOMMENDATIONS/SUMMARY The current study makes the editors’ perspectives and decision-making processes more explicit to readers. These decision-making processes are full of careful considerations and also challenges. By doing so, we hope to help the members of the EER community gain a better understanding of what is going on backstage of the peer review process. « less
Authors:
; ;
Award ID(s):
1762436
Publication Date:
NSF-PAR ID:
10378440
Journal Name:
Research in Engineering Education Symposium & Australasian Association for Engineering Education Conference
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ( , 2022 Annual American Society of Engineering Education (ASEE) Conference and Exposition)
    This paper describes the Engineering Education Research (EER) Peer Review Training (PERT) project, which is designed to develop EER scholars’ peer review skills through mentored reviewing experiences. Supported by the National Science Foundation, the overall programmatic goals of the PERT project are to establish and evaluate a mentored reviewer program for 1) EER journal manuscripts and 2) EER grant proposals. Concurrently, the project seeks to explore how EER scholars develop schema for evaluating EER scholarship, whether these schema are shared in the community, and how schema influence recommendations made to journal editors during the peer review process. To accomplish these goals, the PERT project leveraged the previously established Journal of Engineering Education (JEE) Mentored Reviewer Program, where two researchers with little reviewing experience are paired with an experienced mentor to complete three manuscript reviews collaboratively. In this paper we report on focus group and exit survey findings from the JEE Mentored Reviewer Program and discuss revisions to the program in response to those findings.
  2. ; ; ; ; ; ( , ASEE Annual Conference proceedings)
    This paper describes the Engineering Education Research (EER) Peer Review Training (PERT) project, which is designed to develop EER scholars’ peer review skills through mentored reviewing experiences. Supported by the National Science Foundation, the overall programmatic goals of the PERT project are to establish and evaluate a mentored reviewer program for 1) EER journal manuscripts and 2) EER grant proposals. Concurrently, the project seeks to explore how EER scholars develop schema for evaluating EER scholarship, whether these schema are shared in the community, and how schema influence recommendations made to journal editors during the peer review process. To accomplish these goals, the PERT project leveraged the previously established Journal of Engineering Education (JEE) Mentored Reviewer Program, where two researchers with little reviewing experience are paired with an experienced mentor to complete three manuscript reviews collaboratively. In this paper we report on focus group and exit survey findings from the JEE Mentored Reviewer Program and discuss revisions to the program in response to those findings.
  3. ; ; ( , Hawaii International Conference on Education)
    Engineering Education Research (EER) is an emerging interdisciplinary field (Beddoes, 2014a; Jesiek, Newswander, & Borrego, 2009). Having emerged less than twenty years ago, the field’s boundaries and normativities are still shifting and being formed. Furthermore, EER is inherently interdisciplinary, drawing on theories and methods from other fields, including education, psychology, and anthropology, among others (Beddoes, 2014b). These characteristics - the age and interdisciplinary nature of the field - make EER a particularly interesting site for examining a discipline in the making. One process through which the field’s boundaries and normativities are being formed is peer review (Beddoes, 2011). Therefore, the overarching goal of this project is to identify the kinds of scholarship that are readily accepted into the field and the kinds that are not. Examining this boundary work can produce new insights into the social construction of knowledge in EER, as well as in other interdisciplinary fields. As a first step toward the overarching goal, this paper presents preliminary findings that address the question: What differences exist in the experiences, perceptions, and understandings of those who have submitted articles to the Journal of Engineering Education (JEE) within the past 5 years?
  4. ; ( , Science Education)
    Across a broad range of disciplines, research has found that inequity is systemic in the journal review process. Collectively, however, this study does not specifically examine racial inequity. Moreover, literature on the peer review process in science education, in particular, does not foreground equity as a subject of study. The present study aims to address this void by examining racial equity in the peer review process with a specific focus on journals in science education. Data are collected from lead editors of major science education journals through the form of interviews, focus groups, and critical arts-based methods. The two research questions driving data collection are (a) In what ways does the science education journal peer review process promote racial equity? and (b) How are science education journal editors’ perceptions of racial inequity reflected in the peer review process? McNair and colleagues’ racial equity framework informs the explorations of journal review in science education from the lead editors’ perspectives. From our findings, we offer four suggestions for moving toward greater racial equity in the science education peer review process.
  5. ; ; ; ; ; ( , 2020 ASEE Virtual Annual Conference)
    As the field continues to grow, engineering education is continually challenged with finding engineering education research (EER) positions that align with the broad abilities and interests of its members. EER positions exist in engineering education departments, traditional engineering departments (e.g., mechanical, civil), and in non-degree granting programs (e.g., centers for teaching and learning, engineering programs). These positions vary across their emphasis on research, teaching, and service and provide access to different resources and mechanisms to impact engineering education. Given the range of positions available in EER and the emergence of new EER programs, it can be challenging for graduate students and postdocs to navigate the job search process and identify a position that aligns with their professional goals. The purpose of this research was to better understand the EER job market as it relates to what applicants (i.e., graduates and post-docs) experience as they navigate the job-search and decision-making process. For this study, we conducted interviews with seven transitioning first-year EER faculty members. These individuals were transitioning into various EER faculty positions (e.g. Lecturer, Teaching Fellow, Assistant Professor, Research Assistant Professor) with different backgrounds in EER based on their graduate training experiences which included established EER programs as well asmore »traditional engineering departments with EER advisor(s). We asked questions that focused on the individual’s new faculty position, their perception of the weekly time requirements, their job search process, and factors that influenced their final decision of which job to select. Each interview was conducted by two graduate students and was then transcribed and verified for accuracy. Three faculty members performed holistic coding of the transcripts focused on three areas: EER position types, job search process, and job decision making process. The Qualifying Qualitative research Quality framework (Q3) was used as a guide throughout our data collection and analysis process to ensure reliability and trustworthiness of the data collected. Through our analysis process, we developed a visual representation that provides a guide to assist EER graduate students and postdocs with their job search process. The first figure captures the diversity of positions along with the types of institutions where these positions exist to provide a starting point for individuals on their job search process. The second figure includes a timeline to help capture the average time frames for different phases of the job search process. Factors associated with final decisions based on the interviews conducted are also outlined to provide areas of consideration for individuals undergoing this process in the future. This work provides insight to aspiring academics about the range of opportunities available to those with a background in EER and how they can pursue finding alignment between their interests and positions that are available.« less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email