skip to main content

Explore Research Products in the NSF-PAR

Title: Gender differences in peer review of innovation
Abstract Research Summary

Gender differences in peer review and the associated impact on innovation financing are well documented but less well understood. We study peer review in the National Aeronautics and Space Administration Small Business Innovation Research program, a public initiative seeking to increase women's access to innovation funds. We theorize that reviewers use status characteristics inappropriately as heuristics and create gender bias. Econometric analysis shows evidence of direct bias against female applicants, an effect linked to challenges for newcomers in demonstrating individual legitimacy rather than concerns about the organizational legitimacy of the associated firm. We also demonstrate a corrective redistribution to reverse this bias and create equity in the funding outcome. As these results negatively impact diversity in innovation, we propose policy recommendations to overcome this bias.

 Managerial Summary

Peer review is an important mechanism to rank and select technical proposals for funding. We examine the role of gender in a government program conducting this process. Controlling for the proposal quality and other factors, we show that the gender of the proposer is linked to lower scores. This effect is associated with proposals from females who are new to the program, suggesting their challenges in demonstrating credibility as leaders of these projects, and exacerbated by the fact that women represent a disproportionately high share of newcomers. Subsequently, the program reverses this bias such that the funding outcomes do not show the same inequities. This has important implications for policies supporting gender diversity in innovation.

  more » « less
Award ID(s):
1740721
NSF-PAR ID:
10375148
Author(s) / Creator(s):
 ;  ;  ;  
Publisher / Repository:
Wiley Blackwell (John Wiley & Sons)
Date Published:
Journal Name:
Strategic Entrepreneurship Journal
Volume:
16
Issue:
2
ISSN:
1932-4391
Page Range / eLocation ID:
p. 255-280
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ( , Research Integrity and Peer Review)
    Abstract Background

    In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications.

     Methods

    We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical “toy” examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges’ evaluations.

     Results

    For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions.

     Conclusions

    A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.

     
    more » « less
  2. ; ; ; ; ; ; ( , Proceedings ASEE annual conference)
    Peer review of grant proposals is critical to the National Science Foundation (NSF) funding process for STEM disciplinary and education research. Despite this, scholars receive little training in effective and constructive review of proposals beyond definitions of review criteria and an overview of strategies to avoid bias and communicate clearly. Senior researchers often find that their reviewing skills improve and develop over time, but variations in reviewer starting points can have a negative impact on the value of reviews for their intended audiences of program officers, who make funding recommendations, and principal investigators, who drive the research or want to improve their proposals. Building on the journal review component of the Engineering Education Research Peer Review Training (EER PERT) project, which is designed to develop EER scholars’ peer review skills through mentored reviewing experiences, this paper describes a program designed to provide professional development for proposal reviewing and provides initial evaluation results. 
    more » « less
  3. ; ; ; ; ; ; ( , ASEE)
    Peer review of grant proposals is critical to the National Science Foundation (NSF) funding process for STEM disciplinary and education research. Despite this, scholars receive little training in effective and constructive review of proposals beyond definitions of review criteria and an overview of strategies to avoid bias and communicate clearly. Senior researchers often find that their reviewing skills improve and develop over time, but variations in reviewer starting points can have a negative impact on the value of reviews for their intended audiences of program officers, who make funding recommendations, and principal investigators, who drive the research or want to improve their proposals. Building on the journal review component of the Engineering Education Research Peer Review Training (EER PERT) project, which is designed to develop EER scholars’ peer review skills through mentored reviewing experiences, this paper describes a program designed to provide professional development for proposal reviewing and provides initial evaluation results. 
    more » « less
  4. ; ; ( , Social Issues and Policy Review)
    Abstract

    We draw from ecological systems and social psychological theories to elucidate macrosystem‐ and microsystem‐level variables that promote and maintain gender inequities in science, technology, engineering, and math (STEM). Because gender‐STEM stereotypes undermine girls’ (and women's), but boosts boys’ (and men's), STEM interest and success, we review how they operate in STEM learning environments to differentially socialize girls and boys and undermine gender integroup relations. We propose seven practice recommendations to improve STEM K‐12 education: (1) design relational classrooms, (2) teach the history of gender inequality and bias, (3) foster collaborative and cooperative classrooms, (4) promote active learning and growth mindset strategies, (5) reframing STEM as inclusive, (6) create near‐peer mentorship programs, and (7) re‐imagine evaluation metrics. To support these practice recommendations, three policy recommendations are posited: (1) increase teacher autonomy, training, and representation, (2) re‐evaluate standardized testing, and (3) reallocate and increase government funding for public schools.

     
    more » « less
  5. ; ; ( , Science Education)
    While there is evidence to support the existence of identity-based disparities, inequities, and biases in the academic journal peer-review process, little research supports the presence of this bias in the peer-review process for academic journals in science education. Through an analysis of six leading journals in science education, we aimed to investigate the extent to which diversity, equity, and inclusion (DEI), as well as the presence of bias in the peer-review process, are addressed by these journals. We analyzed trends in the gender/sex, geographical affiliation, race/ethnicity, and the presence of equity-centered research focus for members of these journals' editors and editorial boards. We found that although gender/sex is well-balanced in these journals' editors and editorial boards, they are typically North American centric, and White individuals are overwhelmingly represented. Four journals had a quarter or more of individuals who pursue equity-centered research. Only two journals provided detailed information on how manuscripts are reviewed in their author submission guidelines. All used a double-blind approach to peer-review. One of the journals includes an explicit position on DEI. Based on the analyses and reflections on our own experiences, we recommend science education journals consider ways to probe whether bias does exist in their peer-review process, diversify their board to be more inclusive of scholars from communities historically marginalized, and move to a triple-blind approach to their peer-review process as mechanisms to mitigate bias in the journal peer review. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email