skip to main content


Title: Do Rubrics Live up to Their Promise? Examining How Rubrics Mitigate Bias in Faculty Hiring
Many colleges and universities now require faculty search committees to use rubrics when evaluating faculty job candidates, as proponents believe these “decision-support tools” can reduce the impact of bias in candidate evaluation. That is, rubrics are intended to ensure that candidates are evaluated more fairly, which is then thought to contribute to the enhanced hiring of candidates from minoritized groups. However, there is scant — and even contradictory — evidence to support this claim. This study used a multiple case study methodology to explore how five faculty search committees used rubrics in candidate evaluation, and the extent to which using a rubric seemed to perpetuate or mitigate bias in committee decision-making. Results showed that the use of rubrics can improve searches by clarifying criteria, encouraging criteria use in evaluation, calibrating the application of criteria to evidence, and in some cases, bringing diversity, equity, and inclusion work (DEI) into consideration. However, search committees also created and implemented rubrics in ways that seem to perpetuate bias, undermine effectiveness, and potentially contribute to the hiring of fewer minoritized candidates. We conclude by providing stakeholders with practical recommendations on using rubrics and actualizing DEI in faculty hiring.  more » « less
Award ID(s):
1820975 1820974
NSF-PAR ID:
10430722
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
The Journal of Higher Education
ISSN:
0022-1546
Page Range / eLocation ID:
1 to 28
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Research has documented the presence of bias against women in hiring, including in academic science, technology, engineering, and mathematics (STEM). Hiring rubrics (also called criterion checklists, decision support tools, and evaluation tools) are widely recommended as a precise, cost-effective remedy to counteract hiring bias, despite a paucity of evidence that they actually work (see table S8). Our in-depth case study of rubric usage in faculty hiring in an academic engineering department in a very research-active university found that the rate of hiring women increased after the department deployed rubrics and used them to guide holistic discussions. Yet we also found evidence of substantial gender bias persisting in some rubric scoring categories and evaluators’ written comments. We do not recommend abandoning rubrics. Instead, we recommend a strategic and sociologically astute use of rubrics as a department self-study tool within the context of a holistic evaluation of semifinalist candidates. 
    more » « less
  2. Efforts to mitigate bias in faculty hiring processes are well-documented in the literature. Yet, significant barriers to the hiring of racially minoritized and White women in many STEM fields remain. An underreported barrier to inclusive hiring is assessment of risk. Guided by theory from behavioral economics, social psychology, and decision-making, we examine the inner workings of five faculty search committees to understand how committee members identified and assessed risk with particular attention to assessments of risk that became intermingled with social biases. Committees identified and assessed five risks, including candidate interest, candidate disciplinary expertise, candidate competence, candidate collegiality, and the timing and oversight of the search process itself. We discuss implications of risk identification and assessment for effective and inclusive searches. 
    more » « less
  3. This research paper describes a study designed to help inform STEM faculty hiring practices at institutions of higher education in the U.S., where over the past two decades, diversity statements have become more popular components of application packages for faculty jobs. The purpose is to explore the ways and extent to which diversity statements are utilized in evaluating faculty applicants. The research questions are: (1) To what extent do universities equip search committees to evaluate applicants’ diversity statements? (2) What are STEM faculty’s perspectives of diversity statements in job applications? This paper is derived from a larger two-phase sequential mixed methods study examining the factors current faculty members and administrators consider important when hiring new STEM faculty. During the first phase, we deployed a nationwide survey to STEM faculty members and administrators who have been involved in faculty searches, with 151 of 216 respondents answering questions specific to diversity statements. About 29% of survey respondents indicated their departments required diversity statements; 59% indicated their institutions did not provide guidance for evaluating them. The second phase was a phenomenological study involving interviews of 25 survey respondents. Preliminary analyses of interview data indicated that a little more than half (52%) of participants’ departments required a diversity statement. Of the departments that required diversity statements, a little more than half used a rubric for evaluation, whether as part of a larger holistic rubric, or as a standalone rubric. For some departments that did not require diversity statements, applicants were required to discuss diversity within their other application materials. Regarding faculty members’ perceptions of diversity statements, some felt that diversity statements were necessary to assess candidates’ beliefs and experiences. Some noted that when diversity is discussed as part of another document and is not required as a stand-alone statement, it feels like the candidate “slaps on a paragraph” about diversity. Others viewed diversity statements as a “bump” that gives candidates “bonus points.” A few faculty felt that diversity statements were “redundant,” and if applicants were passionate about diversity, they would organically discuss it in the other required documents. Many shared frustrations regarding the requirement and evaluation practices. Most participants indicated their postings provided applicants with little to no guidance on what search committees were looking for in submitted diversity statements; they felt it would be beneficial for both the search committee and the applicants to have this guidance. Shared through a traditional lecture, results from this study may be used to help inform strategies for recruiting faculty who are committed to diversity - and ideally, equity and inclusion - and for addressing equity in faculty hiring. 
    more » « less
  4. Abstract Background

    The lack of racial diversity in science, technology, engineering, and mathematics (STEM) disciplines is perhaps one of the most challenging issues in the United States higher education system. The issue is not only concerning diverse students, but also diverse faculty members. One important contributing factor is the faculty hiring process. To make progress toward equity in hiring decisions, it is necessary to better understand how applicants are considered and evaluated. In this paper, we describe and present our study based on a survey of current STEM faculty members and administrators who examined applicant qualifications and characteristics in STEM faculty hiring decisions.

    Results

    There are three key findings of the present research. First, we found that faculty members placed different levels of importance on characteristics and qualifications for tenure track hiring and non-tenure track hiring. For example, items related to research were more important when evaluating tenure track applicants, whereas items related to teaching and diversity were more important when evaluating non-tenure track applicants. Second, faculty members’ institutional classification, position, and personal identities (e.g., gender, race/ethnicity) had an impact on their evaluation criteria. For instance, we found men considered some diversity-related items more important than women. Third, faculty members rated the importance of qualifications with diversity, equity, and inclusion (DEI)-related constructs significantly lower than qualifications that did not specify DEI-related constructs, and this trend held for both tenure track and non-tenure track faculty hiring.

    Conclusions

    This study was an attempt to address the issue of diversity in STEM faculty hiring at institutions of higher education by examining how applicant characteristics are considered and evaluated in faculty hiring practices. Emphasizing research reputation and postdoctoral reputation while neglecting institutional diversity and equitable and inclusive teaching, research, and service stunt progress toward racial diversity because biases—both implicit and explicit, both positive and negative—still exist. Our results were consistent with research on bias in recruitment, revealing that affinity bias, confirmation bias, and halo bias exist in the faculty hiring process. These biases contribute to inequities in hiring, and need to be addressed before we can reach, sustain, and grow desired levels of diversity.

     
    more » « less
  5. null (Ed.)
    Informal learning institutions, such as museums, science centers, and community-based organizations, play a critical role in providing opportunities for students to engage in science, technology, engineering, and mathematics (STEM) activities during out-of-school time hours. In recent years, thousands of studies, evaluations, and conference proceedings have been published measuring the impact that these programs have had on their participants. However, because studies of informal science education (ISE) programs vary considerably in how they are designed and in the quality of their designs, it is often quite difficult to assess their impact on participants. Knowing whether the outcomes reported by these studies are supported with sufficient evidence is important not only for maximizing participant impact, but also because there are considerable economic and human resources invested to support informal learning initiatives. To address this problem, I used the theories of impact analysis and triangulation as a framework for developing user-friendly rubrics for assessing quality of research designs and evidence of impact. I used two main sources, research-based recommendations from STEM governing bodies and feedback from a focus group, to identify criteria indicative of high-quality STEM research and study design. Accordingly, I developed three STEM Research Design Rubrics, one for quantitative studies, one for qualitative studies, and another for mixed methods studies, that can be used by ISE researchers, practitioners, and evaluators to assess research design quality. Likewise, I developed three STEM Impact Rubrics, one for quantitative studies, one for qualitative studies, and another for mixed methods studies, that can be used by ISE researchers, practitioners, and evaluators to assess evidence of outcomes. The rubrics developed in this study are practical tools that can be used by ISE researchers, practitioners, and evaluators to improve the field of informal science learning by increasing the quality of study design and for discerning whether studies or program evaluations are providing sufficient evidence of impact. 
    more » « less