skip to main content


This content will become publicly available on July 20, 2024

Title: Evaluating a Learned Admission-Prediction Model as a Replacement for Standardized Tests in College Admissions
A growing number of college applications has presented an annual challenge for college admissions in the United States. Admission offices have historically relied on standardized test scores to organize large applicant pools into viable subsets for review. However, this approach may be subject to bias in test scores and selection bias in test-taking with recent trends toward test-optional admission. We explore a machine learning-based approach to replace the role of standardized tests in subset generation while taking into account a wide range of factors extracted from student applications to support a more holistic review. We evaluate the approach on data from an undergraduate admission office at a selective US institution (13,248 applications). We find that a prediction model trained on past admission data outperforms an SAT-based heuristic and matches the demographic composition of the last admitted class. We discuss the risks and opportunities for how such a learned model could be leveraged to support human decision-making in college admissions.  more » « less
Award ID(s):
2008139
NSF-PAR ID:
10466326
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
ACM Conference on Learning at Scale (L@S)
Page Range / eLocation ID:
195 to 203
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. This paper is a work-in-progress, focused on the utilization of the Rising Scholars Program to introduce minority students to experiential engineering projects within Agricultural and Biological Engineering. Traditional admissions processes at top institutions predominately utilize standardized test scores when comparing student applications. The equity of these high-stakes tests most severely affects students of low socioeconomic status (SES). The NSF-sponsored program, Rising Scholars: Web of Support used as an Indicator of Success in Engineering, was created to investigate whether alternative admission criteria could be used to identify low-SES applicants who would excel within STEM fields in higher education, even if they did not have the superior standardized testing metrics preferred by current admissions processes. The students underwent a pre-selection process to determine their eligibility. The overall experience was designed to enhance student connectivity within the collegiate environment. The Gallup-Purdue Index (2014) found that feeling supported and having learning experiences that illustrated learned principles produced a graduate who would be engaged in their work. The Rising Scholar (RS) program utilized a prescribed path through college designed to enhance these features. These positive experiences are exemplified by the Purdue Agricultural and Biological Engineering (ABE) department and how they approach the overall educational process. Faculty are motivated in their teaching, research, and extension efforts by a focus on meeting the world’s grand challenges, in which most college students are also highly interested. The Rising Scholars Program utilized the Vertically Integrated Projects model to introduce their students to real-life projects at the freshman and sophomore level, which could potentially be continued on into graduate school. Several of the RS students have worked with the Purdue ABE Hog Cooling Pad Project and these students have conducted research, prototyping, and design modifications on the pad. They have participated in five experimental bench tests of the design and four consecutive live animal studies related to the pad performance. Within these experiments, Rising Scholars students were able to work on real-life projects, with real-world impact. The preliminary hypothesis question is: Are future graduates of the Rising Scholars Program more likely to thrive in all areas of well-being due to their collegiate experiences? 
    more » « less
  2. Traditional admissions processes at top institutions predominately utilize standardized test scores when comparing student applications. The equity of these high-stakes tests most severely affects students of low socioeconomic status (SES). The NSF-sponsored program, Rising Scholars: Web of Support used as an Indicator of Success in Engineering, was created to investigate whether alternative admission criteria could be used to identify low-SES applicants who would excel within STEM fields in higher education, even if they did not have the superior standardized testing metrics preferred by current admissions process. The quality of the student’s support networks and their readiness for higher education as determined by an in-person interview with the selection committee were used as input data for a Web of Support characterization model to predict a student’s likely collegiate success at the matriculation point. There were three cohorts with a total of 21 students chosen for the program during their entry to the university which included applicants of low-SES and under-represented minority status. A significant programmatic element for these students was their involvement in experiential activities through pre-existing programs in the institution. It was reasonably assumed that the Rising Scholars student population could be positively influenced toward long-term educational commitment through experiential activities providing realistic views of professional activity. The prescribed collegiate path for these students contained an experiential educational element for each summer between admission and graduation. A summer research project with a faculty-directed laboratory before the sophomore year and a self-directed research project prior to the junior year were used to build project management experience, along with a paid, external internship in a professional organization likely to hire within the student’s major. Based upon the limited data collected so far, the researchers seem to have been conclusively demonstrated that a structured, ‘high-touch’ program with a heavy experiential component can successfully move low-SES students with STEM inclinations through a highly ranked institution. Counselling to reduce the anxiety surrounding the collegiate process for first generation students and some form of scholarship support to reduce the financial burden are both crucial underlying elements to this program’s success, but the importance of hands-on, experiential activities that help the student visualize their professional career cannot be under-estimated. 
    more » « less
  3. While there is movement to create more equitable and holistic admission review processes, faculty continue to place strong emphasis on a single piece of information when making admissions decisions: standardized test scores. This study used an experimental design to test whether instructions provided to faculty prior to assessing doctoral applicants could support holistic review by reducing the weight of the general record examination (GRE) in faculty appraisals of competence and merit for graduate study. Tenured and/or tenure-track faculty ( N =271) were randomly assigned to one of three instructional conditions: Control (no instruction), “Diamond in the Rough,” and “Weed Out.” In addition, faculty participants were randomly assigned to read one of two vignettes of a prospective first-generation student who either received high or average GRE scores. Faculty then rated the applicant’s competence using a three-item survey. As expected, faculty who read the vignette describing the candidate with the high GRE rated him as more competent than faculty who read the average GRE vignette. In addition, being instructed to seek out diamonds in the rough buffered the effect of the GRE score on competence. Faculty were also asked to indicate whether they would need additional information to make an admissions decision. They were more likely to ask about grades and research skills than about psychosocial factors that might contextualize the candidate’s performance and perceived competence. The results of this study have implications for creating more equitable doctoral admissions processes that center equity, diversity, and inclusion in decision making. 
    more » « less
  4. We study a game theoretic model of standardized testing for college admissions. Students are of two types; High and Low. There is a college that would like to admit the High type students. Students take a potentially costly standardized exam which provides a noisy signal of their type. The students come from two populations, which are identical in talent (i.e. the type distribution is the same), but differ in their access to resources: the higher resourced population can at their option take the exam multiple times, whereas the lower resourced population can only take the exam once. We study two models of score reporting, which capture existing policies used by colleges. The first policy (sometimes known as "super-scoring") allows students to report the max of the scores they achieve. The other policy requires that all scores be reported. We find in our model that requiring that all scores be reported results in superior outcomes in equilibrium, both from the perspective of the college (the admissions rule is more accurate), and from the perspective of equity across populations: a student's probability of admission is independent of their population, conditional on their type. In particular, the false positive rates and false negative rates are identical in this setting, across the highly and poorly resourced student populations. This is the case despite the fact that the more highly resourced students can -- at their option -- either report a more accurate signal of their type, or pool with the lower resourced population under this policy. 
    more » « less
  5. Ratings are present in many areas of assessment including peer review of research proposals and journal articles, teacher observations, university admissions and selection of new hires. One feature present in any rating process with multiple raters is that different raters often assign different scores to the same assessee, with the potential for bias and inconsistencies related to rater or assessee covariates. This paper analyzes disparities in ratings of internal and external applicants to teaching positions using applicant data from Spokane Public Schools. We first test for biases in rating while accounting for measures of teacher applicant qualifications and quality. Then, we develop model-based inter-rater reliability (IRR) estimates that allow us to account for various sources of measurement error, the hierarchical structure of the data, and to test whether covariates, such as applicant status, moderate IRR. We find that applicants external to the district receive lower ratings for job applications compared to internal applicants. This gap in ratings remains significant even after including measures of qualifications and quality such as experience, state licensure scores, or estimated teacher value added. With model-based IRR, we further show that consistency between raters is significantly lower when rating external applicants. We conclude the paper by discussing policy implications and possible applications of our model-based IRR estimate for hiring and selection practices in and out of the teacher labor market. 
    more » « less