More Like this

1. Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability

   https://doi.org/http://dx.doi.org/10.1371/journal.pone.0203002  

   Martinková, P.; Goldhaber, D.; Erosheva, E. (January 2018, PloS one)

   Ratings are present in many areas of assessment including peer review of research proposals and journal articles, teacher observations, university admissions and selection of new hires. One feature present in any rating process with multiple raters is that different raters often assign different scores to the same assessee, with the potential for bias and inconsistencies related to rater or assessee covariates. This paper analyzes disparities in ratings of internal and external applicants to teaching positions using applicant data from Spokane Public Schools. We first test for biases in rating while accounting for measures of teacher applicant qualifications and quality. Then, we develop model-based inter-rater reliability (IRR) estimates that allow us to account for various sources of measurement error, the hierarchical structure of the data, and to test whether covariates, such as applicant status, moderate IRR. We find that applicants external to the district receive lower ratings for job applications compared to internal applicants. This gap in ratings remains significant even after including measures of qualifications and quality such as experience, state licensure scores, or estimated teacher value added. With model-based IRR, we further show that consistency between raters is significantly lower when rating external applicants. We conclude the paper by discussing policy implications and possible applications of our model-based IRR estimate for hiring and selection practices in and out of the teacher labor market. 

   more » « less
2. Use of Personas in Exploring Scholarship Applicants

   Rynearson, Anastasia; Gartner, Jacqueline; Miller, Michele (July 2021, ASEE Annual Conference proceedings)

   null (Ed.)

   A recent S-STEM award has allowed the engineering program in a rural, liberal arts institution to offer a need-based scholarship program for its students. The engineering program has a number of veteran, underrepresented minority, transfer, and nontraditional students. Many students are also first-generation college students. The institution and engineering program matriculate a number of under-served populations, students who may have needs that are not well understood in the typical engineering education literature. The scholarship program and its associated mentoring and activities will assist workforce development and will also incorporate a number of research avenues to better understand and serve the student population in this unique setting. To apply for the program, students must fill out an application with four 250 – 500 word essay responses relating to their academic progress, perceived barriers to degree completion, and how this award would help them to complete their degree. This study continues work using personas, a method used in human-centered design. Using the first round of scholarship application essays as a source, three personas were developed, one successful applicant, one unsuccessful applicant, and one general applicant. Personas are detailed profiles of a fake person who could reasonably be in each category of interest. In human-centered design, personas are detailed descriptions of likely clients or end-users, developed to help the engineers focus on who they might be designing for. The profiles developed in this study were used to gain insight into which students were likely to choose to apply and which students may be missing out on this opportunity. It is time for another round of applications for this grant and the use of personas will continue and expand as part of this study. Before reviewing applications, the committee will create two personas as ideal candidates instead of developing a standardized rubric. Subsequently, three personas will be developed from the Fall 2020 applications, one for all applicants, one for successful applicants, and one for unsuccessful applicants. These personas will then be compared to the personas created by the application review committee and the personas created from the Fall 2018 applicants. Similarities and differences across the persona groups will be explored to determine whether the applicants are what the reviewers expected and whether the pool of applicants has evolved or remained mostly the same throughout the scholarship program. Review committee members will also be interviewed in a focus group setting to discuss their experiences using common personas rather than standardized rubrics in the application review process. At this time, the applications are not yet due and the analysis has not yet begun. Initial interest for the grant has been strong and we anticipate at least thirty applications for the nineteen available grants. Results presented will include the student profiles and faculty experiences with the use of personas as a metric for reviewing student applications. 

   more » « less
3. Finding optimal non-datapath caching strategies via network flow

   Steven Lyons, Raju Rangaswami (February 2023, Theoretical computer science)

   Flash and non-volatile memory (NVM) devices have only a limited number of write-erase cycles. Consequently, when employed as caches, cache management policies may choose not to cache certain requested items in order to extend device lifespan. In this work, we propose a simple single-parameter utility function to model the trade-off between maximizing hit-rate and minimizing write-erase cycles for such caches, and study the problem of developing an off-line strategy for deciding whether to write a new item to cache, and if so which item already in the cache to replace. Our main result is, , an efficient, network flow based algorithm which finds optimal cache management policy under this new setting. 

   more » « less
4. Exploring the Impact of Program Name Change on Gender Diversity in STEM

   https://doi.org/10.18260/1-2--46850  

   Zafar, Faiza; Nichol, Carolyn (June 2024, ASEE Conference)

   Gender disparities in STEM remain a pressing concern, especially in engineering, where women receive only 20% of undergraduate degrees. This concise research paper delves into the potential influence of altering the name of an undergraduate research program on attracting female applicants. Specifically, we examine how changing the program's name from "Nanotechnology Research with a Focus on Community College Students" to "Climate and Sustainability Research in Nanotechnology and Electrochemical Devices for Community College Students" affects the gender balance among applicants, with the other program components remaining the same. Our research reveals a notable shift in the applicant pool following the program's name change. Initially, we had fewer female applicants in the “Nanotechnology Research with a Focus on Community College Students” program than male students. However, following the transition to "Climate and Sustainability," the number of female applicants increased, equaling almost all male applicants. To assess the statistical significance of the observed difference, we conducted the two-sample proportion test, which resulted in a p-value of 0.233, yielding insufficient evidence of a significant difference in the proportions of female applicants between the two programs. However, an intriguing finding emerged when we focused exclusively on the "Nanotechnology" program applicants. We expect both male and female applicants to be 50% each. However, the proportion of female applicants in this group did not align with the expected 50/50 gender distribution and yielded a p-value of 0.042, suggesting that female applicants in the "Nanotechnology" applicant pool were significantly lower than 50%. While the overall comparison between the two program names did not yield statistical significance, a deeper analysis within the "Nanotechnology" program uncovered a significant difference. These findings underscore the importance of program nomenclature as a factor in promoting gender diversity in STEM education and creating a more inclusive environment for underrepresented groups in the field. 

   more » « less
5. Online Learning with an Unknown Fairness Metric

   Gillen, S; Jung, C; Kearns, M; Roth, A (January 2019, Neural information Processing Systems (NeurIPS))

   We consider the problem of online learning in the linear contextual bandits setting, but in which there are also strong individual fairness constraints governed by an unknown similarity metric. These constraints demand that we select similar actions or individuals with approximately equal probability, which may be at odds with optimizing reward, thus modeling settings where profit and social policy are in tension. We assume we learn about an unknown Mahalanobis similarity metric from only weak feedback that identifies fairness violations, but does not quantify their extent. This is intended to represent the interventions of a regulator who “knows unfairness when he sees it” but nevertheless cannot enunciate a quantitative fairness metric over individuals. Our main result is an algorithm in the adversarial context setting that has a number of fairness violations that depends only logarithmically on T, while obtaining an optimal O(√T) regret bound to the best fair policy. 

   more » « less