An official website of the United States government
When professors assign group work, they assume that peer ratings are a valid source of information, but few studies have evaluated rater consensus in such ratings. We analyzed peer ratings from project teams in a second-year university course to examine consensus. Our first goal was to examine whether members of a team generally agreed on the competence of each team member. Our second goal was to test if a target’s personality traits predicted how well they were rated. Our third goal was to evaluate whether the self-rating of each student correlated with their peer rating. Data were analyzed from 130 students distributed across 21 teams (mean team size = 6.2). The sample was diverse in gender and ethnicity. Social relations model analyses showed that on average 32% of variance in peer-ratings was due to “consensus,” meaning some targets consistently received higher skill ratings than other targets did. Another 20% of the variance was due to “assimilation,” meaning some raters consistently gave higher ratings than other raters did. Thus, peer ratings reflected consensus (target effects), but also assimilation (rater effects) and noise. Among the six HEXACO traits that we examined, only conscientiousness predicted higher peer ratings, suggesting it may be beneficial to assign one highly conscientious person to every team. Lastly, there was an average correlation of.35 between target effects and self-ratings, indicating moderate self-other agreement, which suggests that students were only weakly biased in their self-ratings.
more »
« less
Debiasing Evaluations That Are Biased by Evaluations
It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference organizers ask authors of submissions to evaluate the quality of the reviews. However, in these applications, students often give a higher rating to a course if they receive higher grades in a course, and authors often give a higher rating to the reviews if their papers are accepted to the conference. In this work, we call these external factors the" outcome" experienced by people, and consider the problem of mitigating these outcome-induced biases in the given ratings when some information about the outcome is available. We formulate the information about the outcome as a known partial ordering on the bias. We propose a debiasing method by solving a regularized optimization problem under this ordering constraint, and also provide a carefully designed cross-validation method that adaptively chooses the appropriate amount of regularization. We provide theoretical guarantees on the performance of our algorithm, as well as experimental evaluations.
more »
« less
- Award ID(s):
- 1942124
- PAR ID:
- 10570458
- Publisher / Repository:
- Journal of Machine Learning Research
- Date Published:
- Journal Name:
- Journal of machine learning research
- ISSN:
- 1532-4435
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
In order be successful, engineers must ask their clients, coworkers, and bosses questions. Asking questions can improve work quality and make the asker appear smarter. However, people often hesitate to ask questions for fear of seeming incompetent or inferior. This study investigates: what characteristics and experiences are connected to engineering students’ perceptions of asking questions? We analyzed data from a survey of over a thousand engineering undergraduates across a nationally representative sample of 27 U.S. engineering schools. We focused on three dependent variables: question-asking self-efficacy (how confident students are in their ability to ask a lot of questions), social outcome expectations around asking questions (whether students believe if they ask a lot of questions, they will earn the respect of their colleagues), and career outcome expectations (whether they believe asking a lot of questions will hurt their chances for getting ahead at work). We were surprised to find that question-asking self-efficacy or outcome expectations did not significantly vary by gender, under-represented minority status, and school size. However, students with high question-asking self-efficacy and outcome expectations were more likely to have engaged in four extracurricular experiences: participating in an internship or co-op, conducting research with a faculty member, participating in a student group, and holding a leadership role in an organization or student group. The number of different types of these extracurricular activities a student engaged in correlated with question-asking self-efficacy and positive outcome expectations around asking questions. The results illustrate the relationship between extracurricular activities and students’ self-efficacy and behavior outcome expectations. The college experience is more than just formal academic classes. Students learn from experiences that occur after class or during the summer, and ideally these experiences complement class-derived skills and confidence in asking questions.more » « less
-
Attending computer science conferences can give students insight into the research process and how academic work is disseminated. This study examines undergraduate student perceptions about attending an interdisciplinary computational biology conference. The study was conducted over four academic years with a mix of participants who attended a conference as part of a course and participants who received an undergraduate travel award. Results from 70 students enrolled in nearly 30 different institutions indicate that attending conferences helped them learn about different careers, gave them a sense of what computational biology research entails, and provided insight into giving an effective oral presentation. We found that students who received a travel award felt more comfortable at the conferences than students who attended as part of a course. Based on these findings, we provide guidance about developing programs for undergraduate conference attendance.more » « less
-
In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper qualities from an author with multiple submissions to help improve the conference's decisions. However, the isotonic mechanism relies on the assumption that the author's utility is both an increasing and a convex function with respect to the review score, which is often violated in realistic settings (e.g.~when authors aim to maximize the number of accepted papers). In this paper, we propose a sequential review mechanism that can truthfully elicit the ranking information from authors while only assuming the agent's utility is increasing with respect to the true quality of her accepted papers. The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers. Advantages of the sequential review mechanism include: 1) eliciting truthful ranking information in a more realistic setting than prior work; 2) reducing the reviewing workload and increasing the average quality of papers being reviewed; 3) incentivizing authors to write fewer papers of higher quality.more » « less
-
Teaching Undergraduate Artificial Intelligence Classes: An Experiment with an Attendance RequirementWe report on an experiment that we performed when we taught the undergraduate artificial intelligence class at the University of Southern California. We taught it - under very similar conditions - once with and once without an attendance requirement. The attendance requirement substantially increased the attendance of the students. It did not substantially affect their performance but decreased their course ratings across all categories in the official course evaluation, whose results happened to be biased toward the opinions of the students attending the lectures. For example, the overall rating of the instructor was 0.89 lower (on a 1-5 scale) with the attendance requirement and the overall rating of the class was 0.85 lower. Thus, the attendance requirement, combined with the policy for administering the course evaluation, had a large impact on the course ratings, which is a problem if the course ratings influence decisions on promotions, tenure, and salary increments for the instructors but also demonstrates the potential for the manipulation of course ratings.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
