<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>What Do Peer Evaluations Represent? A Study of Rater Consensus and Target Personality</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>03/21/2022</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10335853</idno>
					<idno type="doi">10.3389/feduc.2022.746457</idno>
					<title level='j'>Frontiers in Education</title>
<idno>2504-284X</idno>
<biblScope unit="volume">7</biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Chris C. Martin</author><author>Kenneth D. Locke</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[When professors assign group work, they assume that peer ratings are a valid source of information, but few studies have evaluated rater consensus in such ratings. We analyzed peer ratings from project teams in a second-year university course to examine consensus. Our first goal was to examine whether members of a team generally agreed on the competence of each team member. Our second goal was to test if a target’s personality traits predicted how well they were rated. Our third goal was to evaluate whether the self-rating of each student correlated with their peer rating. Data were analyzed from 130 students distributed across 21 teams (mean team size = 6.2). The sample was diverse in gender and ethnicity. Social relations model analyses showed that on average 32% of variance in peer-ratings was due to “consensus,” meaning some targets consistently received higher skill ratings than other targets did. Another 20% of the variance was due to “assimilation,” meaning some raters consistently gave higher ratings than other raters did. Thus, peer ratings reflected consensus (target effects), but also assimilation (rater effects) and noise. Among the six HEXACO traits that we examined, only conscientiousness predicted higher peer ratings, suggesting it may be beneficial to assign one highly conscientious person to every team. Lastly, there was an average correlation of.35 between target effects and self-ratings, indicating moderate self-other agreement, which suggests that students were only weakly biased in their self-ratings.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head>INTRODUCTION</head><p>When professors assign group work to students, they must grade students on work that they cannot observe. Many professors use summative peer evaluations to handle this challenge <ref type="bibr">(Ohland et al., 2012;</ref><ref type="bibr">Riebe et al., 2016)</ref>. If students complete those evaluations honestly and professors follow best practices, the scores should reflect the students' performance <ref type="bibr">(Donia et al., 2018)</ref> and promote accountability <ref type="bibr">(Brooks and Ammons, 2003)</ref>. Assuming score validity, the professor may then use these scores to assign grades <ref type="bibr">(Beatty et al., 1996)</ref> and ask students to use the scores to appraise their strengths and weaknesses <ref type="bibr">(Donia et al., 2018)</ref>. However, relatively few researchers have evaluated whether these scores reflect consensus within the team about each person's performance (e.g., <ref type="bibr">Loignon et al., 2017)</ref>. Our goal is to conceptually replicate earlier findings about consensus in peer ratings and to validate them further by examining-in the context of an undergraduate course emphasizing teamworkthe association between peer ratings and both personality traits and self-ratings.</p><p>Analyzing the degree of consensus-the first goal-entails discerning how much of the variance in the scores is attributable to the evaluation target. Ideally, variance should only hinge on the person being evaluated, indicating perfect consensus: each target receives similar ratings from every peer who evaluates them. But in actuality some variance is explained by the idiosyncrasies of each perceiver and other factors <ref type="bibr">(Scullen et al., 2000;</ref><ref type="bibr">Schmidt et al., 2021)</ref>. A number of studies have explored how much peer or observer ratings reflect consensus about the target rather than rater biases or other influences <ref type="bibr">(Kenny, 2020)</ref>. However, to our knowledge, few previous peer-reviewed publications have used a social relations model to assess consensus in the peer-ratings made by members of student teams (e.g., <ref type="bibr">Greguras et al., 2001;</ref><ref type="bibr">Mayfield and Tombaugh, 2019;</ref><ref type="bibr">Schmidt et al., 2021)</ref>. Therefore, our first goal was to add to this limited literature by assessing the consensus in peer-ratings using a different sample, setting, and rating system.</p><p>Our second goal was to test if the peer ratings of team members could be predicted by those members' personality traits. Personality traits are dimensions on which an individual's actions, behavior, and thoughts can be characterized <ref type="bibr">(John and Srivastava, 1999;</ref><ref type="bibr">Sherman et al., 2015;</ref><ref type="bibr">Quirin et al., 2020)</ref>. We rely on the HEXACO model of personality <ref type="bibr">(Ashton and Lee, 2007)</ref>. The acronym stands for these six constituent traits, where capital letters correspond to the acronym: Honesty-humility, Emotionality, eXtraversion, Agreeableness, Conscientiousness, and Openness to experience. Students who are high in conscientiousness are organized, diligent, and dutiful <ref type="bibr">(Dollinger and Orf, 1991)</ref>, which forestalls problems such as social loafing, inequity, and inequity-based motivation loss <ref type="bibr">(Mulvey and Klein, 1998;</ref><ref type="bibr">Davies, 2009;</ref><ref type="bibr">Wilson et al., 2018)</ref>. Conscientiousness should therefore be valued by both professors and students; a target's conscientiousness should stand out as a predictor of peerevaluation score, relative to other traits. If substantial variance is explained by another trait, say extraversion, that would suggest that peer ratings may hinge more on a person's charisma than on their dependability as a team member, which would warrant further inquiry.</p><p>Our third goal was to test if peer ratings correlate with self-ratings. In other words, do people who are rated by others as more/less competent also rate themselves as more/less competent? Whereas some research shows agreement between self-rated and peer-rated assessments (e.g., <ref type="bibr">Watson and Clark, 1991)</ref>, other research shows mixed results (e.g., <ref type="bibr">Heintzelman et al., 2020, 13)</ref> or no significant agreement <ref type="bibr">(Mayfield and Tombaugh, 2019)</ref>. In the current data, significant positive correlations would suggest peer ratings and self-ratings both measure the same construct-presumably, teamwork skills; in contrast, weak or non-significant correlations would suggest that these ratings are a poor indicator of teamwork skills, which could be imputed to less competent individuals being particularly prone to overestimate their competence <ref type="bibr">(Pennycook et al., 2017)</ref>.</p><p>If we conceptualize teamwork skills as a latent variable along which each team member has a true score, then peer-ratings are valid to the degree they reflect those true scores. To the degree that members' true scores determine peer-ratings, peer-ratings will show interrater reliability or consensus (see Goal 1 above). If members' personalities influence their true scores, then to the degree that members' true scores determine peer-ratings, peerratings will be associated with members' personality traits (see Goal 2). And if members' true scores also influence their selfratings, then to the degree that members' true scores determine peer-ratings, peer-ratings will converge with self-ratings (see Goal 3). Thus, finding that peer-ratings show consensus across perceivers, agreement with self-ratings, and associations with personality traits should all strengthen our confidence that peer-ratings are valid indicators of students' teamwork skills. Nonetheless, absent objective indicators of teamwork skills, we should recognize that other explanations remain possible; for example, team members' with more upbeat personalities (but not necessarily more skills) may give themselves nicer ratings and also receive nicer ratings from their teammates.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Social Relations Model</head><p>To achieve our aims, we employ Social Relations Model (SRM) analyses <ref type="bibr">(Kenny, 1994;</ref><ref type="bibr">Locke, 2016)</ref>. The SRM conceptualizes within-team differences in peer ratings as resulting from three influences: rater or perceiver effects, ratee or target effects, and residual perceiver x target interaction effects. A member's target effect reflects the ratings that member typically receives. A member's perceiver effect reflects the ratings that member typically gives. Finally, the residual interaction or relationship effect reflects a perceiver's distinctive perception of a target that cannot be explained by the perceiver's perceiver effect or the target's target effect. To illustrate, imagine a team of four students-A, B, C, and D-rate each other on "collaboration skills". If B rates C positively, then that may reflect a target effect (the target, C, receives positive ratings from everyone), a perceiver effect (the perceiver, B, gives everyone positive ratings), and a residual rater x ratee or relationship effect (B rates C more positively than would be predicted from C's target effect and B's perceiver effect).</p><p>Whereas target effects and self-ratings are ratings of the same target (the self) by different perceivers (teammates versus self), perceiver effects and self-ratings reflect ratings of different targets (teammates versus self) by the same perceiver (the self). The current study primarily concerns target effects. Specifically, our second and third goals (see above) concern if individuals' target effects-the distinctive ratings they typically receive from their teammates-correlate with their HEXACO traits and self-ratings. And our first goal concerns consensus, which is the variation in team members' target effects or "target variance."</p><p>Significant target variance or consensus means that some members consistently receive higher peer ratings than others do. In other words, raters agree about who is above average and who is below average. For example, if everyone on the team agrees about which members deserve higher ratings and which members deserve lower ratings, consensus is high. In contrast, the more each member receives a mix of higher ratings from some teammates and lower ratings from others, the lower the consensus. Everyone on the team receiving similar ratings (e.g., everyone gets high ratings) would not increase consensus because consensus reflect agreement about how team members differ from each other rather than agreement about how team members are similar to each other.</p><p>Variation in team members' perceiver effects is perceiver variance. Significant perceiver variance indicates that some members consistently give higher ratings than others do. Perceiver variance is also called assimilation because it indicates how much a perceiver is apt to place different people into the same judgment category. For example, if members A and D give everyone positive ratings, member B gives everyone middle ratings, and member D gives everyone negative ratings, then perceiver variance is high. In contrast, the more each member gives a mix of higher ratings to some teammates and lower ratings to others, the lower the perceiver variance. Again, everyone on the team giving similar ratings would not increase assimilation because assimilation reflects consistencies in how perceivers differ from each other.</p><p>Finally, variation in team members' relationship effects is residual (i.e., relationship or error) variance, and indicates the degree to which a specific team member's peer rating of a specific teammate cannot be predicted from their respective perceiver and target effects.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>MATERIALS AND METHODS</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Participants</head><p>Participants were recruited from a second-year undergraduate course in biomedical engineering at a large public university in the U.S. Teams in this course are required to work on a semester-long entrepreneurial project (see Supplementary Data Sheet 1). Students who either did not contribute any peer ratings or did not consent to participate were not included in the study. Also, since social relations model analyses require a minimum group size of four, we removed three teams in which fewer than three members made peer ratings that we had permission to use. After these exclusions, the sample consisted of 130 students distributed across 21 teams (M team size = 6.2, SD = 0.9, range = 4 -8 members). These students made a total of 688 round-robin peer ratings. The sample's gender composition was 30% male, 67% female, 0% transgender, 1% other, 2% missing; and its racial composition was 41% non-Hispanic White, 10% Hispanic only/Hispanic White, 10% African American/Black, 28% Asian American/Asian, 3% Asian-White biracial, 2% Middle Eastern/Arab/Arab American, 3% other, and 3% missing.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Measures</head><p>Self-Ratings and Peer Ratings Team members rated themselves and each other in four skill domains: inquiry skills, knowledge building skills, problemsolving skills, and collaboration skills (see Supplementary Table <ref type="table">1</ref>). Inquiry skills pertain to the acquisition and classification of robust and pertinent secondary research. Knowledge building skills pertain to the identification and remediation of knowledge gaps, the use of probing questions (posed to peers or oneself), the practice of teaching peers, and the practice of learning from peers. Problem-solving skills pertain to the development of a mathematical and physical model, the framing of hypotheses to be tested with those models, and the practice of clearly defining and expressing problems and solutions during meetings. Collaboration skills pertain to personal initiative and assertiveness; warm, honest, and responsive treatment of peers; and the maintenance of psychological safety within the team <ref type="bibr">(Edmondson, 1999)</ref>. They also pertain to the distribution and coordination of work in a fair manner.</p><p>At the beginning of the semester, each student was given a table with three to five rows of behavioral anchors per domain, each ordered from worst (left) to best (right) across four columns (see Supplementary Table <ref type="table">1</ref>). For instance, in one row, the worst anchor was "Does little to help produce papers &amp; presentations" and the best was "Assures papers &amp; presentations are well-written, in one voice, &amp; properly structured." Students were asked to review this table in order to understand their success criteria. In the middle of the semester, an oral evaluation was conducted: each student stated their selfrating and the student's peers then provided qualitative feedback on the accuracy of that self-rating. At the end of the semester, students were asked to refer to this table, assign numerical ratings to oneself and one's peers, and write qualitative evaluations to accompany the ratings. For the numerical ratings-analyzed in the current study-students entered a score in the 1-to-5 range for each domain, with non-integer ratings (e.g., 4.5) allowed but discouraged.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Personality Traits</head><p>The 100-item HEXACO-PI-R <ref type="bibr">(Lee and Ashton, 2018)</ref> was used to measure six traits:</p><p>1. honesty-humility (vs. selfish deceitfulness), 2. emotionality (vs. assuredness or stability), 3. extraversion (vs. docility or depression), 4. agreeableness (vs. angry or uncompassionate), 5. conscientiousness (vs. imprudence or irresponsibility), and 6. openness to experience (vs. conventionality).</p><p>Each trait corresponds to distinct outcomes <ref type="bibr">(Zettler et al., 2020)</ref> and distinctions between these six traits can be found in the lexicons of numerous languages <ref type="bibr">(Lee and Ashton, 2008)</ref>. For a comparison with the more familiar Big Five framework, see <ref type="bibr">Barford et al. (2015)</ref>. The 100-item HEXACO-PI-R contains 16 items per trait, each answered on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). The respective reliabilities (Cronbach's &#945;s) were 0.85, 0.82, 0.87, 0.94, 0.86, and 0.78. Participants completed the HEXACO-PI-R early in the semester.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>RESULTS</head><p>Table <ref type="table">1</ref> shows descriptive statistics for the peer ratings and self-ratings in each domain. The ratings had negatively skewed distributions (i.e., there were relatively few low ratings). In the analyses below we did not employ procedures to normalize the distributions such as elimination of outliers or algebraic transformations. Such procedures are rarely used in real-world classroom or organizational settings, nor are they ever used prior to SRM analyses. Nonetheless, to ensure the robustness of our linear regression results, we re-ran those regressions after log transforming self-ratings and target effects, and (as detailed in Supplementary Tables <ref type="table">3,</ref><ref type="table">4</ref>) in no case did those transformations alter the key findings.</p><p>As noted earlier, 130 members of 21 teams made 688 roundrobin peer ratings. To decompose the sources of variation in the peer ratings, we conducted SRM analyses <ref type="bibr">(Kenny, 1994)</ref> using the R package TripleR <ref type="bibr">(Sch&#246;nbrodt et al., 2012)</ref>.</p><p>First, we computed target effects (the average peer rating each member received, controlling for perceiver effects), perceiver effects (the average peer rating each member made, controlling for target effects), and residual effects (the residual peer rating after controlling for target and perceiver effects). Second, within each team we computed the variance in target effects, perceiver effects, and residual effects. Finally, we computed a weighted average of these target, perceiver, and residual variance estimates across teams. Table <ref type="table">2</ref> shows the results.</p><p>There was statistically significant (p &lt; 0.05) target variance and perceiver variance in ratings of all four skills (see righthand columns of Table <ref type="table">2</ref>). Target variance (consensus) was consistently greater than perceiver variance (assimilation). The significant assimilation indicates that ratings varied partly because some members gave generally higher or lower ratings than other members. But the significant consensus indicates that ratings also varied partly because some team members received generally higher or lower ratings than other members; that is, teammates showed agreement about who deserved higher versus lower skill ratings.</p><p>To facilitate interpretation, we computed the proportion of variance attributable to each component by dividing each component by the sum of all three components (see first column of Table <ref type="table">2</ref>). These standardized estimates show that on average approximately 32% (i.e., 23-39%) of the variance in peer ratings was attributable to targets, 20% (18-23%) was attributable to perceivers, and the remaining 49% (43-55%) was not attributable to either targets or perceivers. Second, we asked if team members' target effects-tendencies to be judged as more or less skillful-were associated with their personality traits. To answer this question, we regressed members' target effects simultaneously on their six HEXACO traits, the skill being rated (coded using three effect-code variables, and the 18 possible trait x skill interaction terms). Because targets were nested within teams, we allowed target effects intercept to vary across teams; all other parameters were treated as fixed. We standardized the outcome and predictors (except the trait x skill interactions) across the entire sample making the resulting regression coefficients interpretable as standardized coefficients. The complete model and results are detailed in the Supplementary Material (see Supplementary Table <ref type="table">3</ref>). There was a robust positive association between self-rated conscientiousness and peer-rated skills, &#946; 50 = 0.165, SE = 0.047, 95% CI [.073,.257], t = 3.54, p &lt; 0.001. There was a weak positive association between self-rated honestyhumility and peer-rated skills, &#946; 10 0.098, SE = 0.048, 95% CI [0.003, 0.192], t = 2.03, p &lt; 0.05; since this latter effect would be non-significant following any adjustment for Type I error due to model complexity, we will not discuss it further.</p><p>Third, we asked whether self-ratings correlate with target effects. To answer this question, we used a similar regression model as above except with self-ratings replacing HEXACO traits as the predictor. The complete model and results are detailed in the Supplementary Material (see Supplementary Table <ref type="table">4</ref>). Self-ratings and target effects were strongly positively related: Individuals who received higher (lower) ratings from their peers tended to give themselves higher (lower) ratings, &#946; 10 = 0.310, SE = 0.040, 95% CI [0.232, 0.388], t = 7.85, p &lt; 0.001. There was also one weak (p = 0.040) moderating effect of skill type, but since it was no longer significant (p = 0.161) after log transforming the skewed rating variables, we will not discuss it further.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>DISCUSSION</head><p>Our first aim was to assess agreement in peer-ratings. On average 32% of variance in peer-ratings reflected consensus (target variance) regarding who was higher or lower on each skill.</p><p>In showing that approximately one third of the variance in peer ratings reflected determinants of peer ratings that were shared across team members, the current results simultaneously showed that approximately two thirds of the variance in peer ratings reflected determinants of peer ratings that varied across team members.</p><p>One source of disagreement was assimilation: Across targets the tendency for some raters to make relatively high and others to make relatively low ratings of a particular skill explained approximately 20% of the variance. The other source of disagreement was residual variance that could not be attributed to variance in either target-or perceiver-effects. Residual variance may reflect random unsystematic sources of variance (such as careless responding or misreading the rating criteria) but may also reflect relationship-specific variance (such as team members having distinctive biases towards or against specific teammates). Residual variance was approximately half of the overall variance in peer ratings.</p><p>Thus, while peer ratings may be a useful indicator of student performance and skill levels, they are a noisy indicator carrying considerable perceiver variance and other sources of error. On the one hand, averaging multiple peer ratings should decrease noise by averaging out the effects of unsystematic sources of error and the rating biases reflected in perceiver and relationship effects. On the other hand, averaging will not dampen-and instead may amplify-the impact of shared biases, such as some targets tending to receive more lenient evaluations from everyone (e.g., because they tend to evoke more liking or sympathy) independent of their true skill levels.</p><p>Three previous studies analyzed target variance in peer-ratings in higher-education courses. <ref type="bibr">Greguras et al. (2001)</ref> found on average 26% of the variance in peer ratings was target variance and 29% was perceiver variance. <ref type="bibr">Mayfield and Tombaugh (2019)</ref> found on average 22% of the variance in peer ratings was target variance and 36% was perceiver variance. <ref type="bibr">Schmidt et al. (2021)</ref> found on average 28% was target variance and 35% was perceiver variance. Thus, perceiver variance tended to be lower (and target variance slightly higher) in the current study than in previous studies, perhaps because the students we studied were explicitly encouraged to "not give everyone the same <ref type="bibr">[rating]</ref>."</p><p>Future research could assess the specific attitudes and beliefs that might contribute to perceiver variance (e.g., a "4" on the 1to-5 scale may have seemed like a "good" score to some raters but a "bad" score to other raters; and whereas some raters may have construed giving a mediocre performer a mediocre score as a responsible behavior, others may have construed it as an unkind behavior). Presumably, improving the instructions for making peer ratings (e.g., anchoring scale points to simple observable criteria) reduces perceiver and error variance and increases target variance. Thus, future research could systematically vary instructions and criteria across teams and then measure which teams show more consensus.</p><p>For example, in our analysis, consensus was greatest for ratings of problem-solving skills and lowest for knowledge-building skills. Previous research suggests that consensus tends to be greater for qualities that are more observable <ref type="bibr">(Kenny and West, 2010)</ref>. Accordingly, one possible explanation for the differences in consensus is that the criteria for problem-solving skills (e.g., "actively participating in finding a solution. . . use white boards. . .or just sit passively") referred to more concrete and observable behaviors than did the criteria for knowledge-building skills (e.g., "regularly assess where s/he needed to develop deeper understanding or was satisfied with surface understanding"). If so, then articulating more visible behavioral indicators of knowledge-building skills might improve consensus in ratings of those skills.</p><p>Having found significant consensus in peer ratings, our study's second and third goals involved testing if those peer ratings correlated with personality traits and selfratings. Conscientiousness was positively associated with peer evaluations, which accords with our expectations and previous research. The other HEXACO traits-which are less directly linked to task behavior-were unrelated to peer ratings.</p><p>Finally, we found moderate associations (average r = 0.35) between target effects and self-ratings, which is stronger than that reported in the previous studies of student teams <ref type="bibr">(Loignon et al., 2017;</ref><ref type="bibr">Mayfield and Tombaugh, 2019)</ref>. As with peer-peer consensus, peer-self agreement can be interpreted as evidence for the accuracy of both self-and peer-ratings (i.e., presumably the ratings of different raters converge because they are all influenced by a target's true skill level). But as with peer-peer consensus, other interpretations are possible (e.g., the ratings might converge because they are all influenced by the same biases).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Limitations</head><p>On one hand, the current findings obtained from student teams may not generalize to teams that are dissimilar in size, professional field, average age, timeline, and goals. On the other hand, the basic structure of these teams resembles that of many teams operating in "real-world" organizational and applied contexts: members were given a meaningful problem to solve and were accountable for their performance. These findings may also not generalize to evaluations conducted early in the semester <ref type="bibr">(Mayfield and Tombaugh, 2019)</ref>, but those evaluations are formative rather than summative-and thus serve a different purpose. Better rubrics could also potentially pull more target variance and less perceiver variance, which means the ratio of target-to-perceiver variance can change across studies.</p><p>Also, as a cautionary tale for future researchers and instructors, we will admit that we originally had a fourth aim which we had to abandon: To check if the peer ratings showed convergent validity with observer ratings made by teaching assistants who attended the students' team meetings. However, after we found suspiciously high correlations between peer ratings and observer ratings, further inquiries revealed that the teaching assistants had peeked at the peer ratings before making their own ratings of each student, thereby rendering their observer ratings unusable as a validity criterion. Our takehome lesson was that some raters-and not just peer raters but also teaching assistants and perhaps instructors-may need to be reminded that aggregating across raters can reduce or eliminate the impact of each individual's idiosyncratic mistakes and biases, but only if those individuals make their ratings completely independently of each other.</p><p>However, even if peer ratings were validated by evaluations made by independent observers, those observers would be informed by team members' comments and susceptible to the same biases as the ratings made by peers (such as halo effects for more likable team members). Thus, future research employing objective performance metrics would enhance confidence that the peer ratings measured team skills rather than being confounded with other qualities.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Implications</head><p>The current results uphold the robustness of earlier findings about consensus among peers in an engineering education and other settings <ref type="bibr">(Greguras et al., 2001;</ref><ref type="bibr">Loignon et al., 2017)</ref>. By studies examining different rating systems in different samples and contexts, collectively this expanding literature provides an increasingly generalizable foundation from which to predict how much consensus new rating systems in new contexts might achieve. This study also adds indirect support for the finding that behavioral anchors are useful <ref type="bibr">(Ohland et al., 2005)</ref>, although there was no control group in our case.</p><p>A novel finding is that conscientiousness exclusively predicted how well students were evaluated. When configuring teams, instructors may leverage this knowledge and ensure there's at least one highly conscientious member in every team. Given that conscientiousness can be measured with freely available scales in multiple languages (e.g., <ref type="bibr">Lee and Ashton, 2004;</ref><ref type="bibr">Soto and John, 2017)</ref>, this is feasible for most instructors. Prompting peers to appraise each other's conscientiousness may also be useful as formative feedback. Likewise, structured feedback forms can include items with behavioral anchors to measure conscientiousness. In the peer-ratings literature, scholars have devoted relatively little attention to the content validity of peer ratings, and the current study's findings about traits point to further avenues for research on content.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" xml:id="foot_0"><p>Frontiers in Education | www.frontiersin.org</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" xml:id="foot_1"><p>March 2022 | Volume 7 | Article 746457</p></note>
		</body>
		</text>
</TEI>
