An official website of the United States government
Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students ( n = 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity.
more »
« less
From Likert to Forced Choice: Statement Parameter Invariance and Context Effects in Personality Assessment
Users of forced-choice questionnaires (FCQs) to measure personality commonly assume statement parameter invariance across contexts – between Likert and forced-choice (FC) items and between different FC items that share a common statement. In this paper, an empirical study was designed to check these two assumptions for an FCQ assessment measuring interpersonal and intrapersonal skills. We compared parameters of common statements between two Likert forms and two FCQ forms with a block size of two (statement pairs) and among five FCQ pair forms. In three of the five FCQ forms, statements were paired only with a statement they had appeared with in a triplet block. In the other two FCQ forms, statements were paired with statements they had not been paired with in a triplet block. This design allows us to evaluate statement parameter changes due to changes in context. The results do not support the statement parameter invariance assumption between Likert and FC items or the assumption between FC items when recombining statements form new items. However, the assumption between FC items was generally held for pairs formed by dropping a statement from a triplet item. There were some suggestions for sources of context effects, but the analyses were not definitive. Implications of the findings for test practice are discussed.
more »
« less
- Award ID(s):
- 2201888
- PAR ID:
- 10495888
- Publisher / Repository:
- Taylor & Francis Group
- Date Published:
- Journal Name:
- Measurement: Interdisciplinary Research and Perspectives
- ISSN:
- 1536-6367
- Page Range / eLocation ID:
- 1 to 17
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Little is known about the relation between law enforcement interviewing behaviors and commercially sexually exploited children’s (CSEC) reluctance. This study examined the relation between officers’ use of maximization, (references to) expertise, minimization, and support and adolescent CSEC victims’ reluctance in a small sample of police interviews (n = 2,416 question-answer pairs across ten interviews). Twenty-six percent of officers’ utterances contained at least one interviewing tactic. When statements were paired with maximization, they were correlated with more reluctance than when they were not paired with an interviewing tactic. Contrary to predictions, support was also related to greater reluctance. Open-ended (recall) questions and statements were associated with greater reluctance than closed-ended (recognition) questions. The results highlight the importance of understanding the context in which interviewing strategies are employed when assessing the relation between interviewer behavior and interviewee reluctance.more » « less
-
Briffa, Mark (Ed.)Abstract While female mate choice is well established, mutual choice may play a larger role in mate selection than currently recognized. Assortative mating is a common form of nonrandom mating in animals that can result from mutual choice. However, few studies address assortative patterns beyond the social pair, potentially overlooking assortativity in the mating pair and in the social environment that shapes reproductive decisions. We asked whether North American barn swallows (Hirundo rustica erythrogaster) breeding in a large colony form pairs, mate (through both within-pair and extra-pair fertilizations), and interact assortatively by ventral plumage color, wing length, and age. Social interactions were tracked using proximity loggers, which recorded close contact between tagged individuals when birds were mating and laying eggs. Barn swallows paired and mated assortatively by their ventral plumage color; however, the assortative patterns in mating pairs were not as strong as they were in social pairs. Barn swallows also interacted assortatively, associating more often with individuals of both sexes who had similar phenotypes relative to the other birds in the colony. Finally, older males and females with darker ventral plumage achieved the highest reproductive success. Investigation of assortative behavior beyond the level of the social pair provides a more complete understanding of mate choice and suggests a mechanism that may maintain the large variation in ventral plumage color in North American barn swallows.more » « less
-
Karunakaran, S. S.; Reed, Z.; Higgins, A. (Ed.)Future mathematics teachers must be able to interpret a wide range of mathematical statements, in particular conditional statements. Literature shows that even when students are familiar with conditional statements and equivalence to the contrapositive, identifying other equivalent and non-equivalent forms can be challenging. As a part of a larger grant to enhance and study prospective secondary teachers’ (PSTs’) mathematical knowledge for teaching proof, we analyzed data from 26 PSTs working on a task that required rewriting a conditional statement in several different forms and then determining those that were equivalent to the original statement. We identified three key strategies used to make sense of the various forms of conditional statements and to identify equivalent and non-equivalent forms: meaning making, comparing truth-values and comparing to known syntactic forms. The PSTs relied both on semantic meaning of the statements and on their formal logical knowledge to make their judgments.more » « less
-
When creating assessments, computer science educators and researchers must balance items' cognitive complexity and authenticity against scoring efficiency. In this poster, the author reports results from an end-of-course assessment administered to over 500 high school students in an introductory block-based programming course. The poster focuses on three atypical multiple-choice items, in which students had to select all the correct responses. The items were designed to be more cognitively complex than simple multiple choice questions while remaining easy to score. Results show that this type of item was challenging for students but was predictive of their overall performance.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1080/15366367.2023.2258482
