skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • Using Evidence from Task-Based Interviews for Development and Validation of Classroom Assessments: Approaches and Applications in Early Grades Mathematics
Title: Using Evidence from Task-Based Interviews for Development and Validation of Classroom Assessments: Approaches and Applications in Early Grades Mathematics
This article illustrates and differentiates the unique role cognitive interviews and think-aloud interviews play in developing and validating assessments. Specifically, we describe the use of (a) cognitive interviews to gather empirical evidence to support claims about the intended construct being measured and (b) think-aloud interviews to gather evidence about the problem-solving processes students use while completing tasks assessing the intended construct. We illustrate their use in the context of a classroom assessment of an early mathematics construct – numeric relational reasoning – for kindergarten through Grade 2 students. This assessment is intended to provide teachers with data to guide their instructional decisions. We conducted 64 cognitive interviews with 32 students to collect evidence about students’ understanding of the construct. We conducted 106 think-aloud interviews with 14 students to understand how the prototypical items elicited the intended construct. The task-based interview results iteratively informed assessment development and contributed important sources of validity evidence.  more » « less
Award ID(s):
1721100
PAR ID:
10575796
Author(s) / Creator(s):
; ;
Publisher / Repository:
Routledge
Date Published:
Journal Name:
Educational Assessment
Volume:
29
Issue:
4
ISSN:
1062-7197
Page Range / eLocation ID:
229 to 250
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, Practical assessment research evaluation)
    Cognitive interviews play an important role in articulating the intended construct of educational assessments. This paper describes the iterative development of protocols for cognitive interviews with kindergarten through second-grade children to understand how their spatial reasoning skill development aligns with intended constructs. We describe the procedures used to gather evidence of construct relevance and improved alignment to task-based interview items through multiple pilot rounds before conducting cognitive interviews. We found improved alignment and reduced construct irrelevant variance after protocol revisions. 
    more » « less
  2. ; ; ; ; ; ; ; ; (, Journal of cyber security)
    We conducted an observational study to document student misconceptions about cybersecurity using thematic analysis of 25 think-aloud interviews. By understanding patterns in student misconceptions, we provide a basis for developing rigorous evidence-based recommendations for improving teaching and assessment methods in cybersecurity and inform future research. This study is the first to explore student cognition and reasoning about cybersecurity. We interviewed students from three diverse institutions. During these interviews, students grappled with security scenarios designed to probe their understanding of cybersecurity, especially adversarial thinking. We analyzed student statements using a structured qualitative method, novice-led paired thematic analysis, to document patterns in student misconceptions and problematic reasoning that transcend institutions, scenarios, or demographics. Themes generated from this analysis describe a taxonomy of misconceptions but not their causes or remedies. Four themes emerged: overgeneralizations, conflated concepts, biases, and incorrect assumptions. Together, these themes reveal that students generally failed to grasp the complexity and subtlety of possible vulnerabilities, threats, risks, and mitigations, suggesting a need for instructional methods that engage students in reasoning about complex scenarios with an adversarial mindset. These findings can guide teachers’ attention during instruction and inform the development of cybersecurity assessment tools that enable cross-institutional assessments that measure the effectiveness of pedagogies. 
    more » « less
  3. ; ; ; ; ; ; (, International Society of the Learning Sciences)
    This poster presents findings on middle school students’ understanding of core Computer Science (CS) concepts, such as variables and control structures, using cognitive think-aloud interviews with eight students. Each student worked on 16-22 formative assessment tasks designed to assess understanding on the ‘Algorithms and Programming’ middle school CS standards. Our study describes students’ interpretations of the CS concepts and discusses potential factors influencing student interpretations. Significance and next steps are described. 
    more » « less
  4. (, Applied measurement in education)
    Response process validity evidence provides a window into a respondent’s cognitive processing. The purpose of this study is to describe a new data collection tool called a whole-class think aloud (WCTA). This work is performed as part of test development for a series of problem-solving measures to be used in elementary and middle grades. Data from third-grade students were collected in a 1-1 think aloud setting and compared to data from similar students as part of WCTAs. Findings indicated that students performed similarly on the items when the two think-aloud settings were compared. Respondents also needed less encouragement to share ideas aloud during the WCTA compared to the 1-1 think aloud. They also communicated feeling more comfortable in the WCTA setting compared to the 1-1 think aloud. Drawing the findings together, WCTAs functioned as well if not better, than 1-1 think alouds for the purpose of contextualizing third grade students’ cognitive processes. Future studies using WCTAs are recommended to explore their limitations and other factors that might impact their success as data gathering tools. 
    more » « less
  5. ; ; ; (, ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering)
    Software engineering candidates commonly participate in whiteboard technical interviews as part of a hiring assessment. During these sessions, candidates write code while thinking aloud as they work towards a solution, under the watchful eye of an interviewer. While technical interviews should allow for an unbiased and inclusive assessment of problem-solving ability, surprisingly, technical interviews may be instead a procedure for identifying candidates who best handle and migrate stress solely caused by being examined by an interviewer (performance anxiety). To understand if coding interviews---as administered today---can induce stress that significantly hinders performance, we conducted a randomized controlled trial with 48 Computer Science students, comparing them in private and public whiteboard settings. We found that performance is reduced by more than half, by simply being watched by an interviewer. We also observed that stress and cognitive load were significantly higher in a traditional technical interview when compared with our private interview. Consequently, interviewers may be filtering out qualified candidates by confounding assessment of problem-solving ability with unnecessary stress. We propose interview modifications to make problem-solving assessment more equitable and inclusive, such as through private focus sessions and retrospective think-aloud, allowing companies to hire from a larger and diverse pool of talent. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email