An official website of the United States government
Abstract—This WIP research paper presents validity evidence for a survey instrument designed to assess student learning in makerspaces. We report findings from expert reviews of item content and student interpretations of survey questions. The instrument was developed using a theory-driven approach to define constructs, followed by the development of questions aligned with those constructs. We solicited written feedback from 30 experts in instrument development and/or makerspaces, who rated the alignment of items with our constructs. Based on this input, we revised our items for clarity and consistency. We then conducted 25 cognitive interviews with a diverse group of students who use makerspaces, asking them to explain their understanding of each item and the reasoning behind their responses. Our recruitment ensured diversity in terms of race, gender, ethnicity, and academic background, extending beyond engineering majors. From our initial 45 items, we removed 6, modified 36, and added 1 based on expert feedback. During cognitive interviews, we began with 40 items, deleted one, and revised 23, resulting in 39 items for the pilot survey. Key findings included the value of examples in clarifying broad terms and improved student engagement with a revised rating scale—shifting from a 7-point Likert agreement scale to a self-description format encouraged fuller use of the scale. Our study contributes to the growing body of research on makerspaces by offering insights into how students describe their learning experiences and by providing initial validation evidence for a tool to assess those experiences, ultimately strengthening the credibility of the instrument.
more »
« less
Designing items for measuring PCK for proportional reasoning.
The purpose of this poster is to report on findings from our development efforts. In prior papers, we have reflected on some of the challenges in writing items to measure teachers’ specialized content knowledge (e.g., Orrill et al, 2015). In this paper, we reflect on our analysis of think-aloud interviews to identify what we have learned about the development of PCK items for proportional reasoning.
more »
« less
- Award ID(s):
- 1813760
- PAR ID:
- 10343734
- Editor(s):
- Olanoff, D.; Johnson, K.; & Spitzer, S.
- Date Published:
- Journal Name:
- Proceedings of the fourth-third annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education
- Page Range / eLocation ID:
- 492-493
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Despite the importance of developing elementary science teachers' content knowledge for teaching (CKT), there are limited assessments that have been designed to measure the full breadth of their CKT at scale. Our overall research project addressed this gap by developing an online assessment to measure elementary preservice teachers' CKT about matter and its interactions. This study, which was part of our larger project, reports on findings from one component of the item development process examining the construct validity of 118 different CKT about matter assessment items. In this study, 86 elementary teachers participated in cognitive interviews to examine: (a) the knowledge and reasoning they used when responding to these CKT about matter assessment items and (b) the nature of the content challenges and the content teaching challenges they encountered. Findings showed that over 80% of participant interview responses indicated that the CKT about matter items functioned as hypothesized, providing evidence to support future use of these items on a large‐scale assessment and in studies of science teachers' CKT. When responding to the items, participants showed evidence of four main challenges with the science content: (a) using scientific concepts to reason about science tasks, (b) using adequate evidence to reason about science phenomenon, (c) drawing upon examples of scientific phenomena, and (d) drawing upon science vocabulary. Findings also showed that participants experienced challenges regarding the following content teaching aspects when responding to these items: (a) connecting to key scientific concepts involved in the work of teaching science, (b) attending to instructional goal(s), and (c) recognizing features of grade‐level appropriateness. Implications for using CKT items as part of large‐scale science assessment systems and identifying areas to target in elementary science teachers' CKT development are addressed.more » « less
-
nterest in science, technology, engineering, and mathematics (STEM) begins as early as elementary and middle school. As youth enter adolescence, they begin to shape their personal identities and start making decisions about who they are and could be in the future. Students form their career aspirations and interests related to STEM in elementary school, long before they choose STEM coursework in high school or college. Much of the literature examines either science or STEM identity and career aspirations without separating out individual sub-disciplines. Therefore, the purpose of this paper is to describe the development of a survey instrument to specifically measure engineering identity and career aspirations in adolescents and preadolescents. When possible, we utilized existing measures of STEM identity and career aspirations, adapting them when necessary to the elementary school level and to fit the engineering context. The instrument was developed within the context of a multi-year, NSF-funded research project examining the dynamics between undergraduate outreach providers and elementary students to understand the impact of the program on students’ engineering identity and career aspirations. Three phases of survey development were conducted that involved 492 elementary students from diverse communities in the United States. Three sets of items were developed and/or adapted throughout the four phases. The first set of items assessed Engineering Identity. Recent research suggests that identity consists of three components: recognition, interest, and performance/competence. Items assessing each of these constructs were included in the survey. The second and third sets of items reflected Career Interests and Aspirations. Because elementary and middle school students often have a limited or nascent awareness of what engineers do or misconceptions about what a job in science or engineering entails, it is problematic to measure their engineering identity or career aspirations by directly asking them whether they want to be a scientist/engineer or by using a checklist of broad career categories. Therefore, similar to other researchers, the second set of items assessed the types of activities that students are interested in doing as part of a future career, including both non-STEM and STEM (general and engineering-specific) activities. These items were created by the research team or adapted from activity lists used in existing research. The third set of items drew from career counseling measures relying on Holland’s Career Codes. We adapted the format of these instruments by asking students to choose the activity they liked the most from a list of six activities that reflected each of the codes rather than responding to their interest about each activity. Preliminary findings for each set of items will be discussed. Results from the survey contribute to our understanding of engineering identities and career aspirations in preadolescent and adolescent youth. However, our instrument has the potential for broader application in non-engineering STEM environments (e.g., computer science) with minor wording changes to reflect the relevant science subject area. More research is needed in determining its usefulness in this capacity.more » « less
-
The recognition of middle grades as a critical juncture in CS education has led to the widespread development of CS curricula and integration efforts. The goal of many of these interventions is to develop a set of underlying abilities that has been termed computational thinking (CT). This goal presents a key challenge for assessing student learning: we must identify assessment items associated with an emergent understanding of key cognitive abilities underlying CT that avoid specialized knowledge of specific programming languages. In this work we explore the psychometric properties of assessment items appropriate for use with middle grades (US grades 6-8; ages 11-13) students. We also investigate whether these items measure a single ability dimension. Finally, we strive to recommend a "lean" set of items that can be completed in a single 50-minute class period and have high face validity. The paper makes the following contributions: 1) adds to the literature related to the emerging construct of CT, and its relationship to the existing CTt and Bebras instruments, and 2) offers a research-based CT assessment instrument for use by both researchers and educators in the field.more » « less
-
As the need for interdisciplinary collaboration increases, industry needs engineers who are not only affluent in technical engineering skills but also efficient in skills such as communication, problem-solving, engineering ethics, and business management. As a result, engineering programs are tasked with providing students with sufficient opportunities to develop non-technical professional skills to better prepare them for the workforce. Previous research has focused on exploring how and where students tend to develop profession skills and assessments have been established to measure the level of professional skills. However, without a means to measure whether students are getting sufficient opportunities for development, it is hard for educators and engineering programs to determine whether or where scaffolding are needed. We developed an instrument to assess undergraduate engineering students’ opportunities for professional skill development. To increase content validity, we conducted 20 think-aloud interviews with students from a large Midwestern university. The aim of this WIP is two-fold. We present the preliminary results of the think-aloud interview to determine what changes need to be made to existing items and what emerging themes appear regarding to participants’ professional skill development opportunities. After thematic analysis of the interview transcripts, we revised 10 items by simplifying the grammar or altering certain words that tend to confuse participants or carry negative connotations. We found that, compared to students who have only been involved in class projects, those with co-curricular experiences tend to report more opportunities in skills related to business management principles and problem-solving skills. Co-curricular activities were also the most referenced in building communication skills. Our next step will be piloting the instrument across multiple institutions and conducting validation analysis.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
