Attention:The NSF Public Access Repository (PAR) system and access will be unavailable from 5:00 PM ET until 8:00 PM ET on Friday, September 11 due to maintenance. We apologize for the inconvenience.


Title: Developing a measure to capture middle school students’ interpretive understanding of engineering design
This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 246) groups, wherein the former participated in a two-to-three-week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection.  more » « less
Award ID(s):
1813572
PAR ID:
10251674
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
2021 American Society for Engineering Education Annual Conference
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non- treatment (n = 246) groups, wherein the former participated in a two-to-three- week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann- Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 
    more » « less
  2. This article describes a professional development (PD) model, the CT-Integration Cycle, that supports teachers in learning to integrate computational thinking (CT) and computer science principles into their middle school science and STEM instruction. The PD model outlined here includes collaborative design (codesign; Voogt et al., 2015) of curricular units aligned with the Next Generation Science Standards (NGSS) that use programmable sensors. Specifically, teachers can develop or modify curricular materials to ensure a focus on coherent, student-driven instruction through the investigation of scientific phenomena that are relevant to students and integrate CT and sensor technology. Teachers can implement these storylines and collaboratively reflect on their instructional practices and student learning. Throughout this process, teachers may develop expertise in CT-integrated science instruction as they plan and use instructional practices aligned with the NGSS and foreground CT. This paper describes an examination of a group of five middle school teachers’ experiences during one iteration of the CT-Integration Cycle, including their learning, planning, implementation, and reflection on a unit they codesigned. Throughout their participation in the PD, the teachers expanded their capacity to engage deeply with CT practices and thoughtfully facilitated a CT-integrated unit with their students. 
    more » « less
  3. null (Ed.)
    This article describes a professional development (PD) model, the CT- Integration Cycle, that supports teachers in learning to integrate computational thinking (CT) and computer science principles into their middle school science and STEM instruction. The PD model outlined here includes collaborative design (codesign; Voogt et al., 2015) of curricular units aligned with the Next Generation Science Standards (NGSS) that use programmable sensors. Specifically, teachers can develop or modify curricular materials to ensure a focus on coherent, student-driven instruction through the investigation of scientific phenomena that are relevant to students and integrate CT and sensor technology. Teachers can implement these storylines and collaboratively reflect on their instructional practices and student learning. Throughout this process, teachers may develop expertise in CT-integrated science instruction as they plan and use instructional practices aligned with the NGSS and foreground CT. This paper describes an examination of a group of five middle school teachers’ experiences during one iteration of the CT- Integration Cycle, including their learning, planning, implementation, and reflection on a unit they codesigned. Throughout their participation in the PD, the teachers expanded their capacity to engage deeply with CT practices and thoughtfully facilitated a CT-integrated unit with their students. 
    more » « less
  4. There is broad consensus among researchers and educators that locally contextualized, language-rich science and engineering-based instruction exploring scientific phenomena is essential to achieving access and rigor for all students, including multilingual learners (MLs), the fastest growing subset of the U.S. student population (NASEM, 2018; NCES, 2021). For this study, we are looking at what it means to teach language and content in an integrated manner to build science literacy in high school science classrooms that include ML students. As part of a 3-year, NSF-funded exploratory grant, this project aims to increase science literacy among MLs by implementing a model of professional learning (PL) for high school teachers in which they learn how to embed the Instructional Conversation (IC) pedagogy (Mellom, Hixon & Weber, 2019; Saunders and Goldenberg, 1999) within standards-aligned scientific and engineering practices (SEPs) (NGSS Lead States, 2013). Under this model, high school science teachers collaborate with high school ESOL teachers to co-develop and implement linguistically-sustaining instructional materials that provide MLs with intentionally scaffolded opportunities to use scientific dialogue as they collaborate to explain natural phenomena. This study examines how the PL experiences supported shifts in the teachers’ instructional practices to foster ML scientific literacy. Data includes: teacher reflection logs, teacher-generated artifacts (e.g. differentiated Task Cards, and multimodal, scaffolded lesson materials), rubric-based data (Luning & Wyatt, 2010), and field notes from classroom observations. The study uses thematic analysis (Braun & Clarke, 2006) and text analysis (Kuckartz, 2019) of these data to explore how the IC pedagogical strategies promote Teachers’ skills needed to enact three instructional shifts identified by the Next Generation Science Standards (NGSS) as particularly relevant to science teaching with MLs: asset-based science teaching with multilingual learners (MLs); integrated content and language instruction; collaboratively engaging MLs in Science and Engineering Practices (SEPs). 
    more » « less
  5. There is broad consensus among researchers and educators that locally-contextualized, language-rich, science-based instruction is essential to achieving access and rigor for all students, including multilingual learners (MLs), the fastest growing subset of the US student population (NASEM, 2018; NCES, 2021). Among the instructional shifts envisioned in the Next Generation Science Standards (NGSS, NGSS Lead States, 2013), three pose particularly acute and ongoing challenges to science teaching with MLs: 1) developing asset-oriented perspectives and instructional practices (Lee, 2020; Lyon et al., 2018); 2) integrating science and language instruction in mutually supportive ways (Lee et al., 2019; Buxton & Lee, 2023), and; 3) collaboratively engaging all students, including MLs, in science and engineering practices (SEPs) (NASEM, 2018; Grapin, 2019). As these shifts involve knowledge, beliefs, and practices largely absent from traditional science classrooms and teacher preparation curricula, teachers require innovative pedagogical support and professional learning (PL) opportunities to achieve them (NASEM, 2016; NRC, 2015). Crucially, such innovations necessitate greater collaboration between the science and ML education communities (Lee et al., 2019; Marshall, et al., 2021). As part of a 3-year, NSF-funded grant, this project aims to increase collaboration among high school science content and ESOL teachers by implementing a model of professional learning (PL) in which they learn how to embed the Instructional Conversation (IC) pedagogy (Mellom, Hixon & Weber, 2019; Saunders and Goldenberg, 1999) within standards-aligned scientific and engineering practices (SEPs) (NGSS Lead States, 2013). This study examines how participating teachers discuss content and language and collaborate across disciplinary divides to plan and implement linguistically-sustaining lessons in science classrooms. Data include: teacher reflection logs, transcripts of recorded face-to-face PL sessions, and transcripts of focus groups. The study uses thematic analysis (Braun & Clarke, 2006) to explore the supports teachers need to facilitate effective multidisciplinary collaboration for NGSS-aligned science teaching with MLs. 
    more » « less