skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Developing a measure to capture middle school students’ interpretive understanding of engineering design
This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 246) groups, wherein the former participated in a two-to-three-week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection.  more » « less
Award ID(s):
1813572
PAR ID:
10251674
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
2021 American Society for Engineering Education Annual Conference
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non- treatment (n = 246) groups, wherein the former participated in a two-to-three- week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann- Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 
    more » « less
  2. This article describes a professional development (PD) model, the CT-Integration Cycle, that supports teachers in learning to integrate computational thinking (CT) and computer science principles into their middle school science and STEM instruction. The PD model outlined here includes collaborative design (codesign; Voogt et al., 2015) of curricular units aligned with the Next Generation Science Standards (NGSS) that use programmable sensors. Specifically, teachers can develop or modify curricular materials to ensure a focus on coherent, student-driven instruction through the investigation of scientific phenomena that are relevant to students and integrate CT and sensor technology. Teachers can implement these storylines and collaboratively reflect on their instructional practices and student learning. Throughout this process, teachers may develop expertise in CT-integrated science instruction as they plan and use instructional practices aligned with the NGSS and foreground CT. This paper describes an examination of a group of five middle school teachers’ experiences during one iteration of the CT-Integration Cycle, including their learning, planning, implementation, and reflection on a unit they codesigned. Throughout their participation in the PD, the teachers expanded their capacity to engage deeply with CT practices and thoughtfully facilitated a CT-integrated unit with their students. 
    more » « less
  3. null (Ed.)
    This article describes a professional development (PD) model, the CT- Integration Cycle, that supports teachers in learning to integrate computational thinking (CT) and computer science principles into their middle school science and STEM instruction. The PD model outlined here includes collaborative design (codesign; Voogt et al., 2015) of curricular units aligned with the Next Generation Science Standards (NGSS) that use programmable sensors. Specifically, teachers can develop or modify curricular materials to ensure a focus on coherent, student-driven instruction through the investigation of scientific phenomena that are relevant to students and integrate CT and sensor technology. Teachers can implement these storylines and collaboratively reflect on their instructional practices and student learning. Throughout this process, teachers may develop expertise in CT-integrated science instruction as they plan and use instructional practices aligned with the NGSS and foreground CT. This paper describes an examination of a group of five middle school teachers’ experiences during one iteration of the CT- Integration Cycle, including their learning, planning, implementation, and reflection on a unit they codesigned. Throughout their participation in the PD, the teachers expanded their capacity to engage deeply with CT practices and thoughtfully facilitated a CT-integrated unit with their students. 
    more » « less
  4. Inquiry is featured prominently in the Next Generation Science Standards (NGSS) as a promising pedagogical approach. Building on current conceptions of inquiry, a mixed-methods research design was used to explore the effects of Project-Based Inquiry (PBI) Global on student science content knowledge, motivation, and perspectives related to inquiry in a cross-school collaboration. The data sources included pre-/post-tests on science content and student motivation (n = 75), transcripts from student focus groups (n = 26), and students’ multimodal learning products (n = 18 teams). The quantitative findings indicated School B students were more motivated by the project than School A students, which mirrored student performance. The student focus group findings generated three themes: constructing empathy, learning for impact, and navigating challenges. The discussion focuses on an integrated view of what students gained and did not gain from the PBI Global experience, including a nuanced explanation of how motivation and content knowledge may be influenced by student experiences and school contextual factors during PBI Global. Implications for instructional practice highlight how relationship building, mutual respect, and consensus making are essential components of constructing cross-school collaborations and the importance of integrating instructional frameworks with teachers and students. Future research will focus on investigating the effects of PBI Global on student learning in cross-school partnerships through experimental-designed studies, and the systemic and structural barriers to scaling cross-school inquiry-based learning. 
    more » « less
  5. We describe a professional development model that supports teachers to integrate computational thinking (CT) and computer science principles into middle school science and STEM classes. The model includes the collaborative design (co-design) (Voogt et al., 2015) of storylines or curricular units aligned with the Next Generation Science Standards (NGSS Lead States, 2013) that utilize programmable sensors such as those contained on the micro:bit. Teachers spend several workshops co-designing CT-integrated storylines and preparing to implement them with their own students. As part of this process, teachers develop or modify curricular materials to ensure a focus on coherent, student driven instruction through the investigation of scientific phenomena that are relevant to the students and utilize sensor technology. Teachers implement the storylines and meet to collaboratively reflect on their instructional practices as well as their students’ learning. Throughout this cyclical, multi-year process, teachers develop expertise in CT-integrated science instruction as they plan for and use instructional practices that align with three dimension science teaching and foreground computational thinking. Throughout the professional learning process, teachers alternate between wearing their “student hats” and their “teacher hats”, in order to maintain both a student and teacher perspective as they co-design and reflect on their implementation of CT-integrated units. This paper illustrates two teachers’ experiences of the professional development process over a two-year period, including their learning, planning, implementation, and reflection on two co-designed units. 
    more » « less