This WIP presentation is intended to share and gather feedback on the development of an observation protocol for K-12 integrated STEM instruction, the STEM-OP. Specifically, the STEM-OP is being developed for use in K-12 science and/or engineering settings where integrated STEM instruction takes place. While the importance of integrated STEM education is established through national policy documents, there remains disagreement on models and effective approaches for integrated STEM instruction. Our broad definition of integrated STEM includes the use of two or more STEM disciplines to solve a real-world problem or design challenge that supports student development of 21st century skills. This issue is confounded by the lack of observation protocols sensitive to integrated STEM teaching and learning that can be used to inform research of the effectiveness of new models and strategies. Existing instruments most commonly used by researchers, such as the Reformed Teaching Observation Protocol (RTOP), were designed prior to the development of the Next Generation Science Standards and the integration of engineering into science standards. These instruments were also designed for use in reform-based science classrooms, not engineering or integrated STEM learning environments. While engineering-focused observation protocols do exist for K-12 classrooms, they do not evaluate beyond an engineering focus, making them limited tools to evaluate integrated STEM instruction. In order to facilitate the implementation of integrated STEM in K-12 classrooms and the development of the nascent integrated STEM education literature, our research team is developing a new integrated STEM observation protocol for use in K-12 science and engineering classrooms. This valid and reliable instrument will be designed for use in a variety of educational contexts and by different education stakeholders to increase the quality of K-12 STEM education. At the end of this project, the STEM-OP will be made available through an online platform that will include an embedded training program to facilitate its broad use.
In the first year of this four-year project, we are working on the initial development of the STEM-OP through video analysis and exploratory factor analysis. We are utilizing existing classroom video from a previous project with approximately 2,000 unique classroom videos representing a variety of grade levels (4-9), science content (life, earth, and physical science), engineering design challenges, and school demographics (urban, suburban). The development of the STEM-OP is guided by published frameworks that focus on providing quality K-12 integrated STEM and engineering education, such as the Framework for Quality K-12 Engineering Education.
Our anticipated results at the time the ASEE meeting will include a review of our item development process and finalized items included on the draft STEM-OP. Additionally, we anticipate being able to share findings from the exploratory factor analysis (EFA) on our video-coded data, which will identify distinct instructional dimensions responsible for integrated STEM instruction. We value the opportunity to gather feedback from the engineering education community as the integration of engineering design and practices is integral to quality integrated STEM instruction.
more »
« less
Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability, Validity, and Feasibility
Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient capture of classroom artifacts in multimedia formats using mobile devices. We assess the psychometric properties of measures of quality instruction in middle school science classrooms derived from the contents of portfolios collected using this novel tool—with instruction operationalized through dimensions aligned to the Next Generation Science Standards. Results reflect low rater error and adequate reliability for several dimensions, a dominant underlying factor, and significant relations to some relevant concurrent indicators. Although no relation was found to student standardized test scores or course grades, portfolio ratings did relate to student self-efficacy perceptions and enjoyment of science. We examine factors influencing measurement error, and consider the broader implications of the results for assessing the validity of portfolio score interpretations, and the feasibility and potential value of this type of tool for summative and formative uses, in the context of large-scale instructional improvement efforts.
more » « less- NSF-PAR ID:
- 10362187
- Publisher / Repository:
- SAGE Publications
- Date Published:
- Journal Name:
- Educational and Psychological Measurement
- Volume:
- 82
- Issue:
- 6
- ISSN:
- 0013-1644
- Page Range / eLocation ID:
- p. 1180-1202
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract This paper presents an implementation of Connected Spaces (CxS)—an ambient help seeking interface designed and developed for a project‐based computing classroom. We use actor network theory (ANT) to provide an underutilized posthumanist lens to understand the creation of collaborative connections in this Computational Action‐based implementation. Posthumanism offers an emerging and critical extension to sociocultural perspectives on understanding learning, by pushing us to decenter the human, and consider the active roles that human and non‐human entities play in learning environments by actively shaping each other. We analyse how students in this class adjusted their help‐seeking and collaborative habits following the introduction of CxS, a tool designed to foster (more inter‐group) collaboration. ANT proposes generalized symmetry—a principle of considering human, non‐human and more than human entities with equivalent and comparable agency, leading to describing phenomena as networks of actors in different evolving relationships with each other. Analysing collaborative interactions as fostered by CxS using an ANT approach supports design‐based research—an iterative design revision process highlighting understandings about design as well as learning—by providing a temporal and informative lens into the relationship between actors and tools within the environment. Our key findings include a framing of technologies in classrooms as bridging
agentic gaps between students and becoming actors engaging in different behaviours; learners enacting new agencies through technologies (for instance a more comfortable non‐intrusive help seeker), and the need for voicing and teachers to connect help networks in CxS equipped classrooms.Practitioner notes What is already known about this topic
Collaborative learning is a valuable skill and practice; opportunities to mentor others are critical in empowering minoritized learners, especially in STEM and computing disciplines.
School norms solidify a power and expertise hierarchy between teachers and learners and fail to productively support learners in learning from each other.
Additionally, lack of awareness about peers' knowledge is a common hindrance in students knowing who to ask for help and how.
What this paper adds
An example of a designed interface called Connected Spaces with potential to foster more inter‐student collaboration, especially outside of mandated within‐group collaboration—in the form of cross‐group help seeking and help giving.
A design based research study using actor network theory highlighting the limitations of Connected Spaces in sparking notable behaviour change among students by itself but being retooled as a teacher support tool in enabling cross‐group collaborations.
Presenting conceptions of collaboration through technologies as bridging agentic gaps and acting with new agencies in performing help‐seeking related actions.
Provoking the idea of testing emerging technologies in classrooms along with sharing our analyses and reflections with the classroom as a key idea in computing education—surfacing the gap between designed intentions and the different kinds of extra social work needed in the on‐ground success of different technologies.
Implications for practice and/or policy
Designers and researchers should create and test more interfaces alongside teachers across different classrooms and contexts aimed at supporting different kinds of voluntary collaborative interactions.
Curricula, standards and school practices should further center providing students with opportunities to engage as mentors and build communities of learning across disciplines to empower minoritized students.
Researchers engaging in design based research should consider using more posthumanist lenses to examine educational technologies and how they affect change in learning environments.
-
Instructional practices in secondary science: How teachers achieve local and standards‐based successmore » « less
Abstract This article reports on analyses of the instructional practices of six middle‐ and high‐school science teachers in the United States who participated in a research‐practice partnership that aims to support reform science education goals at scale. All six teachers were well qualified, experienced, and locally successful—respected by students, parents, colleagues, and administrators—but they differed in their success in supporting students' three‐dimensional learning. Our goal is to understand how the teachers' instructional practices contributed to their similarities in achieving local success and to differences in enabling students' learning, and to consider the implications of these findings for research‐practice partnerships. Data sources included classroom videos supplemented by interviews with teachers and focus students and examples of student work. We also compared students' learning gains by teacher using pre–post assessments that elicited three‐dimensional performances. Analyses of classroom videos showed how all six teachers achieved local success—they led effectively managed classrooms, covered the curriculum by teaching almost all unit activities, and assessed students' work in fair and efficient ways. There were important differences, however, in how teachers engaged students in science practices. Teachers in classrooms where students achieved lower learning gains followed a pattern of practice we describe as
activity‐based teaching , in which students completed investigations and hands‐on activities with few opportunities for sensemaking discussions or three‐dimensional science performances. Teachers whose students achieved higher learning gains combined the social stability characteristic of local classroom success with more demanding instructional practices associated withscientific sensemaking andcognitive apprenticeship . We conclude with a discussion of implications for research‐practice partnerships, highlighting how partnerships need to support all teachers in achieving both local and standards‐based success. -
null (Ed.)Teacher education is facing challenges given the recent incorporation of engineering practices and core ideas into the Next Generation Science Standards and state standards of learning. To help teachers meet these standards in their future classrooms, education courses for preservice teachers [PSTs] must provide opportunities to increase science and engineering knowledge, and the associated pedagogies. To address this need, Ed+gineering, an NSF-funded multidisciplinary service-learning project, was implemented to study ways in which PSTs are prepared to meet this challenge. This study provides the models and supporting data for four unique methods of infusion of engineering skills and practices into an elementary science methods course. The four models differ in mode of course delivery, integration of a group project (with or without partnering undergraduate engineering students), and final product (e.g., no product, video, interactive presentation, live lesson delivery). In three of the models, teams of 4-6 undergraduates collaborated to design and deliver (when applicable) lessons for elementary students. This multiple semester, mixed-methods research study, explored the ways in which four unique instructional models, with varied levels of engineering instruction enhancement, influenced PSTs’ science knowledge and pedagogical understanding. Both quantitative (e.g., science content knowledge assessment) and qualitative (e.g., student written reflections) data were used to assess science knowledge gains and pedagogical understanding. Findings suggest that the PSTs learned science content and were often able to explain particular science/ engineering concepts following the interventions. PSTs in more enhanced levels of intervention also shared ways in which their lessons reflected their students’ cultures through culturally responsive pedagogical strategies and how important engineering integration is to the elementary classroom, particularly through hands-on, inquiry-based instruction.more » « less
-
more » « less
Abstract Teachers play a critical role in successfully implementing science education reforms in the United States to provide high‐quality science learning opportunities to all students. However, the differentiated ways in which teachers make decisions about their science teaching are not well understood. This study takes a person‐centered approach by applying latent profile analysis to examine how cognitive (pedagogical content knowledge) and motivational (instructional goal orientations, self‐efficacy beliefs, and reform values) characteristics combine to form science teacher profiles in middle school. Predictors of profile membership (bachelor's degree, school %FRL) and both teacher (science instructional practices) and student (science achievement, engagement, and self‐efficacy) outcomes related to the teacher profiles were also examined. Five science teacher profiles were identified (severely discouraged but reform oriented, discouraged but reform oriented, conventional, confident and mastery oriented, and confident with multiple goal approaches) that represented unique configurations of cognitive and motivation characteristics. Additionally, findings showed that the teacher profiles were significantly related to three dimensions of science instructional practice including communication, discourse, and reasoning. Finally, the teacher profiles were significantly related to student science achievement and motivational outcomes. Implications for differentiated approaches to teacher professional learning and supports for science instruction are discussed.
