Constructed response items can both measure the coherence of student ideas and serve as reflective experiences to strengthen instruction. We report on new automated scoring technologies that can reduce the cost and complexity of scoring constructed‐response items. This study explored the accuracy of c‐rater‐ML, an automated scoring engine developed by Educational Testing Service, for scoring eight science inquiry items that require students to use evidence to explain complex phenomena. Automated scoring showed satisfactory agreement with human scoring for all test takers as well as specific subgroups. These findings suggest that c‐rater‐ML offers a promising solution to scoring constructed‐response science items and has the potential to increase the use of these items in both instruction and assessment. © 2015 Wiley Periodicals, Inc. J Res Sci Teach 53: 215–233, 2016.
Advances in automated scoring technologies have the potential to support student learning during inquiry instruction by providing timely and adaptive guidance on individual students’ responses. To identify which forms of automated guidance can be beneficial for inquiry learning, we compared reflective guidance to directive guidance for student‐generated concept diagrams in web‐based inquiry instruction. Eleven intact classes were randomly assigned to either a reflective guidance or a directive guidance condition. After creating a concept diagram showing energy flow in life science during the inquiry instruction, the directive group was told specific ways to improve their diagram, while the reflective group was told to revisit a relevant visualization step to locate useful information. The results from the concept diagrams, as well as individual tests, show that both forms of automated guidance helped students add target energy concepts, but reflective guidance was significantly more effective than directive guidance in improving students’ coherent understanding of how energy flows in life science. Analyses of log data revealed that the reflective group was more likely to revisit the visualization step as suggested in the guidance, which significantly enhanced student learning. Detailed analyses suggest that revisiting relevant materials to find useful information challenged students to identify gaps in their understanding and distinguish among multiple ideas. This study shows the value of designing reflective automated guidance for helping students engage in evidence‐gathering practices and enhance their understanding of scientific concepts. The findings suggest promising directions for the design of automated adaptive guidance to support complex science learning. © 2016 Wiley Periodicals, Inc. J Res Sci Teach 53:1003–1035, 2016
- NSF-PAR ID:
- 10240033
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Journal of Research in Science Teaching
- Volume:
- 53
- Issue:
- 7
- ISSN:
- 0022-4308
- Format(s):
- Medium: X Size: p. 1003-1035
- Size(s):
- p. 1003-1035
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
Tanner, Kimberly (Ed.)Two foundational concepts in biology education are 1) offspring are not identical to their parents, and 2) organisms undergo changes throughout their lives. These concepts are included in both international and U.S. curricular standards. Research in psychology has shown that children often have difficulty understanding these concepts, as they are inconsistent with their intuitive theories of the biological world. Additionally, prior research suggests that diagrams are commonly used in instruction and that their features influence student learning. Given this prior work, we explored the characteristics of life cycle diagrams and discuss possible implications for student learning. We examined 75 life cycle diagrams from books, including five biology or general science textbooks and 25 specialized trade books focusing on biology for children. We also examined 633 life cycle diagrams from a publicly available online database of science diagrams. Most diagrams failed to show any within-species variability. Additionally, many diagrams had perceptually rich backgrounds, which prior research suggests might hinder learning. We discuss how the design characteristics of diagrams may reinforce students’ intuitive theories of biology, which might make it difficult for students to understand key biological concepts in the future.more » « less
-
more » « less
Abstract Guiding teachers to customize curriculum has shown to improve science instruction when guided effectively. We explore how teachers use student data to customize a web-based science unit on plate tectonics. We study the implications for teacher learning along with the impact on student self-directed learning. During a professional development workshop, four 7th grade teachers reviewed logs of their students’ explanations and revisions. They used a curriculum visualization tool that revealed the pedagogy behind the unit to plan their customizations. To promote self-directed learning, the teachers decided to customize the guidance for explanation revision by giving students a choice among guidance options. They took advantage of the web-based unit to randomly assign students (
N = 479) to either a guidance Choice or a no-choice condition. We analyzed logged student explanation revisions on embedded and pre-test/post-test assessments and teacher and student written reflections and interviews. Students in the guidance Choice condition reported that the guidance was more useful than those in the no-choice condition and made more progress on their revisions. Teachers valued the opportunity to review student work, use the visualization tool to align their customization with the knowledge integration pedagogy, and investigate the choice option empirically. These findings suggest that the teachers’ decision to offer choice among guidance options promoted aspects of self-directed learning. -
This complete evidence-based practice paper discusses the strategies and results of an introduction to mechanics course, designed to prepare students for introductory-level physics and other fundamental courses in engineering, such as statics, strength of materials, and dynamics. The course was developed to address historically high failure (DFW) rates in the physics courses and is part of a set of interventions implemented to support student success in a college of engineering and computer science. The course focuses on providing in-depth understanding of Newton’s Laws of motion, free-body diagrams, and linear and projectile motion. Because it focuses on a limited number of competencies, it is possible to spend more time on inquiry-based activities and in-class discussions. The course framework was designed considering the Ebbinghaus’ Forgetting Curve, to provide students with learning opportunities in 6-day cycles: (i) day 1: a pre-class learning activity (reading or video) and a quiz; (ii) day 2: in-class Kahoot low-stakes quiz with discussion, a short lecture with embedded time for problem-solving and discussion, and in-class activities (labs, group projects); (iii) day 4: homework due two days after the class; (iv) day 6: homework self-reflection (autopsy based on provided solutions) two days after homework is due. The assessment of course performance is based on the well-characterized force concept inventory (FCI) exam that is administered before the intro to mechanics course and both before and after the Physics I course; and on student performance (grades) in Physics and Statics courses. Results from the FCI pre-test show that students who took the introduction to mechanics course (treatment group) started the physics course with a much better understanding of force concepts than other students in the course. The FCI post-test shows better normalized gain for the treatment group, compared to other students, which is also aligned with student performance in the course. Additionally, student performance is significantly better in statics, with 25% DWF rate compared to 50% for the other students. In summary, the framework of the course, which focuses on providing students with in-depth understanding of force concepts, has led to better learning and performance in Physics I, but importantly it has also helped students achieve better performance in the Statics course, the first fundamental course in civil and mechanical engineering programs.more » « less
-
more » « less
Abstract Energy is a central concept in science in every discipline and also an essential player in many of the issues facing people everywhere on the globe. However, studies have shown that by the end of K‐12 schooling, most students do not reach the level of understanding required to be able to use energy to make sense of a wide range of phenomena. Many researchers have questioned whether the conceptual foundations of traditional approaches to energy instruction may be responsible for students' difficulties. In response to these concerns, we developed and tested a novel approach to middle school physical science energy instruction that was informed by the recommendations of the Framework for K‐12 Science Education (National Research Council, 2012a) and the Next Generation Science Standards (NGSS) (NGSS Lead States, 2013). This new approach differs substantially from more traditional approaches to energy instruction in that it does not require energy forms and it emphasizes connections between energy, systems, and fields that mediate interaction‐at‐a‐distance. We investigated student learning during this novel approach and contrasted it with student learning within a comparable unit based on a more traditional approach to energy instruction. Our findings indicate that students who learned in the new approach outperformed students who learned in the traditional approach in every quantitative and qualitative aspect considered in this study, irrespective of their prior knowledge of energy. They developed more parsimonious knowledge networks in relation to energy that focused primarily around the concept of energy transfer. This study warrants further investigation into the value of this new approach to energy instruction in both middle and high school.
