Flourishing in today's global society requires citizens that are both intelligent consumers and producers of scientific understanding. Indeed, the modern world is facing ever‐more complex problems that require innovative ways of thinking about, around, and with science. As numerous educational stakeholders have suggested, such skills and abilities are not innate and must, therefore, be taught (e.g., McNeill & Krajcik,
This paper describes HASbot, an automated text scoring and real‐time feedback system designed to support student revision of scientific arguments. Students submit open‐ended text responses to explain how their data support claims and how the limitations of their data affect the uncertainty of their explanations. HASbot automatically scores these text responses and returns the scores with feedback to students. Data were collected from 343 middle‐ and high‐school students taught by nine teachers across seven states in the United States. A mixed methods design was applied to investigate (a) how students’ utilization of HASbot impacted their development of uncertainty‐infused scientific arguments; (b) how students used feedback to revise their arguments, and (c) how the current design of HASbot supported or hindered students’ revisions. Paired sample
- NSF-PAR ID:
- 10088378
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Science Education
- Volume:
- 103
- Issue:
- 3
- ISSN:
- 0036-8326
- Page Range / eLocation ID:
- p. 590-622
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
Abstract Journal of Research in Science Teaching ,45 (1), 53–78. 2008). However, such instruction requires a fundamental shift in science pedagogy so as to foster knowledge and practices like deep, conceptual understanding, model‐based reasoning, and oral and written argumentation where scientific evidence is evaluated (National Research Council,Next Generation Science Standards: For States, by States , Washington, DC: The National Academies Press, 2013). The purpose of our quasi‐experimental study was to examine the effectiveness of Quality Talk Science, a professional development model and intervention, in fostering changes in teachers’ and students’ discourse practices as well as their conceptual understanding and scientific argumentation. Findings revealed treatment teachers’ and students’ discourse practices better reflected critical‐analytic thinking and argumentation at posttest relative to comparison classrooms. Similarly, at posttest treatment students produced stronger written scientific arguments than comparison students. Students’ growth in conceptual understanding was nonsignificant. These findings suggest discourse interventions such as Quality Talk Science can improve high‐school students’ ability to engage in scientific argumentation. -
Abstract Argumentation is fundamental to science education, both as a prominent feature of scientific reasoning and as an effective mode of learning—a perspective reflected in contemporary frameworks and standards. The successful implementation of argumentation in school science, however, requires a paradigm shift in science assessment from the measurement of knowledge and understanding to the measurement of performance and knowledge in use. Performance tasks requiring argumentation must capture the many ways students can construct and evaluate arguments in science, yet such tasks are both expensive and resource‐intensive to score. In this study we explore how machine learning text classification techniques can be applied to develop efficient, valid, and accurate constructed‐response measures of students' competency with written scientific argumentation that are aligned with a validated argumentation learning progression. Data come from 933 middle school students in the San Francisco Bay Area and are based on three sets of argumentation items in three different science contexts. The findings demonstrate that we have been able to develop computer scoring models that can achieve substantial to almost perfect agreement between human‐assigned and computer‐predicted scores. Model performance was slightly weaker for harder items targeting higher levels of the learning progression, largely due to the linguistic complexity of these responses and the sparsity of higher‐level responses in the training data set. Comparing the efficacy of different scoring approaches revealed that breaking down students' arguments into multiple components (e.g., the presence of an accurate claim or providing sufficient evidence), developing computer models for each component, and combining scores from these analytic components into a holistic score produced better results than holistic scoring approaches. However, this analytical approach was found to be differentially biased when scoring responses from English learners (EL) students as compared to responses from non‐EL students on some items. Differences in the severity between human and computer scores for EL between these approaches are explored, and potential sources of bias in automated scoring are discussed.
-
Abstract Argumentation, a key scientific practice presented in the
Framework for K-12 Science Education , requires students to construct and critique arguments, but timely evaluation of arguments in large-scale classrooms is challenging. Recent work has shown the potential of automated scoring systems for open response assessments, leveraging machine learning (ML) and artificial intelligence (AI) to aid the scoring of written arguments in complex assessments. Moreover, research has amplified that the features (i.e., complexity, diversity, and structure) of assessment construct are critical to ML scoring accuracy, yet how the assessment construct may be associated with machine scoring accuracy remains unknown. This study investigated how the features associated with the assessment construct of a scientific argumentation assessment item affected machine scoring performance. Specifically, we conceptualized the construct in three dimensions: complexity, diversity, and structure. We employed human experts to code characteristics of the assessment tasks and score middle school student responses to 17 argumentation tasks aligned to three levels of a validated learning progression of scientific argumentation. We randomly selected 361 responses to use as training sets to build machine-learning scoring models for each item. The scoring models yielded a range of agreements with human consensus scores, measured by Cohen’s kappa (mean = 0.60; range 0.38 − 0.89), indicating good to almost perfect performance. We found that higher levels ofComplexity andDiversity of the assessment task were associated with decreased model performance, similarly the relationship between levels ofStructure and model performance showed a somewhat negative linear trend. These findings highlight the importance of considering these construct characteristics when developing ML models for scoring assessments, particularly for higher complexity items and multidimensional assessments. -
Shaffer, Justin (Ed.)ABSTRACT Argumentation is vital in the development of scientific knowledge, and students who can argue from evidence and support their claims develop a deeper understanding of science. In this study, the Argument-Driven Inquiry instruction model was implemented in a two-semester sequence of introductory biology laboratories. Student’s scientific argumentation sessions were video recorded and analyzed using the Assessment of Scientific Argumentation in the Classroom observation protocol. This protocol separates argumentation into three subcategories: cognitive (how the group develops understanding), epistemic (how consistent the group’s process is with the culture of science), and social (how the group members interact with each other). We asked whether students are equally skilled in all subcategories of argumentation and how students’ argumentation skills differ based on lab exercise and course. Students scored significantly higher on the social than the cognitive and epistemic subcategories of argumentation. Total argumentation scores were significantly different between the two focal investigations in Biology Laboratory I but not between the two focal investigations in Biology Laboratory II. Therefore, student argumentation skills were not consistent across content; the design of the lab exercises and their implementation impacted the level of argumentation that occurred. These results will ultimately aid in the development and expansion of Argument-Driven Inquiry instructional models, with the goal of further enhancing students’ scientific argumentation skills and understanding of science.more » « less
-
Socioscientific issues (SSI) are often used to facilitate students’ engagement in multiple scientific practices such as decision-making and argumentation, both of which are goals of STEM literacy, science literacy, and integrated STEM education. Literature often emphasizes scientific argumentation over socioscientific argumentation, which involves considering social factors in addition to scientific frameworks. Analyzing students’ socioscientific arguments may reveal how students construct such arguments and evaluate pedagogical tools supporting these skills. In this study, we examined students’ socioscientific arguments regarding three SSI on pre- and post-assessments in the context of a course emphasizing SSI-based structured decision-making. We employed critical integrative argumentation (CIA) as a theoretical and analytical framework, which integrates arguments and counterarguments with stronger arguments characterized by identifying and refuting counterarguments. We hypothesized that engaging in structured decision-making, in which students integrate multidisciplinary perspectives and consider tradeoffs of various solutions based upon valued criteria, may facilitate students’ development of integrated socioscientific arguments. Findings suggest that students’ arguments vary among SSI contexts and may relate to students’ identities and perspectives regarding the SSI. We conclude that engaging in structured decision-making regarding personally relevant SSI may foster more integrated argumentation skills, which are critical to engaging in information-laden democratic societies.more » « less