An official website of the United States government
This research investigates students’ argumentation quality in engineering design thinking. We implemented Learning by Evaluating (LbE) using Adaptive Comparative Judgment (ACJ), where students assess pairs of items to determine the superior one. In ACJ, students provided rationales for their critiques, explaining their selections. Fifteen students participated in an LbE exercise before starting their backpack design projects, critically evaluating multiple backpack designs and producing 145 comments. Writing comments required students to discern and justify the superior design, fostering informed judgment and articulation of their reasoning. The study used the Claim, Evidence, and Reasoning (CER) framework, adapted for engineering design thinking, to analyse these critiques. The framework emphasized three aspects: Empathy (understanding user needs), Ideation (deriving design inspiration), and Insight (gaining valuable understanding from evaluated designs). We employed both deductive and inductive content analysis to evaluate the argumentation quality in students’ critiques. High-quality argumentation was identified based on six codes: user-focused empathy, design inspirations, logical rationalizations, multi-criteria evaluations, aesthetic considerations, and cultural awareness. Poor-quality argumentation lacked these elements and was characterized by vagueness, uncertainty, brevity, inappropriateness, irrelevance, gender bias, and cultural stereotyping. By identifying critical elements of effective argumentation and common challenges students may face, this study aims to enhance argumentation skills in engineering design thinking at the secondary education level. These insights are intended to help educators prepare students for insightful and successful argumentation in engineering design projects.
more »
« less
A thematic analysis of high school students’ scientific argumentation of what constitutes a ‘better’ engineering design journal
Abstract The current study explores the quality of students’ argumentation within the context of Adaptive Comparative Judgment (ACJ) and Learning by Evaluation (LbE), focusing on the Claim-Evidence-Reasoning (CER) framework. The aim is to understand what students consider essential for superior engineering design journals and why, particularly examining evidence and reasoning components. Thirty-five students from four high schools participated in LbE, justifying their preferences for selected options. These schools were part of a broader five-school project, though one did not conduct the relevant session and was excluded from the study. Utilizing the CER framework, the study analyzed the structure of scientific argumentation, supplemented by thematic analysis to elucidate students' reasoning. Three response models emerged: Claim-Evidence (CE), Claim-Reasoning (CR), and CER. CE responses lacked reasoning, while CR responses lacked evidence. Students favored design portfolios with visual aids, detailed content, documentation of design failures, and clearly stated challenges. For reasoning, students highlighted the value of clear explanations of the design process, facilitation of group and individual work, idea generation, and instructional clarity. The study underscores the importance of teacher-led scaffolding to help students articulate comprehensive claims and suggests structured group discussions and modeling as effective supports.
more »
« less
- Award ID(s):
- 2101235
- PAR ID:
- 10588648
- Publisher / Repository:
- Springer Science + Business Media
- Date Published:
- Journal Name:
- International Journal of Technology and Design Education
- ISSN:
- 0957-7572
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The Next Generation Science Standards [1] recognized evidence-based argumentation as one of the essential skills for students to develop throughout their science and engineering education. Argumentation focuses students on the need for quality evidence, which helps to develop their deep understanding of content [2]. Argumentation has been studied extensively, both in mathematics and science education but also to some extent in engineering education (see for example [3], [4], [5], [6]). After a thorough search of the literature, we found few studies that have considered how teachers support collective argumentation during engineering learning activities. The purpose of this program of research was to support teachers in viewing argumentation as an important way to promote critical thinking and to provide teachers with tools to implement argumentation in their lessons integrating coding into science, technology, engineering, and mathematics (which we refer to as integrative STEM). We applied a framework developed for secondary mathematics [7] to understand how teachers support collective argumentation in integrative STEM lessons. This framework used Toulmin’s [8] conceptualization of argumentation, which includes three core components of arguments: a claim (or hypothesis) that is based on data (or evidence) accompanied by a warrant (or reasoning) that relates the data to the claim [9], [8]. To adapt the framework, video data were coded using previously established methods for analyzing argumentation [7]. In this paper, we consider how the framework can be applied to an elementary school teacher’s classroom interactions and present examples of how the teacher implements various questioning strategies to facilitate more productive argumentation and deeper student engagement. We aim to understand the nature of the teacher’s support for argumentation—contributions and actions from the teacher that prompt or respond to parts of arguments. In particular, we look at examples of how the teacher supports students to move beyond unstructured tinkering (e.g., trial-and-error) to think logically about coding and develop reasoning for the choices that they make in programming. We also look at the components of arguments that students provide, with and without teacher support. Through the use of the framework, we are able to articulate important aspects of collective argumentation that would otherwise be in the background. The framework gives both eyes to see and language to describe how teachers support collective argumentation in integrative STEM classrooms.more » « less
-
Engineering Design (ED) challenges are increasingly used as a context to learn science. Research shows that there is a need for strategies that facilitate learners to identify, apply, and reflect on ways scientific principles can inform creation and evaluation of ED solutions. We investigate the use of contrasting cases and argumentation scaffolds to facilitate use of evidence-based reasoning in a CAD supported ED tasks. Elementary education majors in a physics course analyzed solutions to an ED problem in two conditions: 1) identify similarities and differences, 2) evaluate and produce an argument for a “good” design solution. We found that the argumentation condition used scientific evidence-based reasoning significantly more frequently in their responses than the control. Results indicate that the contrasting cases with argumentation scaffolds shows promise in facilitating students’ use of evidence-based reasoning in their ED tasks.more » « less
-
Classroom research has demonstrated the capacity for significantly influencing student learning by engaging students in evaluation of previously submitted work as an intentional priming exercise for learning; we call this experience Learning by Evaluating (LbE). Expanding on current LbE research, we set forth to investigate the impact on student learning by intentionally differing the quality of examples evaluated by the students using adaptive comparative judgement. In this research, university design students (N = 468 students) were randomly assigned to one of three treatment groups; while each group evaluated previously collected student work as an LbE priming activity, the work evaluated by each group differed in quality. Using a three-group experimental design, one group of students only evaluated high quality examples, the second only evaluated low quality examples, and the third group of students evaluated a set of mixed-quality examples of the assignment they were about to work on. Following these LbE priming evaluations, students completed the assigned work and then their projects were evaluated to determine if there was a difference between student performance by treatment condition. Additional qualitative analysis was completed on student LbE rationales to explore similarities and differences in student cognitive judgments based on intervention grouping. No significant difference was found between the groups in terms of achievement, but several differences in group judgement approach were identified and future areas needing investigation were highlighted.more » « less
-
Leung, Carson (Ed.)Examining the effectiveness of machine learning techniques in analyzing engineering students’ decision-making processes through topic modeling during simulation-based design tasks is crucial for advancing educational methods and tools. Thus, this study presents a comparative analysis of different supervised and unsupervised machine learning techniques for topic modeling, along with human validation. Hence, this manuscript contributes by evaluating the effectiveness of these techniques in identifying nuanced topics within the argumentation framework and improving computational methods for assessing students’ abilities and performance levels based on their informed decisions. This study examined the decision-making processes of engineering students as they participated in a simulation-based design challenge. During this task, students were prompted to use an argumentation framework to articulate their claims, evidence, and reasoning, by recording their informed design decisions in a design journal. This study combined qualitative and computational methods to analyze the students’ design journals and ensured the accuracy of the findings through the researchers’ review and interpretations of the results. Different machine learning models, including random forest, SVM, and K-nearest neighbors (KNNs), were tested for multilabel regression, using preprocessing techniques such as TF-IDF, GloVe, and BERT embeddings. Additionally, hyperparameter optimization and model interpretability were explored, along with models like RNNs with LSTM, XGBoost, and LightGBM. The results demonstrate that both supervised and unsupervised machine learning models effectively identified nuanced topics within the argumentation framework used during the design challenge of designing a zero-energy home for a Midwestern city using a CAD/CAE simulation platform. Notably, XGBoost exhibited superior predictive accuracy in estimating topic proportions, highlighting its potential for broader application in engineering education.more » « less
