An official website of the United States government
This work-in-progress paper describes a collaborative effort between engineering education and machine learning researchers to automate analysis of written responses to conceptually challenging questions in mechanics. These qualitative questions are often used in large STEM classes to support active learning pedagogies; they require minimum calculations and focus on the application of underlying physical phenomena to various situations. Active learning pedagogies using this type of questions has been demonstrated to increase student achievement (Freeman et al., 2014; Hake, 1998) and engagement (Deslauriers, et al., 2011) of all students (Haak et al., 2011). To emphasize reasoning and sense-making, we use the Concept Warehouse (Koretsky et al., 2014), an audience response system where students provide written justifications to concept questions. Written justifications better prepare students for discussions with peers and in the whole class and can also improve students’ answer choices (Koretsky et al., 2016a, 2016b). In addition to their use as a tool to foster learning, written explanations can also provide valuable information to concurrently assess that learning (Koretsky and Magana, 2019). However, in practice, there has been limited deployment of written justifications with concept questions, in part, because they provide a daunting amount of information for instructors to process and for researchers to analyze. In this study, we describe the initial evaluation of large pre-trained generative sequence-to-sequence language models (Raffel et al., 2019; Brown et al., 2020) to automate the laborious coding process of student written responses. Adaptation of machine learning algorithms in this context is challenging since each question targets specific concepts which elicit their own unique reasoning processes. This exploratory project seeks to utilize responses collected through the Concept Warehouse to identify viable strategies for adapting machine learning to support instructors and researchers in identifying salient aspects of student thinking and understanding with these conceptually challenging questions.
more »
« less
WIP: Using Machine Learning to Automate Coding of Student Explanations to Challenging Mechanics Concept Questions
This work-in-progress paper presents a joint effort by engineering education and machine learning researchers to develop automated methods for analyzing student responses to challenging conceptual questions in mechanics. These open-ended questions, which emphasize understanding of physical principles rather than calculations, are widely used in large STEM classes to support active learning strategies that have been shown to improve student outcomes. Despite their benefits, written justifications are not commonly used, largely because evaluating them is time-consuming for both instructors and researchers. This study explores the potential of large pre-trained generative sequence-to-sequence language models to streamline the analysis and coding of these student responses.
more »
« less
- Award ID(s):
- 2226601
- PAR ID:
- 10607923
- Publisher / Repository:
- ASEE 2022 Annual Conference
- Date Published:
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
his work-in-progress paper expands on a collaboration between engineering education researchers and machine learning researchers to automate the analysis of written responses to conceptually challenging questions in statics and dynamics courses (Authors, 2022). Using the Concept Warehouse (Koretsky et al., 2014), written justifications of ConcepTests (CTs) were gathered from statics and dynamics courses in a diverse set of two- and four-year institutions. Written justifications for CTs have been used to support active learning pedagogies which makes them important to investigate how students put together their problem-solving narratives of understanding. However, despite the large benefit that analysis of student written responses may provide to instructors and researchers, manual review of responses is cumbersome, limits analysis, and can be prone to human bias. In efforts to improve the analysis of student written responses, machine learning has been used in various educational contexts to analyze short and long texts (Burstein et al., 2020; Burstein et al., 2021). Natural Language Processing (NLP) uses transformer-based machine learning models (Brown et al., 2020; Raffel et al., 2019) which can be used through fine-tuning or in-context learning methods. NLP can be used to train algorithms that can automate the coding of written responses. Only a few studies for educational applications have leveraged transformer-based machine learning models further prompting an investigation into its use in STEM education. However, work in NLP has been criticized for heightening the possibility to perpetuate and even amplify harmful stereotypes and implicit biases (Chang et al., 2019; Mayfield et al., 2019). In this study, we detail the aim to use NLP for linguistic justice. Using methods like text summary, topic modeling, and text classification, we identify key aspects of student narratives of understanding in written responses to mechanics and statics CTs. Through this process, we seek to use machine learning to identify different ways students talk about a problem and their understanding at any point in their narrative formation process. Thus, we hope to help reduce human bias in the classroom and through technology by giving instructors and researchers a diverse set of narratives that include insight into their students’ histories, identities, and understanding. These can then be used towards connecting technological knowledge to students’ everyday lives.more » « less
-
This paper presents an experience report on using an interactive program visualization tool — Dynamic, Interactive Stack-Smashing Attack Visualization (DISSAV) — and a complementary active-learning exercise to teach stack smashing, a key software security attack. The visualization tool and active-learning exercise work synergistically to guide the student through challenging, abstract concepts in the advanced cybersecurity area. DISSAV and the exercise are deployed within the software security module of an undergraduate cybersecurity course that introduces a broad range of security topics. A study is designed that collects and evaluates student perceptions on the user interface of DISSAV and the effectiveness of the two resources in improving student learning and engagement. The study finds that over 80% of responses to user interface questions, 66% of responses to student learning questions and 64% of responses to student engagement questions are positive, suggesting that the resources improve student learning and engagement in general. The study does not find discernible patterns of difference in responses from students of different ages and varying levels of prior experience with stack smashing attacks, program visualization tools and C programming.more » « less
-
null (Ed.)Teacher responses to student mathematical thinking (SMT) matter because the way in which teachers respond affects student learning. Although studies have provided important insights into the nature of teacher responses, little is known about the extent to which these responses take into account the potential of the instance of SMT to support learning. This study investigated teachers’ responses to a common set of instances of SMT with varied potential to support students’ mathematical learning, as well as the productivity of such responses. To examine variations in responses in relation to the mathematical potential of the SMT to which they are responding, we coded teacher responses to instances of SMT in a scenario-based interview. We did so using a scheme that analyzes who interacts with the thinking (Actor), what they are given the opportunity to do in those interactions (Action), and how the teacher response relates to the actions and ideas in the contributed SMT (Recognition). The study found that teachers tended to direct responses to the student who had shared the thinking, use a small subset of actions, and explicitly incorporate students’ actions and ideas. To assess the productivity of teacher responses, we first theorized the alignment of different aspects of teacher responses with our vision of responsive teaching. We then used the data to analyze the extent to which specific aspects of teacher responses were more or less productive in particular circumstances. We discuss these circumstances and the implications of the findings for teachers, professional developers, and researchers.more » « less
-
Background: Teachers often rely on the use of open‐ended questions to assess students' conceptual understanding of assigned content. Particularly in the context of mathematics; teachers use these types of questions to gain insight into the processes and strategies adopted by students in solving mathematical problems beyond what is possible through more close‐ended problem types. While these types of problems are valuable to teachers, the variation in student responses to these questions makes it difficult, and time‐consuming, to evaluate and provide directed feedback. It is a well‐studied concept that feedback, both in terms of a numeric score but more importantly in the form of teacher‐authored comments, can help guide students as to how to improve, leading to increased learning. It is for this reason that teachers need better support not only for assessing students' work but also in providing meaningful and directed feedback to students. Objectives: In this paper, we seek to develop, evaluate, and examine machine learning models that support automated open response assessment and feedback. Methods: We build upon the prior research in the automatic assessment of student responses to open‐ended problems and introduce a novel approach that leverages student log data combined with machine learning and natural language processing methods. Utilizing sentence‐level semantic representations of student responses to open‐ended questions, we propose a collaborative filtering‐based approach to both predict student scores as well as recommend appropriate feedback messages for teachers to send to their students. Results and Conclusion: We find that our method outperforms previously published benchmarks across three different metrics for the task of predicting student performance. Through an error analysis, we identify several areas where future works maybe able to improve upon our approach.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- The DOI is not currently available.
