More Like this

1. WIP: Using Machine Learning to Automate Coding of Student Explanations to Challenging Mechanics Concept Questions

   Harpreet Auby; Namrata Shivagunde; Anna Rumshisky; Milo Koretsky (June 2022, ASEE Annual Conference proceedings)

   This work-in-progress paper describes a collaborative effort between engineering education and machine learning researchers to automate analysis of written responses to conceptually challenging questions in mechanics. These qualitative questions are often used in large STEM classes to support active learning pedagogies; they require minimum calculations and focus on the application of underlying physical phenomena to various situations. Active learning pedagogies using this type of questions has been demonstrated to increase student achievement (Freeman et al., 2014; Hake, 1998) and engagement (Deslauriers, et al., 2011) of all students (Haak et al., 2011). To emphasize reasoning and sense-making, we use the Concept Warehouse (Koretsky et al., 2014), an audience response system where students provide written justifications to concept questions. Written justifications better prepare students for discussions with peers and in the whole class and can also improve students’ answer choices (Koretsky et al., 2016a, 2016b). In addition to their use as a tool to foster learning, written explanations can also provide valuable information to concurrently assess that learning (Koretsky and Magana, 2019). However, in practice, there has been limited deployment of written justifications with concept questions, in part, because they provide a daunting amount of information for instructors to process and for researchers to analyze. In this study, we describe the initial evaluation of large pre-trained generative sequence-to-sequence language models (Raffel et al., 2019; Brown et al., 2020) to automate the laborious coding process of student written responses. Adaptation of machine learning algorithms in this context is challenging since each question targets specific concepts which elicit their own unique reasoning processes. This exploratory project seeks to utilize responses collected through the Concept Warehouse to identify viable strategies for adapting machine learning to support instructors and researchers in identifying salient aspects of student thinking and understanding with these conceptually challenging questions. 

   more » « less
2. Work in Progress: Using Machine Learning to Map Student Narratives of Understanding and Promoting Linguistic Justice

   Auby, H.; Koretsky, M. (June 2023, 2023 ASEE Annual Conference & Exposition)

   his work-in-progress paper expands on a collaboration between engineering education researchers and machine learning researchers to automate the analysis of written responses to conceptually challenging questions in statics and dynamics courses (Authors, 2022). Using the Concept Warehouse (Koretsky et al., 2014), written justifications of ConcepTests (CTs) were gathered from statics and dynamics courses in a diverse set of two- and four-year institutions. Written justifications for CTs have been used to support active learning pedagogies which makes them important to investigate how students put together their problem-solving narratives of understanding. However, despite the large benefit that analysis of student written responses may provide to instructors and researchers, manual review of responses is cumbersome, limits analysis, and can be prone to human bias. In efforts to improve the analysis of student written responses, machine learning has been used in various educational contexts to analyze short and long texts (Burstein et al., 2020; Burstein et al., 2021). Natural Language Processing (NLP) uses transformer-based machine learning models (Brown et al., 2020; Raffel et al., 2019) which can be used through fine-tuning or in-context learning methods. NLP can be used to train algorithms that can automate the coding of written responses. Only a few studies for educational applications have leveraged transformer-based machine learning models further prompting an investigation into its use in STEM education. However, work in NLP has been criticized for heightening the possibility to perpetuate and even amplify harmful stereotypes and implicit biases (Chang et al., 2019; Mayfield et al., 2019). In this study, we detail the aim to use NLP for linguistic justice. Using methods like text summary, topic modeling, and text classification, we identify key aspects of student narratives of understanding in written responses to mechanics and statics CTs. Through this process, we seek to use machine learning to identify different ways students talk about a problem and their understanding at any point in their narrative formation process. Thus, we hope to help reduce human bias in the classroom and through technology by giving instructors and researchers a diverse set of narratives that include insight into their students’ histories, identities, and understanding. These can then be used towards connecting technological knowledge to students’ everyday lives. 

   more » « less
3. Interactive Program Visualization to Teach Stack Smashing: An Experience Report

   https://doi.org/10.53735/cisse.v10i1.164  

   Ramaprasad, Harini; Sridhar, Meera; Akeyson, Erik (March 2023, Journal of The Colloquium for Information Systems Security Education)

   This paper presents an experience report on using an interactive program visualization tool — Dynamic, Interactive Stack-Smashing Attack Visualization (DISSAV) — and a complementary active-learning exercise to teach stack smashing, a key software security attack. The visualization tool and active-learning exercise work synergistically to guide the student through challenging, abstract concepts in the advanced cybersecurity area. DISSAV and the exercise are deployed within the software security module of an undergraduate cybersecurity course that introduces a broad range of security topics. A study is designed that collects and evaluates student perceptions on the user interface of DISSAV and the effectiveness of the two resources in improving student learning and engagement. The study finds that over 80% of responses to user interface questions, 66% of responses to student learning questions and 64% of responses to student engagement questions are positive, suggesting that the resources improve student learning and engagement in general. The study does not find discernible patterns of difference in responses from students of different ages and varying levels of prior experience with stack smashing attacks, program visualization tools and C programming. 

   more » « less
4. Teachers’ responses to instances of student mathematical thinking with varied potential to support student learning

   https://doi.org/10.1007/s13394-020-00334-x  

   Stockero, Shari L.; Van Zoest, Laura R.; Freeburn, Ben; Peterson, Blake E.; Leatham, Keith R. (January 2020, Mathematics Education Research Journal)

   null (Ed.)

   Teacher responses to student mathematical thinking (SMT) matter because the way in which teachers respond affects student learning. Although studies have provided important insights into the nature of teacher responses, little is known about the extent to which these responses take into account the potential of the instance of SMT to support learning. This study investigated teachers’ responses to a common set of instances of SMT with varied potential to support students’ mathematical learning, as well as the productivity of such responses. To examine variations in responses in relation to the mathematical potential of the SMT to which they are responding, we coded teacher responses to instances of SMT in a scenario-based interview. We did so using a scheme that analyzes who interacts with the thinking (Actor), what they are given the opportunity to do in those interactions (Action), and how the teacher response relates to the actions and ideas in the contributed SMT (Recognition). The study found that teachers tended to direct responses to the student who had shared the thinking, use a small subset of actions, and explicitly incorporate students’ actions and ideas. To assess the productivity of teacher responses, we first theorized the alignment of different aspects of teacher responses with our vision of responsive teaching. We then used the data to analyze the extent to which specific aspects of teacher responses were more or less productive in particular circumstances. We discuss these circumstances and the implications of the findings for teachers, professional developers, and researchers. 

   more » « less
5. Automated Scoring of Image-based responses to Open-ended mathematics question.

   Baral, S; Santhanam, A; Botelho, A; Gurung, A; Heffernan, N (July 2023, The Proceedings of the 16th International Conference on Educational Data Mining)

   Teachers often rely on the use of a range of open-ended problems to assess students’ understanding of mathematical concepts. Beyond traditional conceptions of student open- ended work, commonly in the form of textual short-answer or essay responses, the use of figures, tables, number lines, graphs, and pictographs are other examples of open-ended work common in mathematics. While recent developments in areas of natural language processing and machine learning have led to automated methods to score student open-ended work, these methods have largely been limited to textual an- swers. Several computer-based learning systems allow stu- dents to take pictures of hand-written work and include such images within their answers to open-ended questions. With that, however, there are few-to-no existing solutions that support the auto-scoring of student hand-written or drawn answers to questions. In this work, we build upon an ex- isting method for auto-scoring textual student answers and explore the use of OpenAI/CLIP, a deep learning embedding method designed to represent both images and text, as well as Optical Character Recognition (OCR) to improve model performance. We evaluate the performance of our method on a dataset of student open-responses that contains both text- and image-based responses, and find a reduction of model error in the presence of images when controlling for other answer-level features. 

   more » « less