Search for: All records

This work-in-progress research paper describes a study of different categorical data coding procedures for machine learning(ML) in engineering education. Often left out of methodology sections, preprocessing steps in data analysis can have important ramifications on project outcomes. In this study, we applied three different coding schemes (i.e., scalar conversion, one-hot encoding, and binary) for the categorical variable of Race across three different ML models (i.e., Neural Network, Random Forest, and Naive Bayes classifiers) looking at the four standard measures of ML classification models (i.e., accuracy, precision, recall, and F1-score). Results showed that, in general, the coding scheme did not affect predictive outcomes as much as ML model type did. However, one-hot encoding – the strategy of transforming a categorical variable with k possible values to k binary nodes, a common practice in educational research – does not work well with a Naive Bayes classifier model. Our results indicate that such sensitivity studies at the beginning of ML modeling projects are necessary. Future work includes performing a full range of sensitivity studies on our complete, grant-funded project dataset that has been collected, and publishing our findings. 

This full-length research paper investigates two machine learning (ML) explanation methods applied to a model that predicts engineering student persistence/attrition. Engineering persistence is low in the United States, especially in the first year. It may be possible to increase persistence rates if we identify the students at risk of leaving and individualized malleable factors for intervention. This study is part of NSF Award #2335725, EVT & ML for Engineering Persistence that is working to streamline such a machine learning methodology. In this paper, explanations from SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) were compared for an engineering persistence dataset and predictive model. There was an overlap of 63% when using the default parameters. These preliminary results are promising for the reliable identification of malleable factors for intervention. 

STEM undergraduate instructors teaching remote courses often use traditional lecture-based instruction, despite evidence that active learning methods improve student engagement and learning outcomes. One simple way to use active learning online is to incorporate exploratory learning. In exploratory learning, students explore a novel activity (e.g., problem solving) before a lecture on the underlying concepts and procedures. This method has been shown to improve learning outcomes during in-person courses, without requiring the entire course to be restructured. The current study examined whether the benefits of exploratory learning extend to a remote undergraduate physics lesson, taught synchronously online. Undergraduate physics students (N = 78) completed a physics problem-solving activity either before instruction (explore-first condition) or after (instruct-first condition). Students then completed a learning assessment of the problem-solving procedures and underlying concepts. Despite lower accuracy on the learning activity, students in the explore-first condition demonstrated better understanding on the assessment, compared to students in the instruct-first condition. This finding suggests that exploratory learning can serve as productive failure in online courses, challenging students but improving learning, compared to the more widely-used lecture-then-practice method. 

This WIP paper presents new research on exploratory learning, an educational technique that reverses the order of standard lecture-based instruction techniques. In exploratory learning, students are presented with a novel activity first, followed by instruction. Exploratory learning has been observed to benefit student learning in foundational math and science courses such as calculus, physics, and statistics; however, it has yet to be applied to engineering topics such as programming. In two studies, we tested the effectiveness of exploratory learning in the programming unit of a first-year undergraduate engineering course. We designed a new activity to help students learn about different python error types, ensuring that it would be suitable for exploration. Then we implemented two different orders (the traditional instruct-first versus exploratory learning’s explore-first) across the six sections of the course. In Study 1 (N=406), we did not detect a difference between the instruct-first and explore-first conditions. In Study 2 (N=411), we added more scaffolding to the activity. Students who received the traditional order of instruction followed by the activity scored significantly higher on the assessment. These findings contradict the exploratory learning benefits typically shown, shedding light on potential boundary conditions to this effect. 

Abstract After being taught how to perform a new mathematical operation, students are often given several practice problems in a single set, such as a homework assignment or quiz (i.e., massed practice). An alternative approach is to distribute problems across multiple homeworks or quizzes, increasing the temporal interval between practice (i.e., spaced practice). Spaced practice has been shown to increase the long-term retention of various types of mathematics knowledge. Less clear is whether spacing decreases performance during practice, with some studies indicating that it does and others indicating it does not. To increase clarity, we tested whether spacing produces long-term retention gains, but short-term practice costs, in a calculus course. On practice quizzes, students worked problems on various learning objectives in either massed fashion (3 problems on a single quiz) or spaced fashion (3 problems across 3 quizzes). Spacing increased retention of learning objectives on an end-of-semester test but reduced performance on the practice quizzes. The reduction in practice performance was nuanced: Spacing reduced performance only on the first two quiz questions, leaving performance on the third question unaffected. We interpret these findings as evidence that spacing led to more protracted, but ultimately more robust, learning. We, therefore, conclude that spacing imposes a desirable form of difficulty in calculus learning. 

This study examined the difficulty introduced by spaced retrieval practice in Calculus I for undergraduate engineering students. Spaced retrieval practice is an instructional technique in which students engage in multiple recall exercises on the same topic with intermittent temporal delays in between. Spacing out retrieval practice increases the difficulty of the exercises, reducing student performance on them. However, empirical research indicates that spaced retrieval practice is associated with improvements in students’ long-term memory for the retrieved information. The short-term costs and long-term benefits of spaced retrieval practice is an example of desirable difficulty, when more difficult exercises during the early stages of learning result in longer-lasting memory [1]. With support from the National Science Foundation (NSF), we sought to address: Does spacing decrease performance on retrieval practice exercises in an engineering mathematics course? Results showed that student performance was significantly lower for questions in the spaced condition than questions in the massed condition, indicating that we successfully increased the difficulty of the questions by spacing them out over time. Future work will assess final quiz performance to determine whether spacing improved long-term course performance, i.e., whether the difficulty imposed by spacing was desirable.