skip to main content

Title: Extending the Hint Factory for the Assistance Dilemma: A Novel, Data-driven HelpNeed Predictor for Proactive Problem-solving Help
Determining when and whether to provide personalized support is a well-known challenge called the assistance dilemma. A core problem in solving the assistance dilemma is the need to discover when students are unproductive so that the tutor can intervene. Such a task is particularly challenging for open-ended domains, even those that are well-structured with defined principles and goals. We present a set of datadriven methods to classify, predict, and prevent unproductive problem-solving steps in the well-structured open-ended domain of logic. This approach leverages and extends the Hint Factory, a set of methods that leverages prior student solution attempts to build data-driven intelligent tutors. We present a HelpNeed classification that uses prior student data to determine when students are likely to be unproductive and need help learning optimal problem-solving strategies. We present a controlled study to determine the impact of an Adaptive pedagogical policy that provides proactive hints at the start of each step based on the outcomes of our HelpNeed predictor: productive vs. unproductive. Our results show that the students in the Adaptive condition exhibited better training behaviors, with lower help avoidance, and higher help appropriateness (a higher chance of receiving help when it was likely to be needed), as more » measured using the HelpNeed classifier, when compared to the Control. Furthermore, the results show that the students who received Adaptive hints based on HelpNeed predictions during training significantly outperform their Control peers on the posttest, with the former producing shorter, more optimal solutions in less time. We conclude with suggestions on how these HelpNeed methods could be applied in other well-structured open-ended domains. « less
Authors:
; ; ; ; ;
Award ID(s):
1726550
Publication Date:
NSF-PAR ID:
10277371
Journal Name:
Journal of educational data mining
Volume:
12
Issue:
4
Page Range or eLocation-ID:
24-65
ISSN:
2157-2100
Sponsoring Org:
National Science Foundation
More Like this
  1. Within intelligent tutoring systems, considerable research has in-vestigated hints, including how to generate data-driven hints, what hint con-tent to present, and when to provide hints for optimal learning outcomes. How-ever, less attention has been paid to how hints are presented. In this paper, we propose a new hint delivery mechanism called “Assertions” for providing unsolicited hints in a data-driven intelligent tutor. Assertions are partially-worked example steps designed to appear within a student workspace, and in the same format as student-derived steps, to show students a possible subgoal leading to the solution. We hypothesized that Assertions can help address the well-knownmore »hint avoidance problem. In systems that only provide hints upon request, hint avoidance results in students not receiving hints when they are needed. Our unsolicited Assertions do not seek to improve student help-seeking, but rather seek to ensure students receive the help they need. We contrast Assertions with Messages, text-based, unsolicited hints that appear after student inactivity. Our results show that Assertions significantly increase unsolicited hint usage compared to Messages. Further, they show a signifi-cant aptitude-treatment interaction between Assertions and prior proficiency, with Assertions leading students with low prior proficiency to generate shorter (more efficient) posttest solutions faster. We also present a clustering analysis that shows patterns of productive persistence among students with low prior knowledge when the tutor provides unsolicited help in the form of Assertions. Overall, this work provides encouraging evidence that hint presentation can significantly impact how students use them and using Assertions can be an effective way to address help avoidance.« less
  2. Solving open-ended complex problems is an essential part of being an engineer and one of the qualities needed in an engineering workplace. In order to help undergraduate engineering students develop such qualities and better prepare them for their future careers, this study is a preliminary effort to explore the problem solving approaches adopted by a student, faculty, and practicing engineer in civil engineering. As part of an ongoing NSF-funded study, this paper qualitatively investigates how three participants solve an ill-structured engineering problem. This study is guided by the following research question: What are the similarities and differences between a student,more »faculty, and practicing engineer in the approach to solve an ill-structured engineering problem? Verbal protocol analysis was used to answer this research question. Participants were asked to verbalize their response while they worked on the proposed problem. This paper includes a detailed analysis of the observed problem solving processes of the participants. Our preliminary findings indicate some distinct differences between the student, professor, and practicing engineer in their problem solving approaches. The student and practicing engineer used their prior knowledge to develop a solution, while the faculty did not make any connection to outside knowledge. It was also observed that the faculty and practicing engineer spent a great deal of time on feasibility and safety issues, whereas the student spent more time detailing the tool that would be used as their solution. Through additional data collection and analysis, we will better understand the similarities and differences between students, professionals, and faculty in terms of how they approach an ill-structured problem. This study will provide insights that will lead to the development of ways to better prepare engineering students to solve complex problems.« less
  3. Solving open-ended complex problems is an essential skill for part of being an engineer and a common activity in the one of the qualities needed in an engineering workplace. In order to help undergraduate engineering students develop such qualities and better prepare them for their future careers, this study is a preliminary effort to explore the problem solving approaches adopted by a student, faculty, and practicing engineer in civil engineering. As part of an ongoing NSF-funded study, this paper qualitatively investigates how three participants solve the following research question: What are the similarities and differences between a student, faculty, andmore »practicing engineer in the approach to solve an ill-structured engineering problem? Verbal protocol analysis was used to answer this research question. Participants were asked to verbalize their response while they worked on the proposed problem. This paper includes a detailed analysis of the observed problem-solving processes of the participants. Our preliminary findings indicate some distinct differences between the student, professor, and practicing engineer in their problem-solving approaches. The student and practicing engineer used their prior knowledge to develop a solution, while the faculty did not make any connection to outside knowledge. It was also observed that the faculty and practicing engineer spent a great deal of time on feasibility and safety issues, whereas the student spent more time detailing the tool that would be used as their solution. Through additional data collection and analysis, we will better understand the similarities and differences between students, professionals, and faculty in terms of how they approach an ill-structured problem. This study will provide insights that will lead to the development of ways to better prepare engineering students to solve complex problems.« less
  4. One of the main skills of engineers is to be able to solve problems. It is generally recognized that real-world engineering problems are inherently ill-structured in that they are complex, defined by non-engineering constraints, are missing information, and contain conflicting information. Therefore, it is very important to prepare future engineering students to be able to anticipate the occurrence of such problems, and to be prepared to solve them. However, most courses are taught by academic professors and lecturers whose focus is on didactic teaching of fundamental principles and code-based design approaches leading to predetermined “right” answers. Most classroom-taught methods tomore »solve well-structured problems and the methods needed to solve ill-structured problems are strikingly different. The focus of our current effort is to compare and contrast the problem solving approaches employed by students, academics and practicing professionals in an attempt to determine if students are developing the necessary skills to tackle ill-structured problems. To accomplish this, an ill-structured problem is developed, which will later be used to determine, based on analysis of oral and written responses of participants in semi-structured interviews, attributes of the gap between student, faculty, and professional approaches to ill-structured problem solving. Based on the results of this analysis, we will identify what pedagogical approaches may limit and help students’ abilities to develop fully-formed solutions to ill-structured problems. This project is currently ongoing. This work-in-progress paper will present the study and proposed methods. Based on feedback obtained at the conference from the broader research community, the studies will be refined. The current phase includes three parts, (1) problem formulation; (2) protocol development; and (3) pilot study. For (1), two different ill-structured problems were developed in the Civil Engineering domain. The problem difficulty assessment method was used to determine the appropriateness of each problem developed for this study. For (2), a protocol was developed in which participants will be asked to first solve a simple problem to become familiar with the interview format, then are given 30 minutes to solve the provided ill-structured problem, following a semi-structured interview format. Participants will be encouraged to speak out loud and also write down what they are thinking and their thought processes throughout the interview period. Both (1) and (2) will next be used for (3) the pilot study. The pilot study will include interviewing three students, three faculty members and three professional engineers. Each participant will complete both problems following the same protocol developed. Post-interview discussion will be held with the pilot study participants individually to inquire if there were any portions of the tasks that are unclearly worded or could be improved to clarify what was being asked. Based on these results the final problem will be chosen and refined.« less
  5. Research on non-cognitive factors has shown that persistence in the face of challenges plays an important role in learning. However, recent work on wheel-spinning, a type of unproductive persistence where students spend too much time struggling without achieving mastery of skills, show that not all persistence is uniformly beneficial for learning. For this reason, it becomes increasingly pertinent to identify the key differences between unproductive and productive persistence toward informing interventions in computer-based learning environments. In this study, we use a classification model to distinguish between productive persistence and wheel-spinning in ASSISTments, an online math learning platform. Our results indicatemore »that there are two types of students who wheel-spin: first, students who do not request any hints in at least one problem but request more than one bottom-out hint across any 8 problems in the problem set; second, students who never request two or more bottom out hints across any 8 problems, do not request any hints in at least one problem, but who engage in relatively short delays between solving problems of the same skill. These findings suggest that encouraging students to both engage in spaced practice and use bottom-out hints sparingly is likely helpful for reducing their wheel-spinning and improving learning. These findings also provide insight on when students are struggling and how to make students' persistence more productive.« less