skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: The Impact of Data Quantity and Source on the Quality of Data-Driven Hints for Programming.
In the domain of programming, intelligent tutoring systems increasingly employ data-driven methods to automate hint generation. Evaluations of these systems have largely focused on whether they can reliably provide hints for most students, and how much data is needed to do so, rather than how useful the resulting hints are to students. We present a method for evaluating the quality of data-driven hints and how their quality is impacted by the data used to generate them. Using two datasets, we investigate how the quantity of data and the source of data (whether it comes from students or experts) impact one hint generation algorithm. We find that with student training data, hint quality stops improving after 15–20 training solutions and can decrease with additional data. We also find that student data outperforms a single expert solution but that a comprehensive set of expert solutions generally performs best.  more » « less
Award ID(s):
1726550
PAR ID:
10065912
Author(s) / Creator(s):
Date Published:
Journal Name:
n: Penstein Rosé C. et al. (eds) Artificial Intelligence in Education. AIED 2018. Lecture Notes in Computer Science, vol 10947. Springer, Cham
Page Range / eLocation ID:
476-490
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, Journal of artificial intelligence in education)
    null (Ed.)
    Within intelligent tutoring systems, considerable research has in-vestigated hints, including how to generate data-driven hints, what hint con-tent to present, and when to provide hints for optimal learning outcomes. How-ever, less attention has been paid to how hints are presented. In this paper, we propose a new hint delivery mechanism called “Assertions” for providing unsolicited hints in a data-driven intelligent tutor. Assertions are partially-worked example steps designed to appear within a student workspace, and in the same format as student-derived steps, to show students a possible subgoal leading to the solution. We hypothesized that Assertions can help address the well-known hint avoidance problem. In systems that only provide hints upon request, hint avoidance results in students not receiving hints when they are needed. Our unsolicited Assertions do not seek to improve student help-seeking, but rather seek to ensure students receive the help they need. We contrast Assertions with Messages, text-based, unsolicited hints that appear after student inactivity. Our results show that Assertions significantly increase unsolicited hint usage compared to Messages. Further, they show a signifi-cant aptitude-treatment interaction between Assertions and prior proficiency, with Assertions leading students with low prior proficiency to generate shorter (more efficient) posttest solutions faster. We also present a clustering analysis that shows patterns of productive persistence among students with low prior knowledge when the tutor provides unsolicited help in the form of Assertions. Overall, this work provides encouraging evidence that hint presentation can significantly impact how students use them and using Assertions can be an effective way to address help avoidance. 
    more » « less
  2. (, In: Penstein Rosé C. et al. (eds) Artificial Intelligence in Education. AIED 2018. Lecture Notes in Computer Science, vol 10948. Springer, Cham)
    Within intelligent tutoring systems, hint policies are needed to determine when and how to give hints and what type of hint is most beneficial. In this study, we focus on discovering whether certain hint types influence problem solving behavior. We investigate the influence of two hint types (next-step hints and more abstract high-level hints) on students’ behavior in a college-level logic proof tutor, Deep Thought. The results suggest that hint types can affect student behavior, including hint usage, rule applications, and time in-tutor. 
    more » « less
  3. ; ; ; ; ; (, Journal of educational data mining)
    null (Ed.)
    Determining when and whether to provide personalized support is a well-known challenge called the assistance dilemma. A core problem in solving the assistance dilemma is the need to discover when students are unproductive so that the tutor can intervene. Such a task is particularly challenging for open-ended domains, even those that are well-structured with defined principles and goals. We present a set of datadriven methods to classify, predict, and prevent unproductive problem-solving steps in the well-structured open-ended domain of logic. This approach leverages and extends the Hint Factory, a set of methods that leverages prior student solution attempts to build data-driven intelligent tutors. We present a HelpNeed classification that uses prior student data to determine when students are likely to be unproductive and need help learning optimal problem-solving strategies. We present a controlled study to determine the impact of an Adaptive pedagogical policy that provides proactive hints at the start of each step based on the outcomes of our HelpNeed predictor: productive vs. unproductive. Our results show that the students in the Adaptive condition exhibited better training behaviors, with lower help avoidance, and higher help appropriateness (a higher chance of receiving help when it was likely to be needed), as measured using the HelpNeed classifier, when compared to the Control. Furthermore, the results show that the students who received Adaptive hints based on HelpNeed predictions during training significantly outperform their Control peers on the posttest, with the former producing shorter, more optimal solutions in less time. We conclude with suggestions on how these HelpNeed methods could be applied in other well-structured open-ended domains. 
    more » « less
  4. ; ; ; (, 2019 ASEE Annual Conference & Exposition)
    null (Ed.)
    The pedagogical approach of Zone of Proximal Development (ZPD) is based on the belief that effective learning occurs when students are challenged just beyond the level they can do on their own. An expert teacher looking over the shoulder of a student would give just the right amount of hints; too much hinting gives away the solution which deprives the student of the productive struggle that is needed for learning new concepts. Alternatively, no hinting may leave the student frustrated to the point where they give up. A key challenge with online learning is how to provide the right level of hints as if an expert teacher were there. This paper describes the evolution of hints for spatial visualization training using a mobile app. Students sketch orthographic and isometric sketches, which are automatically graded by the app. When a student draws an assignment incorrectly, they are provided with the option of a hint or peeking at the solution. This paper discusses the development of the app feedback and how it has impacted student behavior in using the app. In a first implementation, some students who excessively peeked at the solution without trying very hard on the assignments, did not significantly improve their spatial visualization ability as measured by the standardized PSVT:R test. To address the over-use of peeking, gamification was added that rewarded students to try on their own before looking at a hint or peek. In this paper, we look at a classroom trial that used a version of the spatial visualization mobile app with gamification. In general, gamification increased the post PSVT:r test scores. However, there was also a partial negative effect that and we see instances where the gamification lead to student frustration and waste of time because they avoided using hints to maximize their gamification points. We realized that the encompassing the knowledge of an expert teacher in providing hints just when needed, is difficult to implement in an algorithm. Specific examples are presented along with proposed improvements to the in-app hints. The final paper will include data comparing results of a class in January 2018 that used the original hints, with a class in January 2019 that will use the newer hints. 
    more » « less
  5. ; ; ; ; (, AAAI)
    The assistance dilemma is a well-recognized challenge to determine when and how to provide help during problem solving in intelligent tutoring systems. This dilemma is particularly challenging to address in domains such as logic proofs, where problems can be solved in a variety of ways. In this study, we investigate two data-driven techniques to address the when and how of the assistance dilemma, combining a model that predicts \textit{when} students need help learning efficient strategies, and hints that suggest \textit{what} subgoal to achieve. We conduct a study assessing the impact of the new pedagogical policy against a control policy without these adaptive components. We found empirical evidence which suggests that showing subgoals in training problems upon predictions of the model helped the students who needed it most and improved test performance when compared to their control peers. Our key findings include significantly fewer steps in posttest problem solutions for students with low prior proficiency and significantly reduced help avoidance for all students in training. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email