skip to main content


This content will become publicly available on July 12, 2025

Title: Building Learner Activity Models From Log Data Using Sequence Mapping and Hidden Markov Models
Open-ended learning environments (OELEs) involve high learner agency in defining learning goals and multiple pathways to achieve those goals. These tasks involve learners transitioning through self-regulated learning (SRL) phases by actively setting goals, applying different strategies for those goals, and monitoring performance to update their strategies. However, because of the flexibility, how learners react to impasses and errors has a critical influence on their learning. An intelligent pedagogical agent (IPA) continuously modeling learner activities could help support learners in these environments. However, this continuous comprehension of behaviors and strategies is difficult in OELEs with evolving goals, ill-defined problem structures, and learning sequences. In this paper, we draw from the literature on SRL phases and cognitive states to investigate the utility of two different methods, Sequence Mapping, and Hidden Markov Models, in building learner activity models from log data collected from a summer camp with 14 middle school girls in an open-design environment. We evaluate the effectiveness of these models separately, and combined, in identifying 7 states: Forethought, Engaged Concentration, Acting, Monitoring, Wheel Spinning, Mind Wandering, and Reflect and Repair. Lastly, we recommend dialogue intervention strategies for an IPA to support learning in OELEs.  more » « less
Award ID(s):
1935801
PAR ID:
10540995
Author(s) / Creator(s):
; ; ; ;
Editor(s):
Benjamin, Paaßen; Carrie, Demmans Epp
Publisher / Repository:
International Educational Data Mining Society
Date Published:
Format(s):
Medium: X
Right(s):
Creative Commons Attribution 4.0 International
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    Intelligent tutoring systems (ITSs) incorporate pedagogical agents (PAs) to scaffold learners' self‐regulated learning (SRL) via prompts and feedback to promote learners' monitoring and regulation of their cognitive, affective, metacognitive and motivational processes to achieve their (sub)goals. This study examines PAs' effectiveness in scaffolding and teaching SRL during learning with MetaTutor, an ITS on the human circulatory system. Undergraduates (N = 118) were randomly assigned to a condition:Control Condition(i.e. learners could only self‐initiate SRL strategies) andPrompt and Feedback Condition(i.e. PAs prompted learners to engage in SRL). Learners' log‐file data captured when strategies were used, the initiator of the strategy (i.e. learner and PA), and the relevance of instructional content pages in relation to learner subgoals. While results showed that PAs were effective scaffolders of SRL in which they prompted learners to engage in SRL strategies more when content was relevant towards their subgoals and as time on page and task increased, there were mixed findings about the effectiveness of PAs as teachers of SRL. Findings show how production rules guiding PA prompts can improve their scaffolding and teaching of SRL across the learning task – through contextualizing SRL strategies to the instructional content and in relation to the relevance of the content to learners' subgoals.Practitioner notesWhat is already known about this topic

    Most learners struggle to efficiently and effectively use self‐regulated learning (SRL) strategies to attain goals and subgoals.

    There is a need for SRL to be scaffolded for learners to manage multiple goals and subgoals while learning about complex STEM topics.

    Intelligent tutoring systems (ITSs) typically incorporate pedagogical agents (PAs) to prompt learners to engage in SRL strategy and provide feedback.

    There are mixed findings on the effectiveness of PAs in scaffolding learners' SRL.

    What this paper adds

    We consider PAs not only scaffolders but also teachers of SRL.

    Results showed that while PAs encouraged the use of SRL strategies when the content was relevant to subgoals, they did not discourage the use of SRL strategies when the content was not relevant.

    Results for this study were mixed in their support of PAs as teachers of SRL.

    Learners increasingly depended on PAs to prompt SRL strategies as time on task progressed.

    Implications for practice and/or policy

    PAs are effective scaffolders of SRL with more research needed to understand their role as teachers of SRL.

    PA scaffolding is more essential as time on task progresses.

    When deploying specific cognitive and metacognitive SRL strategies, the relevance of the content to learners' subgoals should be taken into account.

     
    more » « less
  2. Abstract Background

    Self-regulated learning (SRL) strategies can be domain specific. However, it remains unclear whether this specificity extends to different subtopics within a single subject domain. In this study, we collected data from 210 college students engaged in a computer-based learning environment to examine the heterogeneous manifestations of learning behaviors across four distinct subtopics in introductory statistics. Further, we explore how the time spent engaging in metacognitive strategies correlated with learning gain in those subtopics.

    Results

    By employing two different analytical approaches that combine data-driven learning analytics (i.e., sequential pattern mining in this case), and theory-informed methods (i.e., coherence analysis), we discovered significant variability in the frequency of learning patterns that are potentially associated with SRL-relevant strategies across four subtopics. In a subtopic related to calculations, engagement in coherent quizzes (i.e., a type of metacognitive strategy) was found to be significantly less related to learning gains compared to other subtopics. Additionally, we found that students with different levels of prior knowledge and learning gains demonstrated varying degrees of engagement in learning patterns in an SRL context.

    Conclusion

    The findings imply that the use—and the effectiveness—of learning patterns that are potentially associated with SRL-relevant strategies varies not only across contexts and domains, but even across different subtopics within a single subject. This underscores the importance of personalized, context-aware SRL training interventions in computer-based learning environments, which could significantly enhance learning outcomes by addressing the heterogeneous relationships between SRL activities and outcomes. Further, we suggest theoretical implications of subtopic-specific heterogeneity within the context of various SRL models. Understanding SRL heterogeneity enhances these theories, offering more nuanced insights into learners’ metacognitive strategies across different subtopics.

     
    more » « less
  3. Researchers and educators have developed a variety of computer-based technologies intended to facilitate self-regulated learning (SRL), which refers to iterative learning processes wherein individuals set plans and goals, complete tasks, monitor their progress and outcomes, and adapt future efforts. This paper draws upon the SRL literature and related work to articulate two fundamental principles for designing SRL-promoting technologies: the Platform Principle and the Support Principle. The Platform Principle states that SRL-promoting technologies must incorporate clear platforms (i.e., tools and features) for engaging in planning, enacting, monitoring, and adapting. The Support Principle states that SRL-promoting technologies must include clear scaffolds for strategies, metacognition, motivation, and independence. These principles can be applied heuristically to formatively assess how and whether given learning technologies enable and scaffold self-regulation. More broadly, these assessments can empower educational technology creators and users to strategically design, communicate, and study technologies aligned with self-regulation. An exemplar application of the framework is presented using the PERvasive Learning System (PERLS) mobile SRL technology. 
    more » « less
  4. Self-regulated learning (SRL) is an essential factor in academic success. Self-regulated learning is a process where learners set clear goals, monitor progress toward attainment of those goals, and adapt their strategies to improve their learning. Because SRL is often not explicitly integrated into the classroom, students struggle to identify and use learning techniques empirically proven to be more successful than others. SRL is a learned skill students can develop over time that has been found to be related to high achievement and self-efficacy. This paper examines the effects of introducing SRL strategies into an undergraduate introductory physics classroom. The degree to which the students were self-regulated learners was correlated with their test averages (r = 0.23, p < 0.05). Students reported that they found the SRL instruction helpful (3.5 out of 5.0 on a 5-point scale) and 86% of the students felt the time spent on the instruction was generally appropriate. Students’ preferred study methods changed over the course of the semester, indicating that students applied SRL by adapting their learning processes based on which methods were most effective in helping them study for an upcoming exam and opting not to use techniques no longer perceived as useful. Higher achieving students were more likely to settle on highly effective techniques by the end of the semester, while lower achieving students continued to modify their learning processes. 
    more » « less
  5. Abstract Background

    Providing adaptive scaffolds to help learners develop effective self‐regulated learning (SRL) behaviours has been an important goal for intelligent learning environments. Adaptive scaffolding is especially important in open‐ended learning environments (OELE), where novice learners often face difficulties in completing their learning tasks.

    Objectives

    This paper presents a systematic framework for adaptive scaffolding in Betty's Brain, a learning‐by‐teaching OELE for middle school science, where students construct a causal model to teach a virtual agent, generically named Betty. We evaluate the adaptive scaffolding framework and discuss its implications on the development of more effective scaffolds for SRL in OELEs.

    Methods

    We detect key cognitive/metacognitiveinflection points, that is, moments where students' behaviours and performance change during learning, often suggesting an inability to apply effective learning strategies. At inflection points, Mr. Davis (a mentor agent in Betty's Brain) or Betty (the teachable agent) provides context‐specific conversational feedback, focusing on strategies to help the student become a more productive learner, or encouragement to support positive emotions. We conduct a classroom study with 98 middle schoolers to analyse the impact of adaptive scaffolds on students' learning behaviours and performance. We analyse how students with differential pre‐to‐post learning outcomes receive and use the scaffolds to support their subsequent learning process in Betty's Brain.

    Results and Conclusions

    Adaptive scaffolding produced mixed results, with some scaffolds (viz., strategic hints that supported debugging and assessment of causal models) being generally more useful to students than others (viz., encouragement prompts). Additionally, there were differences in how students with high versus low learning outcomes responded to some hints, as suggested by the differences in their learning behaviours and performance in the intervals after scaffolding. Overall, our findings suggest how adaptive scaffolding in OELEs like Betty's Brain can be further improved to better support SRL behaviours and narrow the learning outcomes gap between high and low performing students.

    Implications

    This paper contributes to our understanding and impact of adaptive scaffolding in OELEs. The results of our study indicate that successful scaffolding has to combine context‐sensitive inflection points with conversational feedback that is tailored to the students' current proficiency levels and needs. Also, our conceptual framework can be used to design adaptive scaffolds that help students develop and apply SRL behaviours in other computer‐based learning environments.

     
    more » « less