Self-regulated learning (SRL) is fundamental for children as it empowers them with the skills and strategies that they need to take control of their learning processes and achieve success [14,51]. Children with strong self-regulation skills can set specic learning goals, create plans, enact their plans, and monitor progress toward achieving these goals. As a result, they foster eective study habits and time management skills, which leads to overall success in their academic careers [64]. Furthermore, SRL skills extend beyond academic success to impact other important facets of life, such as developing eective problem-solving and decision-making skills, and adapting to dierent environments [63]. In addition, SRL skills help children develop a sense of autonomy and a proactive approach to learning. Children develop metacognitive processes, such as planning, monitoring, and reecting, learn to identify obstacles and develop strategies to overcome these obstacles by monitoring and reecting on their evolving solutions [59]. Moreover, instilling SRL into academic curricula equips children with the tools to navigate the complexities of modern education and prepares them for the future 21 BC century workforce, where continuous learning is integral to personal and professional growth [18].
Computer-based open-ended Learning Environments (OELEs) are designed to engage learners in solving complex problems independently, thus providing powerful opportunities to help them develop and utilize SRL strategies in their problem-solving tasks [31]. Their open-ended nature provides opportunities for connecting learning to real-world problem-solving scenarios, making the learning processes authentic, and therefore, more motivating. By oering choices and opportunities for developing monitoring and decisionmaking skills, OELEs empower children to make autonomous decisions about their learning [12], enhancing their ability to set goals, plan their approach to achieving these goals, make informed choices in their problem-solving tasks, and monitor progress toward achieving their goals, all of which are key components of SRL. However, not all students benet equally from OELEs, and a social justice perspective prompts us to delve deeper into the experiences of those who struggle within these environments [7]. Each student is unique, and individual dierences in learning styles, preferences, and cognitive abilities can inuence how they engage with and benet from OELEs. Some students thrive in open-ended scenarios, while others nd them more challenging due to factors such as unfamiliarity with content, diculty making sense of it, or a need to improve their self-regulation skills [22].
Our goals in this paper are to shed light on the intricacies of OELE interactions, with a specic focus on underserved students who encounter challenges within these environments [12,41]. By delving into the behaviors, strategies, and interface design elements that hinder the learning experiences of these students, we aim to contribute valuable insights and oer tangible interface suggestions to the research community [1,46]. Through this exploration, we hope to advance child-centered design principles, ensuring that OELEs are not only eective for some but are genuinely benecial for all learners, ultimately fostering equitable learning outcomes for underserved students. In line with our goals, we address OELE design issues by formulating three research questions:
• RQ1: How can we identify classes of underserved students based on their dierentiating learning outcomes after working on an OELE? • RQ2: What are the barriers and unproductive strategies employed by underserved students while working on an OELE? • RQ3: How do interface design choices in OELEs impact the learning experiences of underserved students?
To investigate these questions, we analyzed data from a comprehensive study involving 99 sixth-grade students engaged in a week-long OELE-based learning experience, specically focusing on learning science concepts by building a causal model of the human causes for the greenhouse eect and the impact of the greenhouse eect on climate. In addressing potential oversights in OELE design that lead to student diculties, our study presents three key contributions. First, we introduce a method for identifying and classifying underserved students through the analysis of quantitative data, including assessments of previous knowledge, learning outcomes, and performance during OELE engagement. Second, our qualitative analysis, grounded in video recordings of student interactions, reveals ve prominent themes hindering the learning experiences of underserved students: challenges in knowledge acquisition, challenges in scaolding learning, disregarding system guidance, not leveraging supporting tools, and getting discouraged by the lack of progress in their learning tasks. Last, having explored how interface design inuenced students' strategies in navigating these challenges, we provide design recommendations for OELEs, acknowledging the social justice imperative of addressing the needs of underserved learners.
This section synthesizes the prior literature on self-regulated learning, open-ended learning environments, and the use of pedagogical agents in these environments to support scaolds, and survivorship bias in learning and design research.
Self-regulated learning (SRL) is characterized by an engaged learner actively overseeing and managing their cognitive and metacognitive processes throughout the learning journey and aligned with their individual goals [38,66]. The emotional dimensions of learners, such as confusion and curiosity, referred to as 'learning emotions, ' are also signicant inuencers of their overall learning outcomes [39]. Curiosity-driven learning has shown signicant positive effects on children's metacognitive eciency and their ability to express their curiosity through questions [48].
Within this framework, three key dimensions are highlighted. First, there is a dual focus on both self-regulation processes and the strategies employed that target these processes. Second, the importance of continuing feedback emerges as a critical facilitator in enabling the self-regulated learning process. Lastly, SRL emphasizes the interdependence between motivation and self-regulating processes. This interconnected relationship has been extensively explored, with the social cognitive view of SRL emphasizing selfecacy as a pivotal measure of self-regulation, acting as a driving force behind motivation. In addition, various authors have armed the positive relationship between self-ecacy and motivational elements, such as goal-setting and planning [24,50].
Zimmerman's cyclical phase model for self-regulated learning (2009) [65] outlines three key phases: Forethought, Performance, and Reection. In the Forethought phase, learners undertake prelearning behaviors, including goal-setting. During the Performance phase, learners actively employ self-control processes and engage in self-observation to glean internal feedback. Progressing to the Reection phase, learners evaluate their progress based on selfobservation and engage in self-judgment. Notably, learners express emotional responses to this judgment, inuencing the input for the subsequent iteration of the self-regulated learning cycle. Winne and Hadwin's SRL model emphasizes a metacognitive perspective, portraying self-regulated students as actively involved in managing their learning using monitoring and metacognitive strategies [60]. The model highlights the goal-driven nature of SRL and the impact of self-regulatory processes on motivation. Studying involves four phases in a loop: task denition, goal setting and planning, enacting study tactics and strategies, and metacognitive adaptation [45]. Though dierent SRL models evaluate dierent areas and aspects of the learning processes, they converge to four primary components that govern self-regulation of learning, as shown in Figure 1.
Open-Ended Learning Environments (OELEs) have gained signicant attention in recent years because of their potential to support student creativity, problem-solving, and self-regulated learning [12].
These environments facilitate a high degree of student autonomy and open-endedness, allowing learners to explore, experiment, design, create, and solve in an unconstrained manner. The current state of the art for OELEs involves a range of technologies, including simulations, virtual and augmented reality, serious games, and pedagogical agents, that aim to support student learning and engagement [27,53,56,57]. According to Land and Jonassen the concept of OELEs is rooted in constructivist theories of learning, which emphasize the importance of learners' active engagement and exploration in the learning process [32]. In this perspective, learners construct their understanding of the world by actively engaging with it, and learning occurs through a process of inquiry, reection, and iteration. Wang et al.stress that the eectiveness of the OELE systems depends on various factors, such as the quality of the learning content, the pedagogical strategies employed, and the level of support provided [58].
OELEs can be particularly eective at supporting SRL as they provide learners with the freedom and exibility to explore and experiment, while also oering guidance and feedback to support self-regulation. They support SRL by providing students with targeted learning goals; a set of tools to facilitate the learning and problem-solving processes; and an open-ended approach that offers choice in how students combine these tools to achieve their learning goals [10,12]. However, as Munshi et al. observed, novice learners might be unfamiliar with these tools and lacking in SRL processes, therefore, they frequently resort to less-than-optimal strategies when approaching learning and problem-solving tasks, thus increasing the diculties they face in their learning tasks [42]. The authors have addressed this issue by designing and implementing an adaptive scaolding framework to help students develop and rene their SRL behaviors while working in an OELE.
Scaolds have been employed in several prominent OELEs to support SRL. Ecolab is a family of environments for learning ecology and adapts to the student's goal to determine the appropriate form of scaolding to support metacognitive monitoring and task selection [37]. nStudy is a web-based application that oers a toolkit for students to dene and evaluate their learning strategies and link them to their learning artifacts, such as bookmarks and notes [61]. MetaTutor employs four pedagogical agents to scaold the development and use of specic SRL processes while students learn about topics in biology [3]. Betty's Brain uses the learning-by-teaching paradigm to help students study and construct causal models of scientic processes using a visual representation. Students do this in the guise of teaching a computer agent, Betty, and their interactions with Betty and a Mentor agent, Mr. Davis, help them develop social, cognitive, and metacognitive skills [12,34].
In traditional adaptive and personalized computer-based learning environments, an individual's preferences, behaviors, and overall learning progress are captured by a user model. The user model provides the basis for the system to carry out tasks related to both adaptation and personalization [24]. Kim et al. studied how pedagogical agents can play a key role in supporting SRL in OELEs by providing learners with feedback, guidance, and prompts for reection [66]. Fu et al. implemented conversational agents to support children's socioemotional learning through self-talk, which aects cognitive performance, the ability to self-regulate, and problemsolving skills [23].
A pedagogical agent may ask learners to reect on their goals and progress and oer suggestions on alternative approaches to a problem. The integration of pedagogical agents and other intelligent technologies into OELEs has the potential to support learners' development of metacognitive and self-regulated learning skills, which in turn also helps with academic achievement. Pedagogical agents can play dierent roles in open-ended learning environments, depending on the specic learning goals and the design of the environment. For example, a pedagogical agent may play the role of a mentor or coach, and provide guidance and support to learners as they engage in self-directed learning activities [8,11]. Pedagogical agents can also take on the role of a peer or collaborator, who interacts with learners more socially and conversationally [19,33]. Blair and Schwartz state that by observing and imitating the pedagogical agent's behavior, learners can acquire new skills and knowledge, which can be useful for tasks that require procedural knowledge or domain-specic expertise [13].
Survivorship bias is the tendency to concentrate on the positive outcomes of a selection process and overlook the results that generate negative outcomes [28]. This has been studied in various disciplines, including nance [17], information retrieval [25], and healthcare [20]. These studies highlight the importance of recognizing and mitigating survivorship bias to ensure a more accurate and comprehensive understanding of outcomes.
In educational research and design, survivorship bias is a crucial phenomenon that demands careful consideration. It arises when disproportionate attention is given to successful outcomes or participants who have completed a specic intervention, resulting in an incomplete understanding of the challenges faced by those who could not. Furthermore, it can signicantly impact User Experience design, introducing challenges that may compromise the quality of user interactions. Survivorship bias often shifts the focus to success stories, which can overshadow failures, thus creating a skewed perception of user requirements. This limitation hampers the ability of designers to comprehensively address user needs, ultimately contributing to a suboptimal experience. Recognizing challenges that can disrupt or render studies and experiments ineective is an important aspect of research, particularly in the context of Human-Computer Interaction. Rukmane, et al. explored failures within the Child-Computer Interaction community and found they are not reported enough in the literature due to the pressure of publishing successful stories [49].
Nevertheless, there has been an increasing amount of research to support underserved children and help equalize opportunities for all. Ruan, et al. have investigated the relationship between emotions experienced during learning and metacognition in typically developing (TD) children and those with autism spectrum disorder (ASD) [47]. They adopt a social justice approach for improving machine learning algorithms that examine the relationship between facial emotion expressions and metacognitive monitoring performance for both TD children and those with ASD. Antle et al. designed a neuro-feedback system and applications that enabled traumatized children living in poverty to learn and practice self-regulation by playing games [2]. The researchers addressed design choices to overcome challenges, such as working with illiterate children, those who did not speak English, and lacked computer experience.
McLaren and Antle presented a mixed methods framework for evaluating whether sound can help children with attentional challenges to self-regulate using a neurofeedback system [40].
Technology is not neutral and often not suitable for all children involved in a study. Despite the extensive research on SRL, OELEs, and the role of pedagogical agents, there remains a notable gap in understanding how these environments cater to the needs of underserved students [31,67]. Existing research often focuses on overall learning gains and successful outcomes, potentially overlooking the challenges faced by students who struggle within these environments [21,53]. Moreover, the phenomenon of survivorship bias in educational research and design further complicates eorts to ensure equity and inclusivity, as it tends to prioritize successful outcomes while neglecting the experiences of students who encounter diculties, and, therefore, fail to make progress in their assigned learning tasks [26]. Posing questions about the suitability of SRL for all kinds of students, and further investigating how their actions should trigger appropriate scaolding mechanisms in these environments, especially when students are having diculties, is of paramount importance.
Our research focuses on expanding knowledge concerning design methods and researcher reexivity. Much like design itself, which is often generative and forward-looking, social justice endeavors share a similar orientation towards the future. By adopting a social justice-based approach and acknowledging the importance of mitigating survivorship bias, we hope to bridge this gap by examining how OELEs and pedagogical agents can be designed and implemented to support underserved students, investigate how their choices and actions in the OELE might be better understood and addressed, and ultimately contribute to the development of more inclusive and eective learning environments that promote equitable learning outcomes for all students.
This paper analyzes data collected from a week-long sixth-grade urban classroom study in the southeastern United States. Students used the Betty's Brain environment to learn about a complex science phenomenon [43]. The analysis focused on students' self-regulation behaviors and the eectiveness of adaptive scaolds delivered by pedagogical agents to help students develop eective learning and problem-solving strategies [41].
Betty's Brain is an OELE that utilizes the learning-by-teaching paradigm to engage students in learning about science topics [10,34].
The system includes hypertext resources that describe the topic under study, and students are expected to read the resources and construct a causal model to teach Betty, the Teachable Agent. By reading and translating the content in the hypertext resources into a correct causal map of science phenomena, the students demonstrate their emerging understanding of the corresponding science topic. The student's overall goal is to learn the scientic topic well enough to teach Betty a complete model of a scientic process. The complete map, generated by the student's teachers or the research team, is used to evaluate the students' model. The topic of study for this project was the human causes of climate change, and the corresponding "expert" causal map is shown in Figure 2.
To build the causal map, students can access the science book to acquire knowledge on the three main topics that make up the causal model: Human Activity that causes the Greenhouse Eect, the Greenhouse eect, and the impact of the Greenhouse eect on Climate change (Figure 3a). Students use the drag-and-drop causal map editor to express how concepts are related to each other(Figure 3b). At any time, students can ask Betty to take a quiz and her performance helps them assess the correctness of their maps (Figure 3c). These quizzes are created dynamically by Mr. Davis, the Mentor agent, who also grades them and keeps track of Betty's overall performance. The quizzes are designed to provide feedback on the correctness and completeness of their scientic model, and students can use this information to determine where they have made msitakes. In addition, Mr. Davis monitors the student's actions as they go about their tasks.
The study reported in this paper took place in December 2018 with 99 consenting sixth-grade students in an urban public school in the southeastern US. Overall, the school's population is 60% White, 25% Black, 9% Asian, and 5% Hispanic, with 8% enrolled in the free/ reduced-price lunch program, which reects the demographics of our study population. Unfortunately, individual classroom demographics were not collected for this study.
The study spanned ve days. On day 1, students took a paper-based pre-test that used multiple-choice and short-answer questions, to assess their comprehension of the domain and prociency in causal reasoning skills. The tests were designed in consultation with middle school educators, keeping the sixth-grade public school curriculum in mind. Day 2 was dedicated to students engaging with a practice unit to acquaint themselves with the Betty's Brain environment. On Days 3-5, students actively built causal models of climate change in the Betty's Brain OELE. On the nal day, the students also completed a post-test identical to the pre-test. Data collected from consenting students included screen recordings, webcam recordings, eye-tracking data, system logs, and the paper-based pre-and post-tests. These tests have previously been used in multiple Betty's Brain classroom studies [43,53].
3.5.1 RQ1: Identification of Underserved Students. In exploring the rst research question (RQ1), we used two dierent data sources to understand how students beneted from the intervention. We started with pre-tests to assess what students already knew about climate change. Then, by comparing pre-and post-test results, we measured how much each student learned overall through our intervention. Additionally, system logs documented the sequence of actions performed by the students,i.e., reading, note-taking, mapbuilding, and quiz-taking activities with corresponding timestamps. The nal map scores, a composite measure derived from the sum of correct links minus incorrect links on the students' maps, were instrumental in gauging not only what students learned but also their potential misconceptions.
In previous studies that evaluated Betty's Brain data, students were grouped into High Learning Gain (High) and Low Learning Gain (Low) categories based on the dierences in their normalized learning gains in domain knowledge and causal reasoning skills [42]. However, categorizing students solely based on pre-to-post test dierences has its limitations. The scenario where a student with a high pre-test score maintains or slightly raises their score in the post-test may result in a categorization of Low Learning Gain, despite the limited room for improvement (ceiling eect). Conversely, a student with a low pre-test score might have more opportunities for growth in the post-test, but this doesn't guarantee the student will have positive learning gains.
The intervention is expected to yield signicant overall learning gains when aggregating students' learning gains. Past studies have shown high aggregated learning gains across all participants in a study [12,34,43]. However, to determine if these are signicant for all of the members in a study, we further delve into the data to identify classes of students based on their diering learning outcomes. We looked for the signicance of these gains for all members of the group individually. To do this, we created a quadrant-based categorization by performing a median split on both pre-test and post-test scores for each student, and categorized them before and after the intervention using the following labels: low-low, low-high, high-low, and high-high. As a clarication, a student with a low-low label had a pre-test score that was below the median score for the class and a post-test score that was also below the median score for the class. This approach provides us with a more nuanced assessment of the role of previous knowledge in students' knowledge acquisition and their relative ranking in the class before and after the intervention. Additionally, this quadrant-based approach also allows us to investigate situations where learning gains and performance within the system, measured by map scores, may diverge, providing a comprehensive understanding of the intervention's impact on dierent student groups.
3.5.2 RQ2 and RQ3: Unproductive Strategies and the Role of the Interface. After categorizing students into quadrants based on pretest and post-test scores in RQ1, our attention turned to two specic groups for RQ2 and RQ3: low-low and high-low. These groups represent underserved students who encountered challenges in deriving comparable benets from the learning environment as their peers. To delve into a detailed analysis of students' behaviors, we used the following data sources: screen recordings depicting student's work on the OELE with an added overlay of their gaze paths (captured using a calibrated Tobi 4C eye tracker) as well as webcam footage showing their interactions with others, reactions, and facial expressions. This multimodal approach enabled a more nuanced analysis not only of their engagement with the OELE and the task, but also their attentional focus, emotional responses, and social dynamics. This provided us with deeper insights into their learning processes, motivations, and emotional responses. We selected 20 students (12 low-low and 8 high-low) who had complete data spanning all four days of working on the climate change unit, totalling 80 hours of videos. This approach enabled us to observe the persistence of specic challenges the students had and their inability to overcome them throughout the intervention.
As a next step, we employed a thematic analysis approach [16], which allowed us to analyze the data and uncover underlying patterns and themes systematically. The analysis was conducted collaboratively by the rst and second co-authors, following the phases of thematic analysis. Initially, we familiarized ourselves with the data and then proceeded to generate initial codes to encapsulate key ideas. Through an iterative process, these codes were conceptually grouped to form overarching themes. All through this process, the last two authors provided valuable input, oering high-level feedback on the coding process and codebook. After rening the codebook, we identied a nal subset of emergent themes that aligned closely with our research questions. We provide details of the codebook in the Appendix. This in-depth exploration not only shed light on the barriers and unproductive strategies employed by underserved students (RQ2) but also delved into the inuence of interface design choices on their learning experiences (RQ3).
Existing approaches usually categorize students into two groups by overall performance High and Low-based solely on dierences in their normalized learning gains from pre-to-post tests. However, this classication overlooks the nuances in students' interactions with the system and their performance, particularly in scenarios where students with high pre-test scores might have limited room for improvement, which could potentially lead to misclassications. Conversely, students with low pre-test scores might have more opportunities for growth in the post-test, but this doesn't guarantee substantial improvement in understanding of the learning materials. Therefore, The research team decided to conduct a more in-depth analysis to gain a deeper understanding of the impact of the intervention on dierent student subgroups. To achieve this, we introduced a quadrant-based categorization scheme that considered median splits on both pre-test and post-test scores independently. This approach, outlined in Table 1, allowed the researchers to more accurately identify and target underserved student populations. Identifying students' prior knowledge of the subject materials allows us to better analyze if and when the system's design choices and content might have been skewed toward students who start with a good grasp of the subject material. Furthermore, by assessing where students t in the median split of the post-test, we can focus our eorts on the students who did not benet from the intervention with the system. The quadrant analysis, dened by the four quadrants outlined in Table 1, showed that a majority of students who scored high on the pre-test also did so on the post-test, conrming how prior knowledge is almost always a pre-requisite for learning and applying SRL skills for complex tasks [54,55]. In addition, students who transitioned from a low score on the pre-test to a high score on the post-test demonstrated signicant improvement, indicating that the OELE provided them with productive learning experiences by correctly attending to their specic needs and skills, which resulted in high gains. However, our goal was to delve deeper into the experiences of the students who did not benet much from the intervention. Therefore, we studied two groups in greater detail:
(1) 31 students who started low on the pre-test and remained low on the post-test, i.e., the Low-Low (L-L) group; and (2) 14 students who started with high scores on the pre-test but ended with low scores on the post-test, i.e., the High-Low (H-L) group.
Furthermore, we compared our quadrant-based classication to two other metrics to better circumscribe the underserved students we selected for a deeper analysis: (1) the nal scores that students achieved on their causal maps in the Betty's Brain environment; and (2) their pre-to-post normalized learning gains. Out of the 31 students classied as low-low in our quadrant, 28 of them also had low map scores at the end of the Betty's Brain intervention and low learning gains, reinforcing their status as underserved students in need of assistance. For the 14 students who were in the highlow quadrant, a signicant proportion had high map scores in the Betty's Brain environment, but they still exhibited low learning gains. This nding led us to hypothesize that some students with sucient previous knowledge may have gotten confused when working with the Betty's Brain environment.
Interface Impact
To answer RQ2 we focused on the students from the Low-Low (L-L) and High-Low (H-L) quadrants that we discussed in RQ1. This categorization helped us identify underserved students who may not have beneted from the learning environment in the same way as their peers. Out of the 20 students selected for in-depth qualitative analysis, 12 belonged to the Low-Low group, while 8 belonged to the High-Low group. We eliminated students for whom we did not have data for all four days of the intervention.
The qualitative analysis involved video data acquired from three sources: (1) screen recordings showing students' interactions with the learning environment; (2) webcam recordings capturing facial expressions and reactions; and (3) eye-tracking overlays providing insights into attention and focus on specic visual elements. Thematic analysis was employed to identify patterns and themes in students' behaviors. Similarly, we focused on the challenges students faced in their interactions with the Betty's Brain environment. We created the code book using the Betty's Brain task model developed in previous work [30,53]. The task model breaks down the primary OELE tasks into three cognitive processes that support metacognition and self-regulation behaviors, i.e. Information Seeking/Acquisition, Solution Construction/Renement, and Solution Assessment. The cognitive processes are directly linked to observable actions students perform on the system's interface, such as reading, taking notes, editing the causal map, asking for the agents' assistance, taking quizzes and evaluating answers.
Thematic analysis of the data resulted in identifying ve major themes that represent the progression of behaviors and strategies of the underserved students, from their getting started with the system until they disengage from it:
(1) Challenges in knowledge acquisition, (2) Challenges in adopting scaolds for learning, (3) Disregarding system guidance, (4) Not leveraging the support tools provided, and (5) Getting discouraged because they could not gure out how to correct incorrect answers in the quizzes. Figure 4 depicts the thematic analysis results on the progression of challenges faced by underserved students. This provides us a visual roadmap illustrating the interconnected nature of the identied themes and codes. It reveals a sequential ordering relation among the problems, where one problem leads to another, reecting the cascading eects of student interactions with the learning environment and the need for rethinking the structure of our interventions.
For instance, consider a student who initially spends a lot of time reading the science book without beginning to translate their understanding to causal links and start building the causal map. This to delayed knowledge application can lead to further problems such as the inability to remember material that was read earlier.
As Theme 1 -Challenges in Knowledge Acquisition progresses into Theme 2 -Challenges in Adopting Scaolds for Learning, the student's disorganized approach hampers eective knowledge acquisition and map construction. Despite the system's attempts to provide guidance through the Mentor Agent and Betty in Theme 3 -Disregarding System Guidance, the student may ignore feedback due to intense focus, inability to interpret the guidance, and even unclear instructions in the guidance. This pattern continues with Theme 4 -Not Leveraging Supporting Tools, as the student neglects to explore available resources provided in the system. Finally, in Theme 5 -Getting Discouraged by Incorrect Answers, the student's frustration peaks when quiz results do not improve in spite of their eorts to add and correct links. The lack of clear returns for their eorts, leads to disengagement [5,36]. This sequence was observed multiple times in the rst two days of the intervention, inuencing students' dispositions to continue putting eort until the end of the activity. This gure elucidates the sequential progression of challenges faced by underserved students, highlighting critical points where intervention and system improvements are needed.
Building on these ndings, we delved into a deeper exploration of the impact of the interface design on students' behaviors and choices, to answer RQ3. Specically, we focused on understanding how the interface played a role in hindering students' performance, failing to support them, and negatively inuencing their learning experiences. For each code and theme, we also looked for distinctions between the High-Low and Low-Low groups, to further our understanding of how the dierence in their previous knowledge and learning skills also inuenced their actions and how they perceived the system.
The students' knowledge acquisition challenges are primarily inuenced by the information seeking task in the OELE. Information seeking covers reading the science book to learn about science concepts and relations and note-taking serves as a memory aid to capture information that is relevant to building the causal map. Fifteen students, 8 from the low-low group and 7 from the high-low group encountered challenges related to Theme 1. Across most codes, there was no signicant disparity between the two groups. However, it is noteworthy that the code about feeling overwhelmed by the amount of content was exclusive to the L-L group. Five students (3 L-L and 2 H-L) faced challenges in initially acquiring knowledge from the Science Book. This implies that the system was not successful in helping the students engage in the reading task so they could nd new information and apply it in an eective manner to build the causal map. Six students (3 L-L and 3 H-L) relied solely on prior knowledge rather than acquiring new information from the science book, opening up the possibilities of prior misconceptions being used to construct the causal map. This challenge highlights how the system failed to help students adapt and become engaged in new learning experiences. Eight students (4 L-L and 4 H-L) had delays in starting their map building task. This is because they focused solely on knowledge acquisition or continued to explore of the system (perhaps, because they did not understand their learning task). Two students from the L-L group showed signs, either through facial expressions or speech, of being overwhelmed by the amount of content presented in the Science Book. In other words, they were overwhelmed because of cognitive load. Furthermore, since the intervention lasted four consecutive days, we observed how not reviewing work from previous days hindered the metacognitive process of reection for eight students (5 L-L and 3 H-L).
The organization of the science book was a major cause for many of the diculties students faced when acquiring knowledge. Its lack of hierachy of content and dense combination of both text and images proved to be overwhelming for students, causing them to bypass the rst process of acquiring knowledge and instead depend on their current knowledge or prematurely create causal maps (refer to Figure 5 for the structure of the interface). Furthermore, certain students felt obligated to thoroughly study all pages before trying to create their maps, which caused a delay in knowledge application (i.e. building their solution). In addition, the organization of the science book includes hyperlinked texts that redirects students to a dierent page on the dictionary. This causes confusion and prevents them from following a clear and logical reading path where they can use the structure of the science book to build a chain of causal links connecting related concepts. The interface's design unintentionally distracts students from their main reading objective, complicating the process of acquiring knowledge.
The combined inuence of these interface design elements aected the students' capacity to acquire and employ information in an eective way for map building, highlighting the crucial function of intuitive interface design in supporting students' productive knowledge acquisition. In addition, at the beginning of the study, the researchers suggested that students should build their models sequentially and in parts, however this group of students either ignored or did not understand this suggestion and the system itself failed to reinforce it as a strategy.
Theme 2 -Challenges in Adopting Scaolds for Learning All 20 students faced challenges in systematically approaching their causal model construction and debugging tasks. This happened in relation to all of their primary cognitive processes: information seeking and acquisition, solution construction and renement, and solution assessment. Seventeen students (12 L-L and 5 H-L) underutilized the note-taking feature. Thirteen students ( 10L-L and 3 H-L) encountered diculties in acquiring knowledge sequentially or understanding how concepts identied in one part of the Science Book might relate to others. This points to issues in understanding the hierarchical structure of the science book and the inability to use the hyperlinks to navigate material in the book to nd relations between concepts. As a result, the students had diculties in nding an eective approach to developing and advancing their causal maps. Sixteen students (8 L-L and 8 H-L) faced challenges in systematically evaluating their work by taking quizzes. Unable to interpret the quiz results, i.e., the implications of right and wrong answers, they resorted to continuing to read the science book in the hope that may help them nd erroneous links, or they just continued to tweak their maps for a long time (a pure trial and error approach) [62]. Eleven students (7 L-L and 4 H-L) struggled to assess their work about the current state of their causal models, taking quizzes on topics and concepts that they had not added to their causal maps. Six students (4 L-L and 2 H-L) often moved on to add additional links to their maps without attempting to correct errors shown in the quiz results. This resulted in the number of errors continuing to grow in their map, which made it even harder to debug their maps [29].
The challenges to scaold learning are intricately connected to the design of the learning environment. The note-taking tool, although designed to facilitate systematic knowledge acquisition, poses a navigational obstacle. The procedure necessitates that students transition to a distinct 'Notes' page to write and access their notes, causing interruptions in their reviewing of the science book and diculty in linking the notes back to science book pages, thus reducing the eectiveness of note-taking as a mechanism for collecting linked information as a reference and organizer for subsequent map building. This design issue frequently leads to students not using the note-taking feature, as the notes are not easily accessible in conjunction with the knowledge acquisition and map-building tasks.
The causal map workspace, in conjunction with the agents and quiz results page, is specically designed to streamline the process of constructing, debugging, and assessing students' work. Nevertheless, the interface fails to suciently motivate students to interact with these aspects systematically and eectively. Students frequently prioritize building or correcting their maps in the causal map workspace to an extreme degree, disregarding the crucial procedures of evaluating their map using the quiz results, pinpointing errors and missing links, and then trying to update their models to correct their errors. in other words, students often resort to trial and error approaches to debugging their map instead of using a systematic assess, locate, and correct process using the quiz results [29]. This disparity suggests that the interface should stimulate and reinforce the signicance of regular and specic checking of the causal map to improve the overall eectiveness of the causal map building (solution construction) process. The Mentor agent and Betty could be more eective in reminding students and facilitating these important SRL processes.
This theme describes the gap in eective student-agent communication. The agents monitor students' actions to provide adaptive scaolds and encouragement, oering tips on specic learning or debugging strategies like periodically taking quizzes to check their map, and cautioning against adding concepts to the map without prior reading. Fourteen students were coded for this theme and two codes revealed a signicant disparity, with a higher count of L-L students neglecting to read feedback or utilize quiz feedback to correct misconceptions in the map. Having the eye-tracking gaze overlaid on the screen recording provided a mechanism for more detailed analysis. On multiple occasions, seven students (6 L-L and 1 H-L) did not engage with or read the feedback provided by the agents, often ignoring them for long periods or just clicking on the dialogue until the agent went away. Furthermore, eleven students (6 L-L and 5 H-L) read agent feedback that provided information on eective information acquisition and solution construction strategies, but chose not to act on the suggestions provided. This included suggestions for reading specic pages that contained information that was relevant to the part of the causal map the student was constructing and tips on how to go about debugging the students' current causal map.
The system utilizes pedagogical agents to motivate and guide the students toward their learning goals, especially the ones who face diculties. However, the ecacy of this assistance is dependent on how well the interface provides the feedback. The feedback, designed to steer students away from unproductive sequences of activities, is shown as conversational prompts at the top of the screen (see Figure 6 for an example of conversational feedback). Nevertheless, this design decision poses two main concerns. First, the positioning and format of the feedback require students to pause their tasks, which may interrupt their learning and thinking processes. Furthermore, the repetitious nature of the feedback, which lacks precise direction tailored to the situation, causes children to either ignore the information or mechanically proceed without actively interacting with the subject on hand. In addition, the system's inability to track and address whether the children take action based on the feedback exacerbates this problem. This is apparent in cases where students fail to inquire into the causes of their incorrect answers, even after getting feedback on their errors. The interface design choices unintentionally lead to learners ignoring system instructions, highlighting the urgent requirement for better user-friendly and prompt feedback systems in educational interfaces.
Theme 4 -Not Leveraging Supporting Tools This theme encompasses various ways students could have been supported by the system, including initiating conversations with agents and exploring functionalities for understanding systematic map construction and map debugging processes. Thirteen out of 20 students struggled with this theme, and across all codes, the L-L group exhibited a higher count than the H-L group. Six students (4 L-L and 2 H-L) did not initiate dialogues with the agents to take advantage of the supportive information provided by them, eleven students (8 L-L and 3 H-L) did not actively seek explore the resources in the Teacher's Guide, which provides valuable tips, and ten students (7 L-L and 3 H-L) were not consistent or did not use the "Mark as Right" tool, a visual tool to both visually reinforce correct links while also dierentiating it from the portions of the map that still require assessment and revision.
The theme of Not Leveraging Supporting Tools is closely linked to how these tools are presented and made accessible through the interface, rather than their complete absence. The essential supporting tools, such as agent feedback, the teacher's guide, and the link-marking feature for the causal map, suer from challenges linked to visibility and user engagement. The teacher's guide closely mirrors the structure of the science book, providing a dense combination of pertinent tips and supplementary information. The intricate nature of this resource poses diculty for children in eciently extracting relevant information, potentially resulting in its disregard. This is especially pertinent for younger children, who may not be good readers and may not like to read long passages. Furthermore, the link-marking feature, which is essential for accurately measuring causal map accuracy and simplifying the error nding and correction process, is not as prominent on the UI as other features that have dedicated buttons. The lack of clarity leads to the underutilization of an essential instrument. Furthermore, the agents' conversational feedback, although intended to provide help, does not successfully inform or remind students about the accessible capabilities, such as the ability to examine particular causal relationships. The deciency in the agents' guiding system exacerbates the underutilization of supporting tools. Together, these interface design elements have a considerable inuence on students' level of involvement with and use of the materials that are accessible to them.
Theme 5 -Getting Discouraged by Incorrect Answers Students constantly got frustrated and discouraged as a result of receiving negative quiz assessments [6,12]. We noticed how this theme was heavily inuenced by challenges students faced previously in other themes. For instance, a student who struggled with acquiring knowledge initially, and then began building the map based solely on previous knowledge (Theme 1) was more likely to doubt system accuracy during the quiz assessments. In all of our codes, the L-L group had only a slightly higher count than the H-L group. Thirteen out of 20 students were coded for this theme, with four (3 L-L and 1 H-L) having openly expressed disbelief about the system being correct, a potential breakdown in trust between the student and the learning environment. Eleven students (7 L-L and 4 H-L) repeatedly deleted their work, either displaying a lack of condence while building their causal maps or after receiving a 0% quiz assessment. Interestingly, some students that chose to delete their existing work and start anew, chose to do so on a dierent topic, without attempting to correct their previous errors. Six students (4 L-L and 2 H-L) gave up on the learning task and disengaged from the system for a considerable period of time after failing a quiz.
The interface primarily communicates correct and incorrect links to the student indirectly via the quiz results (see Figure 3c). Additional information is provided through the agents' feedback. The quiz results page classies each answer to a quiz question as either 'correct', 'incorrect', or 'I don't know', depending on the student's constructed causal map. Additionally, it oers a partial or complete view of the student's causal map in connection to every quiz question. Nevertheless, the interface's approach to simply highlighting incorrect answers without providing thorough explanations may result in student dissatisfaction. This dissatisfaction is evident through behaviors such as completely deleting their map because of low quiz scores and the inability to nd and x errors or growing doubt about the system's accuracy and applicability to the realm of knowledge. Furthermore, the inability of the interface, and by extension, the agents, to clarify why an answer may be incorrect and to initially provide a step-by-step process to help students nd errors in their map may contribute to student disengagement. This can become worse if the situation recurs frequently. The absence of comprehensive feedback and adaptable support may be a primary cause for student disengagement following many failures to understand the reasoning behind graded answers. To summarize, the interface design's constraints in delivering unambiguous, productive feedback and its inability to dynamically interact with students' learning processes greatly contribute to student discouragement.
Our utilization of the quadrant-based approach to characterize and study children's diculties in using an OELE distinguished this work from previous research in the eld. Our results provide a more nuanced understanding of students' learning trajectories from start to nish. By categorizing students into L-L and H-L groups, we investigated the actions and strategies employed by the underserved students. A number of these children started with low prior knowledge but some did not; they started with high prior knowledge and ended up with low post-test scores. We systematically investigated how the system's design might have contributed to students' inability to progress leading to confusion and frustrations. This approach allowed us to analyze the design with a more critical eye, specically in terms of individual student's needs. Each theme identied in our analysis represents an important aspect of design for OELEs in supporting underserved students who usually struggle in such open-ended environments. The progression of themes, from challenges in knowledge acquisition to disregarding system guidance and getting discouraged by incorrect answers, underscores their interconnected nature and impact on student engagement and learning outcomes. Our ndings strongly suggest that underserved students require more support during their initial interactions with the system, which should be gradually faded (i.e., reduced) to provide them with opportunities to develop their own SRL skills [44]. A general issue with providing adaptive scaolding in OELE environments is that the scaold triggering mechanisms re more frequently for students who engage more frequently with the system, resulting in underserved students receiving less actionable feedback [35,41]. Monitoring the actions represented in our code book and addressing these issues could enhance the eectiveness of OELEs in supporting all students' learning journeys.
The scaolding agent feedback framework of Betty's Brain, as described by Munshi, et al [41], has two essential elements: a learner model for tracking trigger conditions and a conversational tree structure for providing feedback when these conditions are met. This paradigm primarily depends on the learner model to capture the context of the present task, the ecacy of recent actions, and unique problems associated with continuing tasks. The signicance of basing scaolds on the specic tasks and model-building eorts of students oers concrete indicators for strategic feedback [52].
Nevertheless, this feedback method has several constraints, especially for underserved students whose learning experiences may not correspond to the pre-established learner model in the system.
A previous empirical study found that there was a discrepancy in the frequency of activities between those who performed well and those who performed poorly, with the high performers participating in a greater number of actions [43]. Hence, the existing scaolding agent feedback architecture tends to prioritize those who excel in terms of the frequency of actions, inadvertently neglecting the needs of those who struggle. This was further emphasized in a subsequent study, revealing that feedback was found to be more benecial to high performers than their peers [42]. Ultimately, the scaolding agent feedback architecture in Betty's Brain is a complex tool that supports students. However, its eectiveness varies across dierent student groups, especially underserved students. This discrepancy highlights the need for more investigation and advancement in this eld. We strive to overcome these restrictions by oering customized design recommendations that cater to the needs of underserved students, therefore promoting a fair and ecient learning experience for all system users.
This work underscores the critical role of rst supporting cognitive abilities to foster eective SRL within OELEs, particularly for underserved students. Our ndings indicate that SRL frameworks may not eectively support these students if they lack well-developed cognitive abilities, which generalizes beyond the OELE context [4,9]. In Betty's Brain, for instance, such cognitive abilities encompass reading and comprehension skills, understanding what a causal map is and how it supports the understanding of relationships between parts of a model, and the ability to reason with correct and incorrect quiz answers and how they relate back to the causal map [12]. Therefore, OELEs must prioritize supporting students' cognitive development early in their engagement with the system. By providing scaolding and assistance tailored to these cognitive abilities, OELEs can empower students to navigate the learning environment more eectively. This initial support lays the groundwork for students to gradually develop SRL, specially their metacognitive and self-regulation processes to gradually become eective learners [15]. Ultimately, this approach enables students to benet from the open-ended nature of OELEs, fostering cognitive and metacognitive skills, and then self-regulation processes that promote independent learning and autonomy as they progress in their educational journey.
Importance of Content Hierarchy and Specicity Theme 1 emerged because of the dierent struggles underserved students went through in the process of acquiring knowledge. When they rst access the Science Book to begin their learning tasks, they are presented with a long list of topics, pages, and dictionary items, all on a scrollable panel, as seen in Figure 5. There are over 30 items in this list, but only 10 pages have concepts that relate to map construction, of which students are unaware. Below is a dialogue between two students who were coded as "Overwhelmed by the amount of content to read": Content pages should not be represented the same way as dictionary pages, teacher's guides, or tips. Pages with dierent purposes should be visually distinct to avoid confusing the students or overwhelming them with long lists.
Support material that might help in the acquisition, application, and assessment of knowledge should be readily available on any screen, independent of which tab the student is working on. Within Betty's Brain, this principle holds particularly true for tips. These tips are concise and practical suggestions located at the end of a series of pages in the Teacher's Guide. Unfortunately, many students fail to take advantage of this valuable source of knowledge. The agent has received feedback indicating that students are reluctant to modify the context and conduct a search on a separate page, despite being advised to read a specic tip.
The note-taking feature is a good example of a feature that is readily accessible at all times, on any screen, as are the agents, however even though it is possible to take notes on every screen, the students cannot read them at all times. There is a specic tab listing all the notes previously taken and if the student wants to use them to assist in the knowledge construction on the map, the system requires constant changes of page context back and forth, increasing the students' cognitive load.
The thematic analysis showed that a lot of the struggles underserved students encountered were in failing to scaold learning. The system's mechanism of adaptive scaolds relies on the students engaging with the agents, which was also shown to be a struggle that students, especially those in the Low-Low group, faced. The length of the content delivered, its perceived utility, and how many times the same content has been delivered before, all these elements play a part in students' choices to avoid agent feedback or disregard their advice. Figure 6 shows Student C's screen and gaze, after receiving feedback from the mentor agent after a failed quiz assessment. The following dialogue took place while the student was engaging with this. *While seeing a lengthy answer from the agent, the student stares in disbelief and says: * Student C: Oh my gosh! I have to read all of this? *Student closed the feedback without reading it. * Student C: I really don't care. *Student disengaged from the system. *
We suggest that agent feedback should have tiers, feedback should not prevent the student from completing the task they are working on, and feedback must be addressed by the student at that time. Choosing how and when to classify feedback as such should be contingent on:
(1) The type of content that is being delivered by the agent. Timely content that serves to course-correct a student before they disengage is an example of that. (2) The number of times a student has ignored specic feedback and continued to struggle with the same problem. The system should be able to change the phrasing on the text to convey its importance to the student before changing its tier, but after multiple failed attempts, make it so the student cannot ignore it and had to take action.
Furthermore, instead of agent conversations just having suggestions on actions to take, could also include actionable tasks that the students must perform before continuing the conversation. The agent would not only suggest the student take an action but walk the student through steps required to complete the action, as a mentor might in real life when a student is struggling to make progress.
The in-depth qualitative ndings constitute a strength of this research, though the small sample size of evaluated students poses a limitation in terms of generalizability. To address this, future studies could leverage our social justice-based framework with a larger and more diverse sample to enhance the applicability of our ndings. Furthermore, though demographic information about students was not collected for this study, future research should prioritize this to contextualize ndings and explore potential disparities among student groups. Additionally, the nature of students' participation in the study, which took place during regular class hours and included parental consent, may have inuenced their motivation and learning outcomes. Investigating the impact of research context on student engagement is essential; hence, future studies should delve into factors such as parental involvement and extrinsic incentives. Lastly, considering the rising prominence of Large Language Models in educational technology, integrating them into OELEs could oer enhanced adaptive learning experiences and provide valuable insights for the design of educational technologies and instructional practices. Addressing these limitations and exploring future research will contribute to advancing the eld of educational technology and promoting equitable learning opportunities.
In this study, we took a social justice-based approach to investigate the experiences of underserved students in OELEs, aiming to identify classes of underserved students, understand barriers, and explore the impact of interface design choices. Through a weeklong classroom study with 99 sixth-grade students, we developed a method for classifying underserved students, utilizing a quadrant approach based on assessments of pre-test and post-tests individually, and validated against learning gains and scored performance within the system. Complementing this, qualitative analysis of video recordings revealed ve prominent themes hindering the learning experiences of underserved students, encompassing challenges in knowledge acquisition, scaolding learning, disregarding system guidance, not leveraging supporting tools, and getting discouraged by incorrect answers. Additionally, our investigation extended to interface design choices, providing insights into their impact on students' strategies. Drawing from these ndings, we offered design recommendations aligning with child-centered design principles to foster more inclusive and equitable learning outcomes for all students.
The study received approval from an Institutional Review Board at a University in the southeastern United States. It took place during regular class hours at a school where children were recruited by their science teachers, who had been collaborating with the research team on multiple studies since 2010. Teachers received consent letters providing detailed information about the research's purpose, activities, data collection, storage, duration, and condentiality measures. Similar consent letters were sent to guardians, emphasizing the voluntary nature of participation, benets of the intervention based on previous studies, and assurances about data privacy. Students received assent letters explaining the research in language suitable for their age and encouraging discussions with their guardians to make a joint decision about participation. All letters highlighted voluntary participation. Researchers worked with teachers to conduct interventions during regular class hours, minimizing disruptions to students' schedules. Consent and assent letters addressed ethical considerations, including data storage, access, and publication anonymization, and emphasized the voluntary nature of participation. Additionally, it was explicitly stated that students could withdraw from the study at any time.