skip to main content


Title: A Heuristic Assessment Framework for the Design of Self-Regulated Learning Technologies
Researchers and educators have developed a variety of computer-based technologies intended to facilitate self-regulated learning (SRL), which refers to iterative learning processes wherein individuals set plans and goals, complete tasks, monitor their progress and outcomes, and adapt future efforts. This paper draws upon the SRL literature and related work to articulate two fundamental principles for designing SRL-promoting technologies: the Platform Principle and the Support Principle. The Platform Principle states that SRL-promoting technologies must incorporate clear platforms (i.e., tools and features) for engaging in planning, enacting, monitoring, and adapting. The Support Principle states that SRL-promoting technologies must include clear scaffolds for strategies, metacognition, motivation, and independence. These principles can be applied heuristically to formatively assess how and whether given learning technologies enable and scaffold self-regulation. More broadly, these assessments can empower educational technology creators and users to strategically design, communicate, and study technologies aligned with self-regulation. An exemplar application of the framework is presented using the PERvasive Learning System (PERLS) mobile SRL technology.  more » « less
Award ID(s):
1712328
NSF-PAR ID:
10377020
Author(s) / Creator(s):
;
Date Published:
Journal Name:
Journal of Formative Design in Learning
ISSN:
2509-8039
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    Undergraduate STEM lecture courses enroll hundreds who must master declarative, conceptual, and applied learning objectives. To support them, instructors have turned to active learning designs that require students to engage inself-regulated learning(SRL). Undergraduates struggle with SRL, and universities provide courses, workshops, and digital training to scaffold SRL skill development and enactment. We examined two theory-aligned designs of digital skill trainings that scaffold SRL and how students’ demonstration of metacognitive knowledge of learning skills predicted exam performance in biology courses where training took place. In Study 1, students’ (n = 49) responses to training activities were scored for quality and summed by training topic and level of understanding. Behavioral and environmental regulation knowledge predicted midterm and final exam grades; knowledge of SRL processes did not. Declarative and conceptual levels of skill-mastery predicted exam performance; application-level knowledge did not. When modeled by topic at each level of understanding, declarative knowledge of behavioral and environmental regulation and conceptual knowledge of cognitive strategies predicted final exam performance. In Study 2 (n = 62), knowledge demonstrated during a redesigned video-based multimedia version of behavioral and environmental regulation again predicted biology exam performance. Across studies, performance on training activities designed in alignment with skill-training models predicted course performances and predictions were sustained in a redesign prioritizing learning efficiency. Training learners’ SRL skills –and specifically cognitive strategies and environmental regulation– benefited their later biology course performances across studies, which demonstrate the value of providing brief, digital activities to develop learning skills. Ongoing refinement to materials designed to develop metacognitive processing and learners’ ability to apply skills in new contexts can increase benefits.

     
    more » « less
  2. Abstract

    Intelligent tutoring systems (ITSs) incorporate pedagogical agents (PAs) to scaffold learners' self‐regulated learning (SRL) via prompts and feedback to promote learners' monitoring and regulation of their cognitive, affective, metacognitive and motivational processes to achieve their (sub)goals. This study examines PAs' effectiveness in scaffolding and teaching SRL during learning with MetaTutor, an ITS on the human circulatory system. Undergraduates (N = 118) were randomly assigned to a condition:Control Condition(i.e. learners could only self‐initiate SRL strategies) andPrompt and Feedback Condition(i.e. PAs prompted learners to engage in SRL). Learners' log‐file data captured when strategies were used, the initiator of the strategy (i.e. learner and PA), and the relevance of instructional content pages in relation to learner subgoals. While results showed that PAs were effective scaffolders of SRL in which they prompted learners to engage in SRL strategies more when content was relevant towards their subgoals and as time on page and task increased, there were mixed findings about the effectiveness of PAs as teachers of SRL. Findings show how production rules guiding PA prompts can improve their scaffolding and teaching of SRL across the learning task – through contextualizing SRL strategies to the instructional content and in relation to the relevance of the content to learners' subgoals.Practitioner notesWhat is already known about this topic

    Most learners struggle to efficiently and effectively use self‐regulated learning (SRL) strategies to attain goals and subgoals.

    There is a need for SRL to be scaffolded for learners to manage multiple goals and subgoals while learning about complex STEM topics.

    Intelligent tutoring systems (ITSs) typically incorporate pedagogical agents (PAs) to prompt learners to engage in SRL strategy and provide feedback.

    There are mixed findings on the effectiveness of PAs in scaffolding learners' SRL.

    What this paper adds

    We consider PAs not only scaffolders but also teachers of SRL.

    Results showed that while PAs encouraged the use of SRL strategies when the content was relevant to subgoals, they did not discourage the use of SRL strategies when the content was not relevant.

    Results for this study were mixed in their support of PAs as teachers of SRL.

    Learners increasingly depended on PAs to prompt SRL strategies as time on task progressed.

    Implications for practice and/or policy

    PAs are effective scaffolders of SRL with more research needed to understand their role as teachers of SRL.

    PA scaffolding is more essential as time on task progresses.

    When deploying specific cognitive and metacognitive SRL strategies, the relevance of the content to learners' subgoals should be taken into account.

     
    more » « less
  3. Michalsky, Tova ; Moos, Daniel (Ed.)
    Teachers’ ability to self-regulate their own learning is closely related to their competency to enhance self-regulated learning (SRL) in their students. Accordingly, there is emerging research for the design of teacher dashboards that empower instructors by providing access to quantifiable evidence of student performance and SRL processes. Typically, they capture evidence of student learning and performance to be visualized through activity traces (e.g., bar charts showing correct and incorrect response rates, etc.) and SRL data (e.g., eye-tracking on content, log files capturing feature selection, etc.) in order to provide teachers with monitoring and instructional tools. Critics of the current research on dashboards used in conjunction with advanced learning technologies (ALTs) such as simulations, intelligent tutoring systems, and serious games, argue that the state of the field is immature and has 1) focused only on exploratory or proof-of-concept projects, 2) investigated data visualizations of performance metrics or simplistic learning behaviors, and 3) neglected most theoretical aspects of SRL including teachers’ general lack of understanding their’s students’ SRL. Additionally, the work is mostly anecdotal, lacks methodological rigor, and does not collect critical process data (e.g. frequency, duration, timing, or fluctuations of cognitive, affective, metacognitive, and motivational (CAMM) SRL processes) during learning with ALTs used in the classroom. No known research in the areas of learning analytics, teacher dashboards, or teachers’ perceptions of students’ SRL and CAMM engagement has systematically and simultaneously examined the deployment, temporal unfolding, regulation, and impact of all these key processes during complex learning. In this manuscript, we 1) review the current state of ALTs designed using SRL theoretical frameworks and the current state of teacher dashboard design and research, 2) report the important design features and elements within intelligent dashboards that provide teachers with real-time data visualizations of their students’ SRL processes and engagement while using ALTs in classrooms, as revealed from the analysis of surveys and focus groups with teachers, and 3) propose a conceptual system design for integrating reinforcement learning into a teacher dashboard to help guide the utilization of multimodal data collected on students’ and teachers’ CAMM SRL processes during complex learning. 
    more » « less
  4. Many historically minoritized graduate students, here defined as Women, Latinx and Black/African American students, in Science, Technology, Engineering and Math (STEM) experience unwelcome or even hostile ecosystems or environments. Many of the social expectations are that historically minoritized graduate students in STEM should assimilate or acclimate to the cultural, where assimilation/acclimation are defined as cultural conformation vs. social acceptance of a student authentic self/identity. They may also experience forms of continuous microaggressions and isolation. The effects of chronic external stressors, such as experiencing discrimination and social isolation, on increased mental health disorders and decreased physiological health is well known [1-3]. Yet, evidence-based practices of support systems specifically for graduate students from historically marginalized communities to reduce the effects of climates of intimidation are not common. Indeed, researchers have found that such students “would benefit if colleges and universities attempted to deconstruct climates of intimidation [4]” and it has also been shown that teaching underrepresented minority students empowerment skills can improve academic success [5]. Self-advocacy originates from the American Counseling Association (ACA) and the Learning Disabilities (LD) communities for effective counseling that promotes academic success and is based on a social justice framework [6]. The underlying principle of self-advocacy is that supporting skills and knowledge development in the three areas of self-advocacy leads to a student’s long term participation and ultimately academic success in areas such as post-secondary education and STEM. The pillars of the self-advocacy program are centered on (i) Empowerment, (ii) Promoting self-awareness and (iii) Social Justice and programming in the GRaduate Education for Academically Talented Students (GREATS) is aligned and repeated along these three pillars. The current professional development program is in its third year of implementation and to date twenty-seven students have participated in the program. This work in progress paper outlines the evaluation of a self-advocacy program for historically marginalized graduate students in STEM at the University of Illinois Chicago is a minority serving institution as both an Hispanic Serving Institution (HSI) and an Asian American Native American Pacific Islander Serving Institution (AANAPISI). [1] S. Stansfeld and B. Candy, "Psychosocial work environment and mental health--a meta-analytic review," ed, 2006. [2] E. M. Smith, "Ethnic minorities: Life stress, social support, and mental health issues," The Counseling Psychologist, vol. 13, no. 4, pp. 537-579, 1985. [3] D. M. Frost, K. Lehavot, and I. H. Meyer, "Minority stress and physical health among sexual minority individuals," Journal of behavioral medicine, vol. 38, no. 1, pp. 1-8, 2015. [4] R. T. Palmer, D. C. Maramba, and T. E. Dancy, "A Qualitative Investigation of Factors Promoting the Retention and Persistence of Students of Color in STEM," The Journal of Negro Education, vol. 80, no. 4, pp. 491-504, 2011. [Online]. Available: http://www.jstor.org/stable/41341155. [5] A. R. Dowden, "Implementing Self-Advocacy Training Within a Brief Psychoeducational Group to Improve the Academic Motivation of Black Adolescents," The Journal for Specialists in Group Work, vol. 34, no. 2, pp. 118-136, 2009/04/28 2009, doi: 10.1080/01933920902791937. 
    more » « less
  5. As society increasingly relies on digital technologies in many different aspects, those who lack relevant access and skills are lagging increasingly behind. Among the underserved groups disproportionately affected by the digital divide are women who are transitioning from incarceration and seeking to reenter the workforce outside the carceral system (women-in-transition). Women-in-transition rarely have been exposed to sound technology education, as they have generally been isolated from the digital environment while in incarceration. Furthermore, while women have become the fastest-growing segment of the incarcerated population in the United States in recent decades, prison education and reentry programs are still not well adjusted for them. Most programs are mainly designed for the dominant male population. Consequently, women-in-transition face significant post-incarceration challenges in accessing and using relevant digital technologies and thus have added difficulties in entering or reentering the workforce. Against this backdrop, our multi-disciplinary research team has conducted empirical research as part of technology education offered to women-in-transition in the Midwest. In this article, we report results from our interviews with 75 women-in-transition in the Midwest that were conducted to develop a tailored technology education program for the women. More than half of the participants in our study are women of color and face precarious housing and financial situations. Then, we discuss principles that we adopted in developing our education program for the marginalized women and participants’ feedback on the program. Our team launched in-person sessions with women-in-reentry at public libraries in February 2020 and had to move the sessions online in March due to COVID-19. Our research-informed educational program is designed primarily to support the women in enhancing their knowledge and comfort with technology and nurturing computational thinking. Our study shows that low self-efficacy and mental health challenges, as well as lack of resources for technology access and use, are some of the major issues that need to be addressed in supporting technology learning among women-in-transition. This research offers scholarly and practical implications for computing education for women-in-transition and other marginalized populations. 
    more » « less