Students who take an online course, such as a MOOC, use the course's discussion forum to ask questions or reach out to instructors when encountering an issue. However, reading and responding to students' questions is difficult to scale because of the time needed to consider each message. As a result, critical issues may be left unresolved, and students may lose the motivation to continue in the course. To help address this problem, we build predictive models that automatically determine the urgency of each forum post, so that these posts can be brought to instructors' attention. This paper goes beyond previous work by predicting not just a binary decision cut-off but a post's level of urgency on a 7-point scale. First, we train and cross-validate several models on an original data set of 3,503 posts from MOOCs at University of Pennsylvania. Second, to determine the generalizability of our models, we test their performance on a separate, previously published data set of 29,604 posts from MOOCs at Stanford University. While the previous work on post urgency used only one data set, we evaluated the prediction across different data sets and courses. The best-performing model was a support vector regressor trained on the Universal Sentence Encoder embeddings of the posts, achieving an RMSE of 1.1 on the training set and 1.4 on the test set. Understanding the urgency of forum posts enables instructors to focus their time more effectively and, as a result, better support student learning.
more »
« less
Seeding Course Forums using the Teacher-in-the-Loop
Online forums are an integral part of modern day courses, but motivating students to participate in educationally beneficial discussions can be challenging. Our proposed solution is to initialize (or “seed”) a new course forum with comments from past instances of the same course that are intended to trigger discussion that is beneficial to learning. In this work, we develop methods for selecting high-quality seeds and evaluate their impact over one course instance of a 186-student biology class. We designed a scale for measuring the “seeding suitability” score of a given thread (an opening comment and its ensuing discussion). We then constructed a supervised machine learning (ML) model for predicting the seeding suitability score of a given thread. This model was evaluated in two ways: first, by comparing its performance to the expert opinion of the course instructors on test/holdout data; and second, by embedding it in a live course, where it was actively used to facilitate seeding by the course instructors. For each reading assignment in the course, we presented a ranked list of seeding recommendations to the course instructors, who could review the list and filter out seeds with inconsistent or malformed content. We then ran a randomized controlled study, in which one group of students was shown seeds that were recommended by the ML model, and another group was shown seeds that were recommended by an alternative model that ranked seeds purely by the length of discussion that was generated in previous course instances. We found that the group of students that received posts from either seeding model generated more discussion than a control group in the course that did not get seeded posts. Furthermore, students who received seeds selected by the ML-based model showed higher levels of engagement, as well as greater learning gains, than those who received seeds ranked by length of discussion.
more »
« less
- Award ID(s):
- 1915724
- NSF-PAR ID:
- 10289680
- Date Published:
- Journal Name:
- LAK21: 11th International Learning Analytics and Knowledge Conference
- Page Range / eLocation ID:
- 22 to 31
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract Capturing evidence for dynamic changes in self‐regulated learning (SRL) behaviours resulting from interventions is challenging for researchers. In the current study, we identified students who were likely to do poorly in a biology course and those who were likely to do well. Then, we randomly assigned a portion of the students predicted to perform poorly to a science of learning to learn intervention where they were taught SRL study strategies. Learning outcome and log data (257 K events) were collected from
n = 226 students. We used a complex systems framework to model the differences in SRL including the amount, interrelatedness, density and regularity of engagement captured in digital trace data (ie, logs). Differences were compared between students who were predicted to (1) perform poorly (control,n = 48), (2) perform poorly and received intervention (treatment,n = 95) and (3) perform well (not flagged,n = 83). Results indicated that the regularity of students' engagement was predictive of course grade, and that the intervention group exhibited increased regularity in engagement over the control group immediately after the intervention and maintained that increase over the course of the semester. We discuss the implications of these findings in relation to the future of artificial intelligence and potential uses for monitoring student learning in online environments.Practitioner notes What is already known about this topic
Self‐regulated learning (SRL) knowledge and skills are strong predictors of postsecondary STEM student success.
SRL is a dynamic, temporal process that leads to purposeful student engagement.
Methods and metrics for measuring dynamic SRL behaviours in learning contexts are needed.
What this paper adds
A Markov process for measuring dynamic SRL processes using log data.
Evidence that dynamic, interaction‐dominant aspects of SRL predict student achievement.
Evidence that SRL processes can be meaningfully impacted through educational intervention.
Implications for theory and practice
Complexity approaches inform theory and measurement of dynamic SRL processes.
Static representations of dynamic SRL processes are promising learning analytics metrics.
Engineered features of LMS usage are valuable contributions to AI models.
-
Instructor-led presentation-based teaching mainly focuses on delivering content. Whereas student active presentations-based teaching approaches require students to take leadership in learning actions. Many teaching and learning strategies were adopted to foster active student participation during in-class learning activities. We developed the student presentation-based effective teaching (SPET) approach in 2014 to make student presentation activity the central element of learning challenging concepts. We have developed several versions to meet the need for teaching small classes (P. Tyagi, "Student Presentation Based Effective Teaching (SPET) Approach for Advanced Courses," in ASME IMECE 2016-66029, V005T06A026), large enrolment classes (P. Tyagi, "Student Presentation Based Teaching (SPET) Approach for Classes With Higher Enrolment," ASME IMECE 2018-88463, V005T07A035), and online teaching during COVID-19. (P. Tyagi, "Second Modified Student Presentation Based Effective Teaching (SPET) Method Tested in COVID-19 Affected Senior Level Mechanical Engineering Course," in ASME IMECE 2020-23615, V009T09A026). The SPET approach has successfully engaged students with varied interests and competence levels in the learning process. SPET approach has also made it possible to cover new topics such as training engineering students about positive intelligence skills to foster lifelong learning aptitude and doing engineering projects in a group setting. However, it was noted that many students who were overwhelmed with parallel academic demands in other courses and different activities were underperforming via SPET-based learning strategies. SPET core functioning depends on the following steps: Step 1: Provide a set of conceptual and topical questions for students to answer individually after self-education from the recommended textbook or course material, Step-2: Group presentations are prepared by the prepared students for in-class discussion, Step-3: Group makes a presentation in class 1-2 weeks after the day of the assignment to seek instructor feedback and to do peer discussion. The instructor noted that students unfamiliar with the new concepts and terminologies in the SPET assignment struggled to respond to questions individually and contribute to the group discussion based on their presentation. Several motivated students who invested time in familiarizing new concepts and terminologies met or exceeded the expectations. However, a significant student population struggled. To alleviate this issue author has implemented a further improvement in SPET approach. This paper reports teaching experiments conducted in MECH 487 Photovoltaic Cells and Solar Thermal Energy System and MECH 462 Design of Energy Systems course. This improvement requires augmenting SPET with instructor-led concept familiarization discussion on the day of issuing the assignment or close to that; for this step instructor utilized exemplary student work from prior SPET-based teaching of the same course. In the survey, many students expressed their views about the improvement and reported introductory discussions were helpful and addressed several reservations and impediments students encountered. This paper will discuss the structure of the new improvement strategy and outcomes-including student feedback and comments.more » « less
-
null (Ed.)In recent years, studies in engineering education have begun to intentionally integrate disability into discussions of diversity, inclusion, and equity. To broaden and advocate for the participation of this group in engineering, researchers have identified a variety of factors that have kept people with disabilities at the margins of the field. Such factors include the underrepresentation of disabled individuals within research and industry; systemic and personal barriers, and sociocultural expectations within and beyond engineering education-related contexts. These findings provide a foundational understanding of the external and environmental influences that can shape how students with disabilities experience higher education, develop a sense of belonging, and ultimately form professional identities as engineers. Prior work examining the intersections of disability identity and professional identity is limited, with little to no studies examining the ways in which students conceptualize, define, and interpret disability as a category of identity during their undergraduate engineering experience. This lack of research poses problems for recruitment, retention, and inclusion, particularly as existing studies have shown that the ways in which students perceive and define themselves in relation to their college major is crucial for the development of a professional engineering identity. Further, due to variation in defining ‘disability’ across national agencies (e.g., the National Institutes of Health, and the Department of Justice) and disability communities (with different models of disability), the term “disability” is broad and often misunderstood, frequently referring to a group of individuals with a wide range of conditions and experiences. Therefore, the purpose of this study is to gain deeper insights into the ways students define disability and disability identity within their own contexts as they develop professional identities. Specifically, we ask the following research question: How do students describe and conceptualize non-apparent disabilities? To answer this research question, we draw from emergent findings from an on-going grounded theory exploration of professional identity formation of undergraduate civil engineering students with disabilities. In this paper, we focus our discussion on the grounded theory analyses of 4 semi-structured interviews with participants who have disclosed a non-apparent disability. Study participants consist of students currently enrolled in undergraduate civil engineering programs, students who were initially enrolled in undergraduate civil engineering programs and transferred to another major, and students who have recently graduated from a civil engineering program within the past year. Sensitizing concepts emerged as findings from the initial grounded theory analysis to guide and initiate our inquiry: 1) the medical model of disability, 2) the social model of disability, and 3) personal experience. First, medical models of disability position physical, cognitive, and developmental difference as a “sickness” or “condition” that must be “treated”. From this perspective, disability is perceived as an impairment that must be accommodated so that individuals can obtain a dominantly-accepted sense of normality. An example of medical models within the education context include accommodations procedures in which students must obtain an official diagnosis in order to access tools necessary for academic success. Second, social models of disability position disability as a dynamic and fluid identity that consists of a variety of physical, cognitive, or developmental differences. Dissenting from assumptions of normality and the focus on individual bodily conditions (hallmarks of the medical model), the social model focuses on the political and social structures that inherently create or construct disability. An example of a social model within the education context includes the universal design of materials and tools that are accessible to all students within a given course. In these instances, students are not required to request accommodations and may, consequently, bypass medical diagnoses. Lastly, participants referred to their own life experiences as a way to define, describe, and consider disability. Fernando considers his stutter to be a disability because he is often interrupted, spoken over, or silenced when engaging with others. In turn, he is perceived as unintelligent and unfit to be a civil engineer by his peers. In contrast, David, who identifies as autistic, does not consider himself to be disabled. These experiences highlight the complex intersections of medical and social models of disability and their contextual influences as participants navigate their lives. While these sensitizing concepts are not meant to scope the research, they provide a useful lens for initiating research and provides markers on which a deeper, emergent analysis is expanded. Findings from this work will be used to further explore the professional identity formation of undergraduate civil engineering students with disabilities. These findings will provide engineering education researchers and practitioners with insights regarding the ways individuals with disabilities interpret their in- and out-of-classroom experiences and navigate their disability identities. For higher education, broadly, this work aims to reinforce the complex and diverse nature of disability experience and identity, particularly as it relates to accommodations and accessibility within the classroom, and expand the inclusiveness of our programs and institutions.more » « less
-
Asynchronous online courses are popular because they offer benefits to both students and instructors. Students benefit from the convenience, flexibility, affordability, freedom of geography, and access to information. Instructors and institutions benefit by having a broad geographical reach, scalability, and cost-savings of no physical classroom. A challenge with asynchronous online courses is providing students with engaging, collaborative and interactive experiences. Here, we describe how an online poster symposium can be used as a unique educational experience and assessment tool in a large-enrollment (e.g., 500 students), asynchronous, natural science, general education (GE) course. The course, Introduction to Environmental Science (ENR2100), was delivered using distance education (DE) technology over a 15-week semester. In ENR2100 students learn a variety of topics including freshwater resources, surface water, aquifers, groundwater hydrology, ecohydrology, coastal and ocean circulation, drinking water, water purification, wastewater treatment, irrigation, urban and agricultural runoff, sediment and contaminant transport, water cycle, water policy, water pollution, and water quality. Here we present a is a long-term study that takes place from 2017 to 2022 (before and after COVID-19) and involved 5,625 students over 8 semesters. Scaffolding was used to break up the poster project into smaller, more manageable assignments, which students completed throughout the semester. Instructions, examples, how-to videos, book chapters and rubrics were used to accommodate Students’ different levels of knowledge. Poster assignments were designed to teach students how to find and critically evaluate sources of information, recognize the changing nature of scientific knowledge, methods, models and tools, understand the application of scientific data and technological developments, and evaluate the social and ethical implications of natural science discoveries. At the end of the semester students participated in an asynchronous online poster symposium. Each student delivered a 5-min poster presentation using an online learning management system and completed peer reviews of their classmates’ posters using a rubric. This poster project met the learning objectives of our natural science, general education course and taught students important written, visual and verbal communication skills. Students were surveyed to determine, which parts of the course were most effective for instruction and learning. Students ranked poster assignments first, followed closely by lectures videos. Approximately 87% of students were confident that they could produce a scientific poster in the future and 80% of students recommended virtual poster symposiums for online courses.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3448139.3448142
