An official website of the United States government
Students who take an online course, such as a MOOC, use the course's discussion forum to ask questions or reach out to instructors when encountering an issue. However, reading and responding to students' questions is difficult to scale because of the time needed to consider each message. As a result, critical issues may be left unresolved, and students may lose the motivation to continue in the course. To help address this problem, we build predictive models that automatically determine the urgency of each forum post, so that these posts can be brought to instructors' attention. This paper goes beyond previous work by predicting not just a binary decision cut-off but a post's level of urgency on a 7-point scale. First, we train and cross-validate several models on an original data set of 3,503 posts from MOOCs at University of Pennsylvania. Second, to determine the generalizability of our models, we test their performance on a separate, previously published data set of 29,604 posts from MOOCs at Stanford University. While the previous work on post urgency used only one data set, we evaluated the prediction across different data sets and courses. The best-performing model was a support vector regressor trained on the Universal Sentence Encoder embeddings of the posts, achieving an RMSE of 1.1 on the training set and 1.4 on the test set. Understanding the urgency of forum posts enables instructors to focus their time more effectively and, as a result, better support student learning.
more »
« less
Seeding Course Forums using the Teacher-in-the-Loop
Online forums are an integral part of modern day courses, but motivating students to participate in educationally beneficial discussions can be challenging. Our proposed solution is to initialize (or “seed”) a new course forum with comments from past instances of the same course that are intended to trigger discussion that is beneficial to learning. In this work, we develop methods for selecting high-quality seeds and evaluate their impact over one course instance of a 186-student biology class. We designed a scale for measuring the “seeding suitability” score of a given thread (an opening comment and its ensuing discussion). We then constructed a supervised machine learning (ML) model for predicting the seeding suitability score of a given thread. This model was evaluated in two ways: first, by comparing its performance to the expert opinion of the course instructors on test/holdout data; and second, by embedding it in a live course, where it was actively used to facilitate seeding by the course instructors. For each reading assignment in the course, we presented a ranked list of seeding recommendations to the course instructors, who could review the list and filter out seeds with inconsistent or malformed content. We then ran a randomized controlled study, in which one group of students was shown seeds that were recommended by the ML model, and another group was shown seeds that were recommended by an alternative model that ranked seeds purely by the length of discussion that was generated in previous course instances. We found that the group of students that received posts from either seeding model generated more discussion than a control group in the course that did not get seeded posts. Furthermore, students who received seeds selected by the ML-based model showed higher levels of engagement, as well as greater learning gains, than those who received seeds ranked by length of discussion.
more »
« less
- Award ID(s):
- 1915724
- PAR ID:
- 10289680
- Date Published:
- Journal Name:
- LAK21: 11th International Learning Analytics and Knowledge Conference
- Page Range / eLocation ID:
- 22 to 31
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Mitrovic, A.; Bosch, N. (Ed.)Emoji are commonly used in social media to convey attitudes and emotions. While popular, their use in educational contexts has been sparsely studied. This paper reports on the students’ use of emoji in an online course forum in which students annotate and discuss course material in the margins of the online textbook. For this study, instructors created 11 custom emoji-hashtag pairs that enabled students to quickly communicate affects and reactions in the forum that they experienced while interacting with the course material. Example reporting includes, inviting discussion about a topic, declaring a topic as interesting, or requesting assistance about a topic. We analyze emoji usage by over 1,800 students enrolled in multiple offerings of the same course across multiple academic terms. The data show that some emoji frequently appear together in posts associated with the same paragraphs, suggesting that students use the emoji in this way to communicating complex affective states. We explore the use of computational models for predicting emoji at the post level, even when posts are lacking emoji. This capability can allow instructors to infer information about students’ affective states during their ”at home” interactions with course readings. Finally, we show that partitioning the emoji into distinct groups, rather than trying to predict individual emoji, can be both of pedagogical value to instructors and improve the predictive performance of our approach using the BERT language model. Our procedure can be generalized to other courses and for the benefit of other instructors.more » « less
-
Preparing Pre-Service Teacher Candidates for the Praxis Exam: An Innovative Model of Blended Supportnull (Ed.)The expansion of K-12 computer science (CS) has driven a dramatic need for educators who are trained in CS content and pedagogy [1]. This poster describes our effort to train teacher candidates (i.e., pre-service teachers who are students seeking degrees within a College of Education), who are specializing in secondary mathematics education, to be future CS educators. We specifically describe our collaboration to provide a blended preparatory six-week training for the ETS CS Praxis exam (5652), assisting our pre-service students in satisfying the CS certification requirements in our state before they graduate and begin their professional teaching career. Given the unique challenges of pre-service CS teacher preparation [2], blended models, which combine both in-person and online instruction, are an effective approach to building a pre-service program. Within our pre-service CS program, students first complete a two-course pathway that prepares them in AP CSP content and pedagogy experiences, including observations in local AP CSP classrooms [3]. After completing the two courses, our students participate in the blended version of the WeTeach_CS Praxis preparation course to achieve certification. The in-person support provided by the blended model contributed significantly to certification success in this project. With a cut-score of 149 for the Praxis exam, all 11 of our pre-service students who completed the course received a passing score (including one student with a perfect score of 200, and another student with a 195); the average score for our pre-service students was 175. An additional 11 in-service teachers, with diverse backgrounds in CS content knowledge, also participated in the blended Praxis preparation course, with an average score of 166. Given the unique challenges of pre-service CS teacher preparation, university pre-service CS teacher programs should look to innovative models of teacher support developed by in-service programs to make substantial gains in CS teacher certification. Incorporating an asynchronous online course that allows teachers with a wide range of prior experience in CS to learn at their own pace with in-person coursework and support appears to be a viable model for assisting non-CS major teacher candidates in achieving a CS certification. With the blended model, even teachers with no background knowledge in CS were successful. Within our pre-service CS program, students first complete a two-course pathway that prepares them in AP CSP content and pedagogy experiences, including observations in local AP CSP classrooms [3]. After completing the two courses, our students participate in the blended version of the WeTeach_CS Praxis preparation course to achieve certification. The in-person support provided by the blended model contributed significantly to certification success in this project. With a cut-score of 149 for the Praxis exam, all 11 of our pre-service students who completed the course received a passing score (including one student with a perfect score of 200, and another student with a 195); the average score for our pre-service students was 175. An additional 11 in-service teachers, with diverse backgrounds in CS content knowledge, also participated in the blended Praxis preparation course, with an average score of 166. Incorporating an asynchronous online course that allows teachers with a wide range of prior experience in CS to learn at their own pace with in-person coursework and support appears to be a viable model for assisting non-CS major teacher candidates in achieving a CS certification. With the blended model, even teachers with no background knowledge in CS were successful.more » « less
-
The purpose of this work was to test the inter-rater reliability (IRR) of a rubric used to grade technical reports in a senior-level chemical engineering laboratory course that has multiple instructors that grade deliverables. The rubric consisted of fifteen constructs that provided students detailed guidance on instructor expectations with respect to the report sections, formatting and technical writing aspects such as audience, context and purpose. Four student reports from previous years were scored using the rubric, and IRR was assessed using a two-way mixed, consistency, average-measures intraclass correlation (ICC) for each construct. Then, the instructors met as a group to discuss their scoring and reasoning. Multiple revisions were made to the rubric based on instructor feedback and constructs rated by ICC as poor. When fair or poor constructs were combined, the ICCs improved. In addition, the overall score construct continued to be rated as excellent, indicating that while different instructors may have variation at the individual construct level, they evaluate the overall quality of the report consistently. A key learning from this process was the importance of the instructor discussion around their reasoning for the scores and the importance of an ‘instructor orientation’ involving discussion and practice using the rubrics in the case of multiple instructors or a change in instructors. The developed rubric has the potential for broad applicability to engineering laboratory courses with technical writing components and could be adapted for alternative styles of technical writing genre.more » « less
-
Frank, B. W.; Jones, D. L.; and Ryan, Q. X. (Ed.)In this paper, we analyze video recordings of students working on tutorials in Zoom breakout rooms in an upper-division quantum mechanics course. We investigate group behaviors in this virtual environment, including the effects of instructor presence. To this end, we modify the Color Frames coding scheme introduced by Scherr to suit the virtual nature of the interactions. By broadening the frames and allowing for multiple overlapping frames, we are able to describe some group behaviors not otherwise captured. For example, in some instances, students take on an authoritative role in the group, and in other instances, groups engage in overtly casual behavior while nonetheless having on-topic discussions. We observe significant variation in how much time each group spends in each frame, but find that all groups spend some time in all frames. Instructors can be present without dominating or eliminating discussion between students, and their presence need not significantly impact the time students spent in an "informal/friendly'' frame. However, instructor presence significantly reduces time spent working individually. Our findings will support additional research into the dynamics of student discussions during tutorials and aid ongoing development of online tutorials that can, e.g., be assigned for use outside of class.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3448139.3448142
