This content will become publicly available on July 1, 2024
Title: Too long to wait and not much to do: Modeling student behaviors while waiting for help in online office hours.
Promptly addressing students’ help requests on their programming assignments has become more and more challenging in computer science education. Since the pandemic, most instructors use online office hours to answer questions. Prior studies have shown increased student participation with online office hours. This popularity has led to significantly longer wait times in the office hours queue, and various strategies for selecting the next student to help may impact wait time. For example, prioritizing students who have not been seen on the day of the deadline will extend the wait time for students who are frequently rejoining the queue. To better understand this problem, we explored students’ behavior when they are waiting in the queue. We investigate the amount of time students are willing to wait in the queue by modeling the distribution of cancellation time. We find that after waiting for 49 minutes, most students will cancel their help request. Then, we looked at students’ coding actions during the waiting period and found that only 21% of students have commits while waiting. Surprisingly, students who waited for hours did not commit their work for automated feedback. Our findings suggest that time in the queue should be considered in addition to other factors like last interaction when selecting the next student to help during office hours to minimize canceled interactions. more »« less
As Computer Science has increased in popularity so too have class sizes and demands on faculty to provide support. It is therefore more important than ever for us to identify new ways to triage student questions, identify common problems, target students who need the most help, and better manage instructors’ time. By analyzing interaction data from office hours we can identify common patterns, and help to guide future help-seeking. My Digital Hand (MDH) is an online ticketing system that allows students to post help requests, and for instructors to prioritize support and track common issues. In this research, we have collected and analyzed a corpus of student questions from across six semesters of a CS2 with a focus on object-oriented programming course [17]. As part of this work, we grouped the interactions into five categories, analyzed the distribution of help requests, balanced the categories by Synthetic Minority Oversampling Technique (SMOTE) , and trained an automatic classifier based upon LightGBM to automatically classify student requests. We found that over 69% of the questions were unclear or barely specified. We proved the stability of the model across semesters through leave one out cross-validation and the target model achieves an accuracy of 91.8%. Finally, we find that online office hours can provide more help for more students.
Demand for education in Computer Science has increased markedly in recent years. With increased demand has come to an increased need for student support, especially for courses with large programming projects. Instructors commonly provide online post forums or office hours to address this massive demand for help requests. Identifying what types of questions students are asking in those interactions and what triggers their help requests can in turn assist instructors in better managing limited help-providing resources. In this study, we aim to explore students’ help-seeking actions from the two separate approaches we mentioned before and investigate their coding actions before help requests to understand better what motivates students to seek help in programming projects. We collected students’ help request data and commit logs from two Fall offerings of a CS2 course. In our analysis, we first believe that different types of questions should be related to different behavioral patterns. Therefore, we first categorized students’ help requests based on their content (e.g., Implementation, General Debugging, or Addressing Teaching Staff (TS) Test Failures). We found that General Debugging is the most frequently asked question. Then we analyzed how the popularity of each type of request changed over time. Our results suggest that implementation is more popular in the early stage of the project cycle, and it changes to General Debugging and Addressing TS Failures in the later stage. We also calculated the accuracy of students’ commit frequency one hour before their help requests; the results show that before Implementation requests, the commit frequency is significantly lower, and before TS failure requests, the frequency is significantly higher. Moreover, we checked before any help request whether students changed their source code or test code. The results show implementation requests related to higher chances of source code changes and coverage questions related to more test code changes. Moreover, we use a Markov Chain model to show students’ action sequences before, during, and after the requests. And finally, we explored students’ progress after the office hours interaction and found that over half of the students improved the correctness of their code after 20 minutes of their office hours interaction addressing TS failures ends.
POSTER. Presented at the Symposium (9/12/2019)
Abstract:
The Academy of Engineering Success (AcES) employs literature-based, best practices to support and retain underrepresented students in engineering through graduation with the ultimate goal of diversifying the engineering workforce. AcES was established in 2012 and has been supported via NSF S-STEM award number 1644119 since 2016. The 2016, 2017, and 2018 cohorts consist of 12, 20, and 22 students, respectively. Five S-STEM supported scholarships were awarded to the 2016 cohort, seven scholarships were awarded to students from the 2017 cohort, and six scholarships were awarded to students from the 2018 cohort.
AcES students participate in a one-week summer bridge experience, a common fall semester course focused on professional development, and a common spring semester course emphasizing the role of engineers in societal development. Starting with the summer bridge experience, and continuing until graduation, students are immersed in curricular and co-curricular activities with the goals of fostering feelings of institutional inclusion and belonging in engineering, providing academic support and student success skills, and professional development. The aforementioned goals are achieved by providing (1) opportunities for faculty-student, student-student, and industry mentor-student interaction, (2) academic support, and student success education in areas such as time management and study skills, and (3) facilitated career and major exploration.
Four research questions are being examined, (1) What is the relationship between participation in the AcES program and participants’ academic success?, (2) What aspects of the AcES program most significantly impact participants’ success in engineering, (3) How do AcES students seek to overcome challenges in studying engineering, and (4) What is the longitudinal impact of the AcES program in terms of motivation, perceptions, feelings of inclusion, outcome expectations of the participants and retention?
Students enrolled in the AcES program participate in the GRIT, LAESE, and MSLQ surveys, focus groups, and one-on-one interviews at the start and end of each fall semester and at the end of the spring semester. The surveys provide a measure of students’ GRIT, general self-efficacy, engineering self-efficacy, test anxiety, math outcome efficacy, intrinsic value of learning, inclusion, career expectations, and coping efficacy. Focus group and interview responses are analyzed in order to answer research questions 2, 3, and 4. Survey responses are analyzed to answer research question 4, and institutional data such as GPA is used to answer research question 1.
An analysis of the 2017 AcES cohort survey responses produced a surprising result. When the responses of AcES students who retained were compared to the responses of AcES students who left engineering, those who left engineering had higher baseline values of GRIT, career expectations, engineering self-efficacy, and math outcome efficacy than those students who retained. A preliminary analysis of the 2016, 2017, and 2018 focus group and one-on-one interview responses indicates that the Engineering Learning Center, tutors, organized out of class experiences, first-year seminar, the AcES cohort, the AcES summer bridge, the AcES program, AcES Faculty/Staff, AcES guest lecturers, and FEP faculty/Staff are viewed as valuable by students and cited with contributing to their success in engineering. It is also evident that AcES students seek help from peers, seek help from tutors, use online resources, and attend office hours to overcome their challenges in studying engineering.
Undergraduate teaching assistants (UTAs) office hours are an approachable way for students to get help, but little is known about why and for what do the students choose to attend office hours. We sought to understand what kind of help the students believe they need by analyzing the problem-solving step students self-reported when joining the office hours queue app. We used the UPIC framework to aggregate course specific problem-solving steps to enable comparing between seven data sets from a CS1 and a data science course across four semesters. We then compared the class-level and student-level phase distributions to understand the differences between the two courses and the two levels in the courses. We found most students have a "primary phase" where a majority of their interactions fall, and there are significant individual differences in their phase distributions. Moreover, we did not find either students' demographics or the context of their first visits to significantly impact their individual differences in the phase distributions, suggesting students may have fixed beliefs on how to approach office hours. Finally, a strong majority of interactions happen within 3 days of the deadline, such that the UPIC distribution for those days looks like the class-level phase distribution.
Motivation: This is a complete paper. There was a sudden shift from traditional learning to online learning in Spring 2020 with the outbreak of COVID-19. Although online learning is not a new topic of discussion, universities, faculty, and students were not prepared for this sudden change in learning. According to a recent article in ‘The Chronicle of Higher Education, “even under the best of circumstances, virtual learning requires a different, carefully crafted approach to engagement”. The Design Thinking course under study is a required freshmen level course offered in a Mid-western University. The Design Thinking course is offered in a flipped format where all the content to be learned is given to students beforehand and the in-class session is used for active discussions and hands-on learning related to the content provided at the small group level. The final learning objective of the course is a group project where student groups are expected to come up with functional prototypes to solve a real-world problem following the Design Thinking process. There were eighteen sections of the Design Thinking course offered in Spring 2020, and with the outbreak of COVID-19, a few instructors decided to offer synchronous online classes (where instructors were present online during class time and provided orientation and guidance just like a normal class) and a few others decided to offer asynchronous online classes (where orientation from the instructor was delivered asynchronous and the instructor was online during officially scheduled class time but interactions were more like office hours). Students were required to be present synchronously at the team level during the class time in a synchronous online class. In an asynchronous online class, students could be synchronous at the team level to complete their assignment any time prior to the deadline such that they could work during class time but they were not required to work at that time. Through this complete paper, we are trying to understand student learning, social presence and learner satisfaction with respect to different modes of instruction in a freshmen level Design Thinking course. Background: According to literature, synchronous online learning has advantages such as interaction, a classroom environment, and better course quality whereas asynchronous online learning has advantages such as self-controlled and self-directed learning. The disadvantages of synchronous online learning include the learning process, technology issues, and distraction. Social isolation, lack of interaction, and technology issue are a few disadvantages related to asynchronous online learning. Problem Being Addressed: There is a limited literature base investigating different modes of online instruction in a Design Thinking course. Through this paper, we are trying to understand and share the effectiveness of synchronous and asynchronous modes of instruction in an online Flipped Design Thinking Course. The results of the paper could also help in this time of pandemic by shedding light on the more effective way to teach highly active group-based classrooms for better student learning, social presence, and learner satisfaction. Method/Assessment: An end of semester survey was monitored in Spring 2020 to understand student experiences in synchronous and asynchronous Design Thinking course sections. The survey was sent to 720 students enrolled in the course in Spring 2020 and 324 students responded to the survey. Learning was measured using the survey instrument developed by Walker (2003) and the social presence and learner satisfaction was measured by the survey modified by Richardson and Swan (2003). Likert scale was used to measure survey responses. Anticipated Results: Data would be analyzed and the paper would be completed by draft paper submission. As the course under study is a flipped and active course with a significant component of group work, the anticipated results after analysis could be that one mode of instruction has higher student learning, social presence, and learner satisfaction compared to the other.
Gao, Zhikai, Lynch, Collin, and Heckman, Sarah. Too long to wait and not much to do: Modeling student behaviors while waiting for help in online office hours.. Retrieved from https://par.nsf.gov/biblio/10392589. Proceedings of the 7th Educational Data Mining in Computer Science Education (CSEDM) Workshop .
Gao, Zhikai, Lynch, Collin, & Heckman, Sarah. Too long to wait and not much to do: Modeling student behaviors while waiting for help in online office hours.. Proceedings of the 7th Educational Data Mining in Computer Science Education (CSEDM) Workshop, (). Retrieved from https://par.nsf.gov/biblio/10392589.
Gao, Zhikai, Lynch, Collin, and Heckman, Sarah.
"Too long to wait and not much to do: Modeling student behaviors while waiting for help in online office hours.". Proceedings of the 7th Educational Data Mining in Computer Science Education (CSEDM) Workshop (). Country unknown/Code not available. https://par.nsf.gov/biblio/10392589.
@article{osti_10392589,
place = {Country unknown/Code not available},
title = {Too long to wait and not much to do: Modeling student behaviors while waiting for help in online office hours.},
url = {https://par.nsf.gov/biblio/10392589},
abstractNote = {Promptly addressing students’ help requests on their programming assignments has become more and more challenging in computer science education. Since the pandemic, most instructors use online office hours to answer questions. Prior studies have shown increased student participation with online office hours. This popularity has led to significantly longer wait times in the office hours queue, and various strategies for selecting the next student to help may impact wait time. For example, prioritizing students who have not been seen on the day of the deadline will extend the wait time for students who are frequently rejoining the queue. To better understand this problem, we explored students’ behavior when they are waiting in the queue. We investigate the amount of time students are willing to wait in the queue by modeling the distribution of cancellation time. We find that after waiting for 49 minutes, most students will cancel their help request. Then, we looked at students’ coding actions during the waiting period and found that only 21% of students have commits while waiting. Surprisingly, students who waited for hours did not commit their work for automated feedback. Our findings suggest that time in the queue should be considered in addition to other factors like last interaction when selecting the next student to help during office hours to minimize canceled interactions.},
journal = {Proceedings of the 7th Educational Data Mining in Computer Science Education (CSEDM) Workshop},
author = {Gao, Zhikai and Lynch, Collin and Heckman, Sarah},
editor = {Akram, Bita and Shi, Yang and Brusilovsky, Peter and I-han Hsiao, Sharon and Leinonen, Juho}
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
