Title: Can textbook annotations serve as an early predictor of student learning?
When engaging with a textbook, students are inclined to
highlight key content. Although students believe that highlighting
and subsequent review of the highlights will further
their educational goals, the psychological literature provides
no evidence of benefits. Nonetheless, a student’s choice of
text for highlighting may serve as a window into their mental
state—their level of comprehension, grasp of the key
ideas, reading goals, etc. We explore this hypothesis via an
experiment in which 198 participants read sections from a
college-level biology text, briefly reviewed the text, and then
took a quiz on the material. During initial reading, participants
were able to highlight words, phrases, and sentences,
and these highlights were displayed along with the complete
text during the subsequent review. Consistent with past research,
the amount of highlighted material is unrelated to
quiz performance. However, our main goal is to examine
highlighting as a data source for inferring student understanding.
We explored multiple representations of the highlighting
patterns and tested Bayesian linear regression and
neural network models, but we found little or no relationship
between a student’s highlights and quiz performance. Our
long-term goal is to design digital textbooks that serve not
only as conduits of information into the mind of the reader,
but also allow us to draw inferences about the reader at a
point where interventions may increase the effectiveness of
the material. more »« less
Kim, David Y.J.; Scott, Tyler R.; Basu Mallick, Debshila; Mozer, Michael C.(
, Proceedings of the Third Workshop on Intelligent Textbooks (iTextbooks) at the 22nd International Conference on Artificial Intelligence in Education (AIED 2021))
null
(Ed.)
As students read textbooks, they often highlight the material they deem to be most important. We analyze students’ highlights to predict their subsequent performance on quiz questions. Past research in this area has encoded highlights in terms of where the highlights appear in the stream of text—a positional representation. In this work, we construct a semantic representation based on a state-of-the-art deep-learning sentence embedding technique (SBERT) that captures the content-based similarity between quiz questions and highlighted (as well as non-highlighted) sentences in the text. We construct regression models that include latent variables for student skill level and question difficulty and augment the models with highlighting features. We find that highlighting features reliably boost model performance. We conduct experiments that validate models on held-out questions, students, and student-questions and find strong generalization for the latter two but not for held-out
questions. Surprisingly, highlighting features improve models for questions at all levels of the Bloom taxonomy, from straightforward recall questions to inferential synthesis/evaluation/creation questions.
Kim, D. Y.; Scott, T. R.; Mallick, D.; Mozer, M. C.(
, Proceedings of the Third International Workshop on Intelligent Textbooks (iTextbooks))
Sosnovsky, S.; Brusilovsky, P; Baraniuk, R. G.; Lan, A. S.
(Ed.)
As students read textbooks, they often highlight the material they deem to be most important. We analyze students’ highlights to predict their subsequent performance on quiz questions. Past research in this area has encoded highlights in terms of where the highlights appear in the stream of text—a positional representation. In this work, we construct a semantic representation based on a state-of-the-art deep-learning sentence embedding technique (SBERT) that captures the content-based similarity between quiz questions and highlighted (as well as non-highlighted) sentences in the text. We construct regression models that include latent variables for student skill level and question difficulty and augment the models with highlighting features. We find that highlighting features reliably boost model performance. We conduct experiments that validate models on held-out questions, students, and student-questions and find strong generalization for the latter two but not for held-out questions. Surprisingly, highlighting features improve models for questions at all levels of the Bloom taxonomy, from straightforward recall questions to inferential synthesis/evaluation/creation questions.
Kim, D. Y.; Winchell, A.; Water, A. E.; Grimaldi, P. J.; Baraniuk, R.; Mozer, M. C.(
, Intelligent Textbooks 2020)
We investigate whether student comprehension and knowledge retention can be predicted from textbook annotations, specifically
the material that students choose to highlight. Using a digital open-access
textbook platform, Openstax, students enrolled in Biology, Physics, and
Sociology courses read sections of their introductory text as part of required coursework, optionally highlighted the text to flag key material,
and then took brief quizzes as the end of each section. We find that
when students choose to highlight, the specific pattern of highlights can
explain about 13% of the variance in observed quiz scores. We explore
many different representations of the pattern of highlights and discover
that a low-dimensional logistic principal component based vector is most
effective as input to a ridge regression model. Considering the many
sources of uncontrolled variability affecting student performance, we are
encouraged by the strong signal that highlights provide as to a student’s
knowledge state.
This research evaluates the impact of switching college engineering courses from in-person instruction to emergency remote learning among engineering students at a university in the Midwest. The study aimed to answer the question: What were the concerns and perceived challenges students faced when traditional in-person engineering courses suddenly transitioned to remote learning? The goal of this study is to uncover the challenges students were facing in engineering online courses and to understand students’ concerns. Our findings can help improve teaching instruction to provide students with previously unavailable educational assistance for online engineering courses.
We collected online survey responses during weeks 8 and 9 of the academic semester, shortly after the COVID-19 shutdown and emergency transition to remote learning in Spring 2020. The survey included two open-ended questions which inquired about students’ feedback about moving the class online, and one two-item scale which assessed students’ confidence in online engineering learning. Data analysis for the open-ended questions was guided by the theoretical framework - Social Cognitive Career Theory [1] that explores how context, person factors and social cognitions contribute to career goals, interests and actions. A phenomenological approach [2] was conducted to understand the experience of these students. Open coding and axial coding [2] methods were used to create initial categories then themes related to students' concerns and challenges. Data from the two-item scale was evaluated using descriptive statistics: means, standard deviations, and ranges.
Four main themes with separate sub-categories emerged from the student responses: 1) Instructor’s ability to teach course online (Instructional limitations, Seeking help, Increased Workload), 2) Student’s ability to learn online (Time Management, Lower engagement and motivation, Harder to absorb material, Hard to focus, Worry about performance), 3) Difficulties outside of class (Technology issues), and 4) No concerns. Students seemed more concerned about their ability to learn the material (48% of responses) than the instructor’s ability to teach the material (36% of responses). The instructional limitations or lack of instructional support (22% of responses) and time management (12% of responses) were among the major concerns in the sub-categories.
The results from two-item scale indicated participants' s confidence in their ability to master their classroom knowledge was at an intermediate level via online instruction (6/10), and participants' confidence in the instructor's ability to teach knowledge in online classes is moderate to high (7/10). The results align with the open-ended question response in which students were somewhat more concerned about their ability to learn than the instructor’s ability to teach. The themes and analysis will be a valuable tool to help institutions and instructors improve student learning experiences.
Bridge courses are often created to provide participants with remediation instruction on discipline-specific content knowledge, like chemistry and mathematics, before enrollment in regular (semester-long) courses. The bridge courses are then designed to impact student’s academic success in the short-term. Also, as a consequence of the bridge course experience, it is often expected that students’ dropout rates on those regular courses will decrease. However, the bridge courses are often short (ten or fewer days) and packed with content, thus creating challenges for helping students sustain their learning gains over time. With the support of the NSF funded (DUE - Division Of Undergraduate Education) STEM Center at Sam Houston State University, we are designing a course for entering chemistry students that consists of a one-week pre-semester intensive bridge component, which then flows into a one-month co-curricular support component at the beginning of the semester. The primary goals of the bridge component of the course are to strengthen student academic preparedness, calibrated-self-efficacy, and to foster networking leading to a strong learning community. The goal of the co-curricular extension is to help students sustain and build upon the learning gains of the initial bridge component. We plan to extend the co-curricular portion of the course in future years. A key measure of success will be improved participant course grades in the introductory chemistry courses for majors. Our design process has been centered on weekly meetings that alternate between literature review and course design. The design process was initiated with backward design principles and continues with ongoing revision. The goals, design strategy, and design process of this new course will be presented along with the achieved student outcomes during the implementation of the past summer 2020.
Winchell, A., Mozer, M. C., Lan, A., Grimaldi, P., and Pashler, H. Can textbook annotations serve as an early predictor of student learning?. Retrieved from https://par.nsf.gov/biblio/10074162. Proceedings of the 11th International Conference on Educational Data Mining 11.
Winchell, A., Mozer, M. C., Lan, A., Grimaldi, P., & Pashler, H. Can textbook annotations serve as an early predictor of student learning?. Proceedings of the 11th International Conference on Educational Data Mining, 11 (). Retrieved from https://par.nsf.gov/biblio/10074162.
Winchell, A., Mozer, M. C., Lan, A., Grimaldi, P., and Pashler, H.
"Can textbook annotations serve as an early predictor of student learning?". Proceedings of the 11th International Conference on Educational Data Mining 11 (). Country unknown/Code not available. https://par.nsf.gov/biblio/10074162.
@article{osti_10074162,
place = {Country unknown/Code not available},
title = {Can textbook annotations serve as an early predictor of student learning?},
url = {https://par.nsf.gov/biblio/10074162},
abstractNote = {When engaging with a textbook, students are inclined to highlight key content. Although students believe that highlighting and subsequent review of the highlights will further their educational goals, the psychological literature provides no evidence of benefits. Nonetheless, a student’s choice of text for highlighting may serve as a window into their mental state—their level of comprehension, grasp of the key ideas, reading goals, etc. We explore this hypothesis via an experiment in which 198 participants read sections from a college-level biology text, briefly reviewed the text, and then took a quiz on the material. During initial reading, participants were able to highlight words, phrases, and sentences, and these highlights were displayed along with the complete text during the subsequent review. Consistent with past research, the amount of highlighted material is unrelated to quiz performance. However, our main goal is to examine highlighting as a data source for inferring student understanding. We explored multiple representations of the highlighting patterns and tested Bayesian linear regression and neural network models, but we found little or no relationship between a student’s highlights and quiz performance. Our long-term goal is to design digital textbooks that serve not only as conduits of information into the mind of the reader, but also allow us to draw inferences about the reader at a point where interventions may increase the effectiveness of the material.},
journal = {Proceedings of the 11th International Conference on Educational Data Mining},
volume = {11},
author = {Winchell, A. and Mozer, M. C. and Lan, A. and Grimaldi, P. and Pashler, H.},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
