skip to main content
Explore Scholarly Publications and Datasets in the NSF-PAR

Title: Can textbook annotations serve as an early predictor of student learning?
When engaging with a textbook, students are inclined to highlight key content. Although students believe that highlighting and subsequent review of the highlights will further their educational goals, the psychological literature provides no evidence of benefits. Nonetheless, a student’s choice of text for highlighting may serve as a window into their mental state—their level of comprehension, grasp of the key ideas, reading goals, etc. We explore this hypothesis via an experiment in which 198 participants read sections from a college-level biology text, briefly reviewed the text, and then took a quiz on the material. During initial reading, participants were able to highlight words, phrases, and sentences, and these highlights were displayed along with the complete text during the subsequent review. Consistent with past research, the amount of highlighted material is unrelated to quiz performance. However, our main goal is to examine highlighting as a data source for inferring student understanding. We explored multiple representations of the highlighting patterns and tested Bayesian linear regression and neural network models, but we found little or no relationship between a student’s highlights and quiz performance. Our long-term goal is to design digital textbooks that serve not only as conduits of information into the mind of the reader, but also allow us to draw inferences about the reader at a point where interventions may increase the effectiveness of the material.
Authors:
; ; ; ;
Award ID(s):
1631428
Publication Date:
NSF-PAR ID:
10074162
Journal Name:
Proceedings of the 11th International Conference on Educational Data Mining
Volume:
11
Page Range or eLocation-ID:
431-437
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ( , Proceedings of the Third International Workshop on Intelligent Textbooks (iTextbooks))
    Sosnovsky, S. ; Brusilovsky, P ; Baraniuk, R. G. ; Lan, A. S. (Ed.)
    As students read textbooks, they often highlight the material they deem to be most important. We analyze students’ highlights to predict their subsequent performance on quiz questions. Past research in this area has encoded highlights in terms of where the highlights appear in the stream of text—a positional representation. In this work, we construct a semantic representation based on a state-of-the-art deep-learning sentence embedding technique (SBERT) that captures the content-based similarity between quiz questions and highlighted (as well as non-highlighted) sentences in the text. We construct regression models that include latent variables for student skill level and question difficulty andmore »augment the models with highlighting features. We find that highlighting features reliably boost model performance. We conduct experiments that validate models on held-out questions, students, and student-questions and find strong generalization for the latter two but not for held-out questions. Surprisingly, highlighting features improve models for questions at all levels of the Bloom taxonomy, from straightforward recall questions to inferential synthesis/evaluation/creation questions.« less
  2. ; ; ; ( , Proceedings of the Third Workshop on Intelligent Textbooks (iTextbooks) at the 22nd International Conference on Artificial Intelligence in Education (AIED 2021))
    As students read textbooks, they often highlight the material they deem to be most important. We analyze students’ highlights to predict their subsequent performance on quiz questions. Past research in this area has encoded highlights in terms of where the highlights appear in the stream of text—a positional representation. In this work, we construct a semantic representation based on a state-of-the-art deep-learning sentence embedding technique (SBERT) that captures the content-based similarity between quiz questions and highlighted (as well as non-highlighted) sentences in the text. We construct regression models that include latent variables for student skill level and question difficulty andmore »augment the models with highlighting features. We find that highlighting features reliably boost model performance. We conduct experiments that validate models on held-out questions, students, and student-questions and find strong generalization for the latter two but not for held-out questions. Surprisingly, highlighting features improve models for questions at all levels of the Bloom taxonomy, from straightforward recall questions to inferential synthesis/evaluation/creation questions.« less
  3. ; ; ; ; ; ( , Intelligent Textbooks 2020)
    We investigate whether student comprehension and knowledge retention can be predicted from textbook annotations, specifically the material that students choose to highlight. Using a digital open-access textbook platform, Openstax, students enrolled in Biology, Physics, and Sociology courses read sections of their introductory text as part of required coursework, optionally highlighted the text to flag key material, and then took brief quizzes as the end of each section. We find that when students choose to highlight, the specific pattern of highlights can explain about 13% of the variance in observed quiz scores. We explore many different representations of the pattern ofmore »highlights and discover that a low-dimensional logistic principal component based vector is most effective as input to a ridge regression model. Considering the many sources of uncontrolled variability affecting student performance, we are encouraged by the strong signal that highlights provide as to a student’s knowledge state.« less
  4. ( , 2021 ASEE Virtual Annual Conference Content Access)
    Bridge courses are often created to provide participants with remediation instruction on discipline-specific content knowledge, like chemistry and mathematics, before enrollment in regular (semester-long) courses. The bridge courses are then designed to impact student’s academic success in the short-term. Also, as a consequence of the bridge course experience, it is often expected that students’ dropout rates on those regular courses will decrease. However, the bridge courses are often short (ten or fewer days) and packed with content, thus creating challenges for helping students sustain their learning gains over time. With the support of the NSF funded (DUE - Division Ofmore »Undergraduate Education) STEM Center at Sam Houston State University, we are designing a course for entering chemistry students that consists of a one-week pre-semester intensive bridge component, which then flows into a one-month co-curricular support component at the beginning of the semester. The primary goals of the bridge component of the course are to strengthen student academic preparedness, calibrated-self-efficacy, and to foster networking leading to a strong learning community. The goal of the co-curricular extension is to help students sustain and build upon the learning gains of the initial bridge component. We plan to extend the co-curricular portion of the course in future years. A key measure of success will be improved participant course grades in the introductory chemistry courses for majors. Our design process has been centered on weekly meetings that alternate between literature review and course design. The design process was initiated with backward design principles and continues with ongoing revision. The goals, design strategy, and design process of this new course will be presented along with the achieved student outcomes during the implementation of the past summer 2020.« less
  5. ( , 2021 ASEE Virtual Annual Conference Content Access)
    Bridge courses are often created to provide participants with remediation instruction on discipline-specific content knowledge, like chemistry and mathematics, before enrollment in regular (semester-long) courses. The bridge courses are then designed to impact student’s academic success in the short-term. Also, as a consequence of the bridge course experience, it is often expected that students’ dropout rates on those regular courses will decrease. However, the bridge courses are often short (ten or fewer days) and packed with content, thus creating challenges for helping students sustain their learning gains over time. With the support of the NSF funded (DUE - Division Ofmore »Undergraduate Education) STEM Center at Sam Houston State University, we are designing a course for entering chemistry students that consists of a one-week pre-semester intensive bridge component, which then flows into a one-month co-curricular support component at the beginning of the semester. The primary goals of the bridge component of the course are to strengthen student academic preparedness, calibrated-self-efficacy, and to foster networking leading to a strong learning community. The goal of the co-curricular extension is to help students sustain and build upon the learning gains of the initial bridge component. We plan to extend the co-curricular portion of the course in future years. A key measure of success will be improved participant course grades in the introductory chemistry courses for majors. Our design process has been centered on weekly meetings that alternate between literature review and course design. The design process was initiated with backward design principles and continues with ongoing revision. The goals, design strategy, and design process of this new course will be presented along with the achieved student outcomes during the implementation of the past summer 2020.« less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email