An official website of the United States government
Standard learning assessments like multiple-choice questions measure what students know but not how their knowledge is organized. Recent advances in cognitive network science provide quantitative tools for modeling the structure of semantic memory, revealing key learning mechanisms. In two studies, we examined the semantic memory networks of undergraduate students enrolled in an introductory psychology course. In Study 1, we administered a cumulative multiple-choice test of psychology knowledge, the Intro Psych Test, at the end of the course. To estimate semantic memory networks, we administered two verbal fluency tasks: domain-specific fluency (naming psychology concepts) and domain-general fluency (naming animals). Based on their performance on the Intro Psych Test, we categorized students into a high-knowledge or low-knowledge group, and compared their semantic memory networks. Study 1 (N = 213) found that the high-knowledge group had semantic memory networks that were more clustered, with shorter distances between concepts—across both the domain-specific (psychology) and domain-general (animal) categories—compared to the low-knowledge group. In Study 2 (N = 145), we replicated and extended these findings in a longitudinal study, collecting data near the start and end of the semester. In addition to replicating Study 1, we found the semantic memory networks of high-knowledge students became more interconnected over time, across both domain-general and domain-specific categories. These findings suggest that successful learners show a distinct semantic memory organization—characterized by high connectivity and short path distances between concepts—highlighting the utility of cognitive network science for studying variation in student learning.
more »
« less
Semantic Networks Extracted from Students’ Think-Aloud Data are Correlated with Students’ Learning Performance
When students reflect on their learning from a textbook via think aloud, network representations can be used to capture their concepts and relations. What can we learn from these network representations about students’ learning processes, knowledge acquisition, and learning outcomes? This study brings methods from entity and relation extraction using classic and LLM-based methods to the application domain of educational psychology. We built a ground-truth baseline of relational data that represent relevant (to educational science), textbook-based information as a semantic network. We identified SPN4RE and LUKE as the most accurate method to extracting semantic networks capturing the same types of information from transcriptions of verbal student data. Correlating the students’ semantic networks with learning outcomes showed that students’ verbalizations varied in structure, reflecting different learning processes. Denser and more interconnected semantic networks indicated more elaborated knowledge acquisition. Structural features such as the number of edges and surface overlap with textbook networks significantly correlated with students’ posttest performance.
more »
« less
- Award ID(s):
- 2225298
- PAR ID:
- 10656601
- Publisher / Repository:
- Association for Computational Linguistics
- Date Published:
- Page Range / eLocation ID:
- 25802 to 25815
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This investigation is the first of four investigations funded by the NSF (DUE award 2215807) to develop and then field test on open educational browser-based writing-to-learn tool called GIKS. The underlying theory is that writing-to-learn with immediate formative feedback presented as concept networks is engaging and effective for learning concepts covered in lectures. This work was studied in a second year architectural engineering course focusing on building materials, processes and modeling. Participants (n=84) completed a lesson (readings, lecture, and labs) then followed by writing prompts centered on the following topics: Building with Concrete and Wood Construction (3 weeks later). Participants were assigned to one of two counterbalanced groups, group A used GIKS software to write a 300-word summary of the first lesson but did not write in the second lesson, while group B did not write in the first lesson but used GIKS in the second lesson, so that each group served as a control treatment for the other group. All students completed a concept structure survey at the end of each lesson that contained 20 key concepts from that lesson, the two concept structure surveys’ data were transformed into concept networks and then these networks were compared to an expert network benchmark referent, as well as to networks of the textbook chapter and the PowerPoint slides of the related lesson. Then a week after the second lesson students completed the standing end-of-module multiple-choice posttest that included items from these lesson as well as from other lessons in the module. Results to date highlight that for both lessons, the group using GIKS scored higher on the concept structure survey (more like the expert network) BUT lower on the multiple-choice test, the difference was significant for the Building with Concrete lesson (p < .05) but not for the Wood Construction lesson. This interaction has been reported before by Ntshalintshali & Clariana (2020), that improving conceptual knowledge sometimes decreases memory of lesson details. Descriptive analysis of the group-average networks derived from the concept structure surveys for Building with Concrete show that the group-averaged network of those using GIKS compared to the control was more like the expert network (54% vs. 36%), the network of the textbook Chapter (32% vs. 29%), the network of the PowerPoint (PP) (46% vs. 41%), and especially like peers in the other group (67%). For Wood Construction the difference between the groups was less, the group-averaged network of those using GIKS compared to the control was more like the expert (40% vs. 39%), like the light-framed construction PP (28% vs. 24%), and especially like peers in the other group (72%). These findings show that writing-to-learn with GIKS with immediate network feedback improves conceptual knowledge as expected but at the cost of details. Peers conceptual structure of the lesson materials were very similar (peer-peer mental model convergence) and were more like others than like the expert, or the book chapters, or the PowerPoint slides; in addition, the PowerPoint slides appear to influence conceptual structure more than the textbook chapters. Investigation 2 will consider writing-to-learn with or without immediate network feedback in order to isolate the effects of immediate network feedback.more » « less
-
The development and application of deep learning method- ologies has grown within educational contexts in recent years. Perhaps attributable, in part, to the large amount of data that is made avail- able through the adoption of computer-based learning systems in class- rooms and larger-scale MOOC platforms, many educational researchers are leveraging a wide range of emerging deep learning approaches to study learning and student behavior in various capacities. Variations of recurrent neural networks, for example, have been used to not only pre- dict learning outcomes but also to study sequential and temporal trends in student data; it is commonly believed that they are able to learn high- dimensional representations of learning and behavioral constructs over time, such as the evolution of a students’ knowledge state while working through assigned content. Recent works, however, have started to dis- pute this belief, instead finding that it may be the model’s complexity that leads to improved performance in many prediction tasks and that these methods may not inherently learn these temporal representations through model training. In this work, we explore these claims further in the context of detectors of student affect as well as expanding on exist- ing work that explored benchmarks in knowledge tracing. Specifically, we observe how well trained models perform compared to deep learning networks where training is applied only to the output layer. While the highest results of prior works utilizing trained recurrent models are found to be superior, the application of our untrained-versions perform compa- rably well, outperforming even previous non-deep learning approaches. Keywords: Deep Learning · LSTM · Echo State Network · Affect · Knowledge Tracing.more » « less
-
Akram, Bita; Shi, Yang; Brusilovsky, Peter; Price, Thomas; Koedinger, Ken; Carvalho, Paulo; Zhang, Shan; Lan, Andrew; Leinonen, Juho (Ed.)The “Doer Effect” is the empirical phenomenon observed as a stronger correlational relationship between students who complete more activities and their course learning outcomes compared to those who complete fewer activities or watch fewer videos. In this paper, we extended prior evidence of a “Doer Effect” to investigate how doing more can be related not only to better learning outcomes but also to motivational ones. Specifically, we investigated persistence as the student’s willingness to continue working on course activities. We used secondary analyses of data from MOOC that taught Advanced Placement (AP) Introductory Java Programming to high school students using the digital textbook platform RuneStone. Although we failed to identify a doer effect in learning outcomes, our analyses do suggest that completing more activities is related to longer persistence in the course than reading more pages or watching more videos. This effect does not appear to be limited to highly motivated students.more » « less
-
The goal of Project STEMulate, a National Science Foundation ITEST study (DRL 1657625), was to develop, implement, and evaluate a program that fosters success in STEM for underserved and underrepresented high school students. The project was implemented at three sites of the Department of Education Upward Bound Program in Hawaiˋi. Project STEMulate delivered teacher training on Problem-Based Learning curriculum to ensure students were motivated and empowered, and to support STEM- related postsecondary educational success of Hawaiian and Pacific Islander students. A critical design goal of the program was to introduce teaching and learning strategies and processes that were more relevant to underrepresented youth populations than those offered in typical high schools to provide opportunities and to increase participation in the STEM study and career trajectory, something all too often out of mind and scope of these students. This study reports on three years of mixed methods summer academy data on both student and teacher learning outcomes. Teacher dispositions, evidenced through data from interviews, observations, and multi-point surveys improved in a majority of the dimensions, including teaching inquiry-based approaches, integrating technology, and STEM career knowledge and awareness. Student motivation, Science self-efficacy, and STEM career interest, evidenced from similar data sources, increased as well. Finally, we discuss the larger implications of extending this work to impact similar populations elsewhere of isolated, under- resourced and under-exposed youth with these proven strategies.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.18653/v1/2025.emnlp-main.1309
