An official website of the United States government
A model that maps the requisite skills, or knowledge components, to the contents of an online course is necessary to implement many adaptive learning technologies. However, developing a skill model and tagging courseware contents with individual skills can be expensive and error prone. We propose a technology to automatically identify latent skills from instructional text on existing online courseware called Smart (Skill Model mining with Automated detection of Resemblance among Texts). Smart is capable of mining, labeling, and mapping skills without using an existing skill model or student learning (aka response) data. The goal of our proposed approach is to mine latent skills from assessment items included in existing courseware, provide discovered skills with human-friendly labels, and map didactic paragraph texts with skills. This way, mapping between assessment items and paragraph texts is formed. In doing so, automated skill models produced by Smart will reduce the workload of courseware developers while enabling adaptive online content at the launch of the course. In our evaluation study, we applied Smart to two existing authentic online courses. We then compared machine-generated skill models and human-crafted skill models in terms of the accuracy of predicting students’ learning. We also evaluated the similarity between machine-generated and human-crafted skill models. The results show that student models based on Smart-generated skill models were equally predictive of students’ learning as those based on human-crafted skill models— as validated on two OLI (Open Learning Initiative) courses. Also, Smart can generate skill models that are highly similar to human-crafted models as evidenced by the normalized mutual information (NMI) values.
more »
« less
Latent Skill Mining and Labeling from Courseware Content
A model that maps the requisite skills, or knowledge components, to the contents of an online course is necessary to implement many adaptive learning technologies. However, developing a skill model and tagging courseware contents with individual skills can be expensive and error prone. We propose a technology to automatically identify latent skills from instructional text on existing online courseware called Smart (Skill Model mining with Automated detection of Resemblance among Texts). Smart is capable of mining, labeling, and mapping skills without using an existing skill model or student learning (aka response) data. The goal of our proposed approach is to mine latent skills from assessment items included in existing courseware, provide discovered skills with human-friendly labels, and map didactic paragraph texts with skills. This way, mapping between assessment items and paragraph texts is formed. In doing so, automated skill models produced by Smart will reduce the workload of courseware developers while enabling adaptive online content at the launch of the course. In our evaluation study, we applied Smart to two existing authentic online courses. We then compared machine-generated skill models and human-crafted skill models in terms of the accuracy of predicting students’ learning. We also evaluated the similarity between machine-generated and human-crafted skill models. The results show that student models based on Smart-generated skill models were equally predictive of students’ learning as those based on human-crafted skill models— as validated on two OLI (Open Learning Initiative) courses. Also, Smart can generate skill models that are highly similar to human-crafted models as evidenced by the normalized mutual information (NMI) values.
more »
« less
- Award ID(s):
- 2016929
- PAR ID:
- 10558591
- Publisher / Repository:
- Zenodo
- Date Published:
- Journal Name:
- Journal of educational data mining
- ISSN:
- 2157-2100
- Subject(s) / Keyword(s):
- skill model discovery learning engineering massive open online course text mining natural language processing
- Format(s):
- Medium: X
- Right(s):
- Creative Commons Attribution 4.0 International; Open Access
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Physics instructors and education researchers use research-based assessments (RBAs) to evaluate students' preparation for physics courses. This preparation can cover a wide range of constructs including mathematics and physics content. Using separate mathematics and physics RBAs consumes course time. We are developing a new RBA for introductory mechanics as an online test using both computerized adaptive testing and cognitive diagnostic models. This design allows the adaptive RBA to assess mathematics and physics content knowledge within a single assessment. In this article, we used an evidence-centered design framework to inform the extent to which our models of skills students develop in physics courses fit the data from three mathematics RBAs. Our dataset came from the LASSO platform and includes 3,491 responses from the Calculus Concept Assessment, Calculus Concept Inventory, and Pre-calculus Concept Assessment. Our model included five skills: apply vectors, conceptual relationships, algebra, visualizations, and calculus. The "deterministic inputs, noisy 'and' gate'' (DINA) analyses demonstrated a good fit for the five skills. The classification accuracies for the skills were satisfactory. Including items from the three mathematics RBAs in the item bank for the adaptive RBA will provide a flexible assessment of these skills across mathematics and physics content areas that can adapt to instructors' needs.more » « less
-
The Academic Vigilance Environment (AVE) presented is a combination of two innovative tools. AchieveUp's micro-credentialing system identifies and showcase students' skills, while KnowGap's provides personalized learning content that fills knowledge gaps. To meet the growing demand for micro-credentials, AchieveUp integrates this capability into established courses using online quizzes to evaluate skills from a predefined test bank. By leveraging responses from digitized quiz-based assessments, we have developed a synergistic approach with online assessment and remediation protocols. Our Python-based toolkit enables undergraduate tutors to identify and address knowledge gaps among at-risk learners in higher-education courses. Through digitized assessments, personalized tutoring, and automated skill analysis scripts integrated into Canvas LMS, students receive skill-specific badges that provide incremental motivation and enhance their self-efficacy. In a required electrical and computer engineering course here at UCF, the implemented software allowed for the distribution of 17 unique digital badges suitable for LinkedIn posting, benefiting both students and employers by verifying skills, while also providing instructors with insights to improve course instruction.more » « less
-
Modeling student knowledge is important for assessment design, adaptive testing, curriculum design, and pedagogical intervention. The assessment design community has primarily focused on continuous latent-skill models with strong conditional independence assumptions among knowledge items, while the prerequisite discovery community has developed many models that aim to exploit the interdependence of discrete knowledge items. This paper attempts to bridge the gap by asking, "When does modeling assessment item interdependence improve predictive accuracy?" A novel adaptive testing evaluation framework is introduced that is amenable to techniques from both communities, and an efficient algorithm, Directed Item-Dependence And Confidence Thresholds (DIDACT), is introduced and compared with an Item-Response-Theory based model on several real and synthetic datasets. Experiments suggest that assessments with closely related questions benefit significantly from modeling item interdependence.more » « less
-
This paper presents an innovative courseware project based on the Adaptive Distributed Learning (ADL) Initiative’s Total Learning Architecture (TLA [1]), which encompasses a technical framework for education and training based on a data strategy built around open standards to support interoperability across diverse organizations and products ([2]). This framework includes definitions of a set of policies, specifications, and standards that enable a future learning ecosystem to facilitate lifelong learning principles promoting personalized and flexible learning environments that include both formal and informal activities [3]. In Fall 2023, a TLA- inspired course framework was implemented in a data visualization course for senior undergraduates and graduate students, using Moodle and a Learning Record Store (LRS) to track over 200,000 learning records. This system allowed instructors to visually monitor online learning activities for the whole class as well as selected individual learners. As future work, the course will expand to 10 STEM courses across 11 universities in the next three years as part of an existing NSF commitment.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.5281/zenodo.7086211
