An official website of the United States government
Students achieve functional knowledge retention through active, spaced repetition of concepts through homework, quizzes, and lectures. True knowledge retention is best achieved through proper comprehension of the concept. In the engineering curriculum, courses are sequenced into prerequisite chains of three to five courses per subfield –- a design aimed at developing and reinforcing core concepts over time. Knowledge retention of these prerequisite concepts is important for the next course. In this project, concept review quizzes were used to identify the gaps and deficiencies in students' prerequisite knowledge and measure improvement after a concept review intervention. Two quizzes (pre-intervention and post-intervention) drew inspiration from the standard concept inventories for fundamental concepts and include concepts such as Free Body Diagrams, Contact and Reaction Forces, Equilibrium Equations, and Calculation of the Moment. Concept inventories are typically multiple-choice, in this evaluation the concept questions were open-ended. A clear rubric was created to identify the missing prerequisite concepts in the students' knowledge. These quizzes were deployed in Mechanics of Materials, a second-level course in the engineering mechanics curriculum (the second in a sequence of four courses: Statics, Mechanics of Materials, Mechanical Design, and Kinematic Design). The pre-quiz was administered (unannounced) at the beginning of the class. The class then actively participated in a 30-minute concept review. A different post-quiz was administered in the same class period after the review. Quizzes were graded with a rubric to measure the effect of the concept review intervention on the students’ knowledge demonstration and calculations. The study evaluated four major concepts: free body diagrams, boundary reaction forces (fixed, pin, and contact), equilibrium, and moment calculation. Students showed improvements of up to 39\% in the case of drawing a free body diagram with fixed boundary condition, but continued to struggle with free body diagram involving contact forces. This study was performed at a large public institution in a class size of 240 students. A total of 224 students consented to the use of their data for this study (and attended class on the day of the intervention). The pre-quiz is used to determine the gaps (or deficiencies) in conceptual understanding among students. The post-quiz measures the response to the review and is used to determine which concept deficiencies were significantly improved by the review, and which concept deficiencies were not significantly improved by the concept review. This study presents a concept quiz and associated rubric for measuring student improvement resulting from an in-class intervention (concept review). It quantifies a significant improvement in the students’ retrieval of their prerequisite knowledge after a concept review session. This approach, therefore, has utility for improving knowledge retention in programs with a similar, sequenced course design.
more »
« less
Predicting Prerequisite Relations for Unseen Concepts
Concept prerequisite learning (CPL) plays a key role in developing technologies that assist people to learn a new complex topic or concept. Previous work commonly assumes that all concepts are given at training time and solely focuses on predicting the unseen prerequisite relationships between them. However, many real-world scenarios deal with concepts that are left undiscovered at training time, which is relatively unexplored. This paper studies this problem and proposes a novel alternating knowledge distillation approach to take advantage of both content- and graph-based models for this task. Extensive experiments on three public benchmarks demonstrate up to 10% improvements in terms of F1 score.
more »
« less
- Award ID(s):
- 2106282
- PAR ID:
- 10434651
- Date Published:
- Journal Name:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)
- Page Range / eLocation ID:
- 8542–8548
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Humans have the remarkable ability to recognize and acquire novel visual concepts in a zero-shot manner. Given a high-level, symbolic description of a novel concept in terms of previously learned visual concepts and their relations, humans can recognize novel concepts without seeing any examples. Moreover, they can acquire new concepts by parsing and communicating symbolic structures using learned visual concepts and relations. Endowing these capabilities in machines is pivotal in improving their generalization capability at inference time. We introduced Zero-shot Concept Recognition and Acquisition (ZeroC), a neuro-symbolic architecture that can recognize and acquire novel concepts in a zero-shot way. ZeroC represents concepts as graphs of constituent concept models (as nodes) and their relations (as edges). To allow inference time composition, we employed energy-based models (EBMs) to model concepts and relations. We designed ZeroC architecture so that it allows a one-to-one mapping between a symbolic graph structure of a concept and its corresponding EBM, which for the first time, allows acquiring new concepts, communicating its graph structure, and applying it to classification and detection tasks (even across domains) at inference time. We introduced algorithms for learning and inference with ZeroC. We evaluated ZeroC on a challenging grid-world dataset which is designed to probe zero-shot concept recognition and acquisition, and demonstrated its capability.more » « less
-
Concept bottleneck models (CBM) aim to improve model interpretability by predicting human level “concepts” in a bottleneck within a deep learning model architecture. However, how the predicted concepts are used in predicting the target still either remains black-box or is simplified to maintain interpretability at the cost of prediction performance. We propose to use Fast Interpretable Greedy Sum- Trees (FIGS) to obtain Binary Distillation (BD). This new method, called FIGSBD, distills a binary-augmented concept-to-target portion of the CBM into an interpretable tree-based model, while maintaining the competitive prediction performance of the CBM teacher. FIGS-BD can be used in downstream tasks to explain and decompose CBM predictions into interpretable binary-concept-interaction attributions and guide adaptive test-time intervention. Across 4 datasets, we demonstrate that our adaptive test-time intervention identifies key concepts that significantly improve performance for realistic human-in-the-loop settings that only allow for limited concept interventions. All code is made available on Github (https://github.com/mattyshen/adaptiveTTI).more » « less
-
A growing body of research indicates spatial visualization skills are important to success in many STEM disciplines, including several engineering majors that rely on a foundation in engineering mechanics. Many fundamental mechanics concepts such as free-body diagrams, moments, and vectors are inherently spatial in that application of the concept and related analytical techniques requires visualization and sketching. Visualization may also be important to mechanics learners’ ability to understand and employ common mechanics representations and conventions in communication and problem solving, a skill known as representational competence. In this paper, we present early research on how spatial abilities might factor in to students’ conceptual understanding of vectors and associated representational competence. We administered the Mental Cutting Test (MCT), a common assessment of spatial abilities, in the first and last week of the term. We also administered the Test of Representational Competence with Vectors (TRCV), a targeted assessment of vector concepts and representations, in week one and at mid-term. The vector post-test came after coverage of moments and cross products. We collected this assessment data in statics courses across multiple terms at three different colleges. To understand how spatial skills relate to the development of representational competence, we use a multiple regression model to predict TRCV scores using the pre-class MCT scores as well as other measures of student preparation in the form of grades in prerequisite math and physics coursework. We then extend the analysis to consider both MCT and TRCV scores as predictors for student performance on the Concept Assessment Test in Statics. We find that spatial abilities are a factor in students’ development of representational competence with vectors. We also find that representational competence with vectors likely mediates the importance of spatial abilities to student success in developing broader conceptual understanding in statics. We conclude by discussing implications for mechanics instruction.more » « less
-
null (Ed.)A growing body of research indicates spatial visualization skills are important to success in many STEM disciplines, including several engineering majors that rely on a foundation in engineering mechanics. Many fundamental mechanics concepts such as free-body diagrams, moments, and vectors are inherently spatial in that application of the concept and related analytical techniques requires visualization and sketching. Visualization may also be important to mechanics learners’ ability to understand and employ common mechanics representations and conventions in communication and problem solving, a skill known as representational competence. In this paper, we present early research on how spatial abilities might factor in to students’ conceptual understanding of vectors and associated representational competence. We administered the Mental Cutting Test (MCT), a common assessment of spatial abilities, in the first and last week of the term. We also administered the Test of Representational Competence with Vectors (TRCV), a targeted assessment of vector concepts and representations, in week one and at mid-term. The vector post-test came after coverage of moments and cross products. We collected this assessment data in statics courses across multiple terms at three different colleges. To understand how spatial skills relate to the development of representational competence, we use a multiple regression model to predict TRCV scores using the pre-class MCT scores as well as other measures of student preparation in the form of grades in prerequisite math and physics coursework. We then extend the analysis to consider both MCT and TRCV scores as predictors for student performance on the Concept Assessment Test in Statics. We find that spatial abilities are a factor in students’ development of representational competence with vectors. We also find that representational competence with vectors likely mediates the importance of spatial abilities to student success in developing broader conceptual understanding in statics. We conclude by discussing implications for mechanics instruction.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
