An official website of the United States government
We propose a concept map based approach to assessing freely generated student responses. The proposed approach is based on a novel automated tuple extraction system, DT-OpenIE, for automatically extracting concept maps from student responses. The DT-OpenIE system is significantly better, for assessment purposes, in terms of concept map quality than state-of-the-art open information extraction (IE) systems such as Ollie or Stanford as evidenced by our experimental results. The concept map based approach can not only generate a holistic score assessing the accuracy of a student response but also enable diagnostic feedback.
more »
« less
Towards Concept Map Based Free Student Answer Assessment
We propose a concept map based approach to assessing freely generated student responses. The proposed approach is based on a novel automated tuple extraction system, DT-OpenIE, for automatically extracting concept maps from student responses. The DT-OpenIE system is significantly better, for assessment purposes, in terms of concept map quality than state-of-the-art open information extraction (IE) systems such as Ollie or Stanford as evidenced by our experimental results. The concept map based approach can not only generate a holistic score assessing the accuracy of a student response but also enable diagnostic feedback.
more »
« less
- Award ID(s):
- 1822816
- PAR ID:
- 10107803
- Date Published:
- Journal Name:
- Proceedings of the 32nd International Florida Artificial Intelligence Research Society Conference (FLAIRS 2019)
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract BackgroundTeachers often rely on the use of open‐ended questions to assess students' conceptual understanding of assigned content. Particularly in the context of mathematics; teachers use these types of questions to gain insight into the processes and strategies adopted by students in solving mathematical problems beyond what is possible through more close‐ended problem types. While these types of problems are valuable to teachers, the variation in student responses to these questions makes it difficult, and time‐consuming, to evaluate and provide directed feedback. It is a well‐studied concept that feedback, both in terms of a numeric score but more importantly in the form of teacher‐authored comments, can help guide students as to how to improve, leading to increased learning. It is for this reason that teachers need better support not only for assessing students' work but also in providing meaningful and directed feedback to students. ObjectivesIn this paper, we seek to develop, evaluate, and examine machine learning models that support automated open response assessment and feedback. MethodsWe build upon the prior research in the automatic assessment of student responses to open‐ended problems and introduce a novel approach that leverages student log data combined with machine learning and natural language processing methods. Utilizing sentence‐level semantic representations of student responses to open‐ended questions, we propose a collaborative filtering‐based approach to both predict student scores as well as recommend appropriate feedback messages for teachers to send to their students. Results and ConclusionWe find that our method outperforms previously published benchmarks across three different metrics for the task of predicting student performance. Through an error analysis, we identify several areas where future works may be able to improve upon our approach.more » « less
-
Design projects are an important part of many first-year engineering programs. The desire to employ holistic assessment strategies to student work with open-ended and divergent responses has been widely noted in the literature. Holistic strategies can provide insight into the role of qualities (e.g., professional constructs) that are not typically conducive to standard assessment rubrics. Adaptive Comparative Judgement (ACJ) is an assessment approach that is used to assess design projects holistically. The assessment of projects using ACJ can be carried out by experts or students to scaffold their learning experience. This Work-in-Progress paper explores the use and benefits of ACJ for assessing design projects specifically focusing on first-year engineering students and educators. Further, conference attendees will be provided the opportunity throughout the conference to engage with the ACJ software to experience how this system can work in practice for assessing student design projects.more » « less
-
Abstract Background Diabetic retinopathy (DR) is a leading cause of blindness in American adults. If detected, DR can be treated to prevent further damage causing blindness. There is an increasing interest in developing artificial intelligence (AI) technologies to help detect DR using electronic health records. The lesion-related information documented in fundus image reports is a valuable resource that could help diagnoses of DR in clinical decision support systems. However, most studies for AI-based DR diagnoses are mainly based on medical images; there is limited studies to explore the lesion-related information captured in the free text image reports. Methods In this study, we examined two state-of-the-art transformer-based natural language processing (NLP) models, including BERT and RoBERTa, compared them with a recurrent neural network implemented using Long short-term memory (LSTM) to extract DR-related concepts from clinical narratives. We identified four different categories of DR-related clinical concepts including lesions, eye parts, laterality, and severity, developed annotation guidelines, annotated a DR-corpus of 536 image reports, and developed transformer-based NLP models for clinical concept extraction and relation extraction. We also examined the relation extraction under two settings including ‘gold-standard’ setting—where gold-standard concepts were used–and end-to-end setting. Results For concept extraction, the BERT model pretrained with the MIMIC III dataset achieve the best performance (0.9503 and 0.9645 for strict/lenient evaluation). For relation extraction, BERT model pretrained using general English text achieved the best strict/lenient F1-score of 0.9316. The end-to-end system, BERT_general_e2e, achieved the best strict/lenient F1-score of 0.8578 and 0.8881, respectively. Another end-to-end system based on the RoBERTa architecture, RoBERTa_general_e2e, also achieved the same performance as BERT_general_e2e in strict scores. Conclusions This study demonstrated the efficiency of transformer-based NLP models for clinical concept extraction and relation extraction. Our results show that it’s necessary to pretrain transformer models using clinical text to optimize the performance for clinical concept extraction. Whereas, for relation extraction, transformers pretrained using general English text perform better.more » « less
-
Concept maps have emerged as a valid and reliable method for assessing deep conceptual understanding in engineering education within disciplines as well as interdisciplinary knowledge integration across disciplines. Most work on concept maps, however, focuses on undergraduates. In this paper, we use concept maps to examine changes in graduate students’ conceptual understanding and knowledge integration resulting from an interdisciplinary graduate program. Our study context is pair of foundational, team-taught courses in an interdisciplinary Disaster Resilience and Risk Management (DRRM) graduate program. The courses include a 3-hour research course and a 1-hour seminar that aim to build student understanding within and across Urban Affairs and Planning, Civil and Environmental Engineering, Geosciences, and Business Information Technology. The courses introduce core principles of DRRM and relevant research methods in these disciplines, and drive students to understand the intersections of these disciplines in the context of planning for and responding to natural and human-made disasters. To understand graduate student growth from disciplinary-based to interdisciplinary scholars, we pose the research questions: 1) In what ways do graduate students’ understandings of DRRM change as a result of their introduction to an interdisciplinary graduate research program? and 2) To what extent and in what ways do concept maps serve as a tool to capture interdisciplinary learning in this context? Data includes pre/post concept maps centered on disaster resilience and risk management, a one-page explanation of the post-concept map, and ethnographic field notes gathered from class and faculty meetings. Pre-concept maps were collected on the first day of class; post-concept maps will be collected as part of the final course assignment. We assess the students’ concept maps for depth of conceptual understanding within disciplines and interdisciplinary competency across disciplines, using the field notes to provide explanatory context. The results presented in this paper support the inclusion of an explanation component to concept maps, and also suggest that concept maps alone may not be the best measure of student understanding of concepts within and across disciplines in this specific context. If similar programs wish to use concept maps as an assessment method, we suggest the inclusion of an explanation component and suggest providing explicit instructions that specify the intended audience. We also suggest using a holistic scoring method, as it is more likely to capture nuances in the concept maps than traditional scoring methods, which focus solely on counting factors like hierarchies and number of cross-links.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
