More Like this

1. Is it Fair? Automated Open Response Grading

   Erickson, J.A. ; Botelho, Anthony ( January 2021 , International Conference on Educational Data Mining)

   null (Ed.)

   Online education technologies, such as intelligent tutoring systems, have garnered popularity for their automation. Whether it be automated support systems for teachers (grading, feedback, summary statistics, etc.) or support systems for students (hints, common wrong answer messages, scaffolding), these systems have built a well rounded support system for both students and teachers alike. The automation of these online educational technologies, such as intelligent tutoring systems, have often been limited to questions with well structured answers such as multiple choice or fill in the blank. Recently, these systems have begun adopting support for a more diverse set of question types. More specifically, open response questions. A common tool for developing automated open response tools, such as automated grading or automated feedback, are pre-trained word embeddings. Recent studies have shown that there is an underlying bias within the text these were trained on. This research aims to identify what level of unfairness may lie within machine learned algorithms which utilize pre-trained word embeddings. We attempt to identify if our ability to predict scores for open response questions vary for different groups of student answers. For instance, whether a student who uses fractions as opposed to decimals. By performing a simulated study, we are able to identify the potential unfairness within our machine learned models with pre-trained word embeddings. 

   more » « less
2. Is it Fair? Automated Open Response Grading.

   Erickson, J.A. ; Botelho, A. ; Peng, Z. ; Huang, R ; Kasal, M.V. ; Heffernan, N. ( January 2021 , Educational Data Mining)

   Online education technologies, such as intelligent tutoring systems, have garnered popularity for their automation. Whether it be automated support systems for teachers (grading, feedback, summary statistics, etc.) or support systems for students (hints, common wrong answer messages, scaffolding), these systems have built a well rounded support system for both students and teachers alike. The automation of these online educational technologies, such as intelligent tutoring systems, have often been limited to questions with well structured answers such as multiple choice or fill in the blank. Recently, these systems have begun adopting support for a more diverse set of question types. More specifically, open response questions. A common tool for developing automated open response tools, such as automated grading or automated feedback, are pre-trained word embeddings. Recent studies have shown that there is an underlying bias within the text these were trained on. This research aims to identify what level of unfairness may lie within machine learned algorithms which utilize pre-trained word embeddings. We attempt to identify if our ability to predict scores for open response questions vary for different groups of student answers. For instance, whether a student who uses fractions as opposed to decimals. By performing a simulated study, we are able to identify the potential unfairness within our machine learned models with pre-trained word embeddings. 

   more » « less
3. Is It Fair? Automated Open Response Grading

   Erickson, John A ; Botelho, Anthony F ; Peng, Zonglin ; Huang, Rui ; Kasal, Meghana V ; Heffernan, Neil T. ( June 2021 , Proceedings of the 14th International Conference on Educational Data Mining)

   Online education technologies, such as intelligent tutoring systems, have garnered popularity for their automation. Wh- ether it be automated support systems for teachers (grading, feedback, summary statistics, etc.) or support systems for students (hints, common wrong answer messages, scaffold- ing), these systems have built a well rounded support sys- tem for both students and teachers alike. The automation of these online educational technologies, such as intelligent tutoring systems, have often been limited to questions with well structured answers such as multiple choice or fill in the blank. Recently, these systems have begun adopting support for a more diverse set of question types. More specifically, open response questions. A common tool for developing au- tomated open response tools, such as automated grading or automated feedback, are pre-trained word embeddings. Re- cent studies have shown that there is an underlying bias within the text these were trained on. This research aims to identify what level of unfairness may lie within machine learned algorithms which utilize pre-trained word embed- dings. We attempt to identify if our ability to predict scores for open response questions vary for different groups of stu- dent answers. For instance, whether a student who uses fractions as opposed to decimals. By performing a simu- lated study, we are able to identify the potential unfairness within our machine learned models with pre-trained word embeddings. 

   more » « less
4. The automated grading of student open responses in mathematics

   https://doi.org/10.1145/3375462.3375523  

   Erickson, John A. ; Botelho, Anthony F. ; McAteer, Steven ; Varatharaj, Ashvini ; Heffernan, Neil T. ( March 2020 , Tenth International Conference on Learning Analytics & Knowledge)

   The use of computer-based systems in classrooms has provided teachers with new opportunities in delivering content to students, supplementing instruction, and assessing student knowledge and comprehension. Among the largest benefits of these systems is their ability to provide students with feedback on their work and also report student performance and progress to their teacher. While computer-based systems can automatically assess student answers to a range of question types, a limitation faced by many systems is in regard to open-ended problems. Many systems are either unable to provide support for open-ended problems, relying on the teacher to grade them manually, or avoid such question types entirely. Due to recent advancements in natural language processing methods, the automation of essay grading has made notable strides. However, much of this research has pertained to domains outside of mathematics, where the use of open-ended problems can be used by teachers to assess students' understanding of mathematical concepts beyond what is possible on other types of problems. This research explores the viability and challenges of developing automated graders of open-ended student responses in mathematics. We further explore how the scale of available data impacts model performance. Focusing on content delivered through the ASSISTments online learning platform, we present a set of analyses pertaining to the development and evaluation of models to predict teacher-assigned grades for student open responses. 

   more » « less
5. The automated grading of student open responses in mathematics

   Erickson, J. A. ; Botelho, A. F. ; McAteer, S. ; Varatharaj, A. ; Heffernan, N. T. ( January 2020 , Proceedings of the 10th International Conference on Learning Analytics & Knowledge)

   The use of computer-based systems in classrooms has provided teachers with new opportunities in delivering content to students, supplementing instruction, and assessing student knowledge and comprehension. Among the largest benefits of these systems is their ability to provide students with feedback on their work and also report student performance and progress to their teacher. While computer-based systems can automatically assess student answers to a range of question types, a limitation faced by many systems is in regard to open-ended problems. Many systems are either unable to provide support for open-ended problems, relying on the teacher to grade them manually, or avoid such question types entirely. Due to recent advancements in natural language processing methods, the automation of essay grading has made notable strides. However, much of this research has pertained to domains outside of mathematics, where the use of open-ended problems can be used by teachers to assess students’ understanding of mathematical concepts beyond what is possible on other types of problems. This research explores the viability and challenges of developing automated graders of open-ended student responses in mathematics. We further explore how the scale of available data impacts model performance. Focusing on content delivered through the ASSISTments online learning platform, we present a set of analyses pertaining to the development and evaluation of models to predict teacher-assigned grades for student open responses. 

   more » « less