An official website of the United States government
The rise of Large Language Models (LLMs) as powerful knowledge-processing tools has sparked a wave of innovation in tutoring and assessment systems. Despite their well-documented limitations, LLMs offer unique capabilities that have been effectively harnessed for automated feedback generation and grading in intelligent learning environments. In this paper, we introduce {\em Project 360}, an experimental intelligent tutoring system designed for teaching SQL. Project 360 leverages the concept of {\em query equivalence} to assess the accuracy of student queries, using ChatGPT’s advanced natural language analysis to measure their semantic distance from a reference query. By integrating LLM-driven evaluation, Project 360 significantly outperforms traditional SQL tutoring and grading systems, offering more precise assessments and context-aware feedback. This study explores the feasibility and limitations of using ChatGPT as the analytical backbone of Project 360, evaluating its reliability for autonomous tutoring and assessment in database education. Our findings provide valuable insights into the evolving role of LLMs in education, highlighting their potential to revolutionize SQL learning while identifying areas for further refinement and improvement.
more »
« less
This content will become publicly available on March 19, 2026
An Experiment with LLMs as Database Design Tutors: Persistent Equity and Fairness Challenges in Online Learning
As large language models (LLMs) continue to evolve, their capacity to replace humans as their surrogates is also improving. As increasing numbers of intelligent tutoring systems (ITSs) are embracing the integration of LLMs for digital tutoring, questions are arising as to how effective they are and if their hallucinatory behaviors diminish their perceived advantages. One critical question that is seldom asked if the availability, plurality, and relative weaknesses in the reasoning process of LLMs are contributing to the much discussed digital divide and equity and fairness in online learning. In this paper, we present an experiment with database design theory assignments and demonstrate that while their capacity to reason logically is improving, LLMs are still prone to serious errors. We demonstrate that in online learning and in the absence of a human instructor, LLMs could introduce inequity in the form of “wrongful” tutoring that could be devastatingly harmful for learners, which we call ignorant bias, in increasingly popular digital learning. We also show that significant challenges remain for STEM subjects, especially for subjects for which sound and free online tutoring systems exist. Based on the set of use cases, we formulate a possible direction for an effective ITS for online database learning classes of the future.
more »
« less
- Award ID(s):
- 2410668
- PAR ID:
- 10631923
- Publisher / Repository:
- MDPI
- Date Published:
- Journal Name:
- Education Sciences
- Volume:
- 15
- Issue:
- 3
- ISSN:
- 2227-7102
- Page Range / eLocation ID:
- 386
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Most of today’s educators are in no shortage of digital and online learning technologies available at their fingertips, ranging from Learning Management Systems such as Canvas, Blackboard, or Moodle, online meeting tools, online homework, and tutoring systems, exam proctoring platforms, computer simulations, and even virtual reality/augmented reality technologies. Furthermore, with the rapid development and wide availability of generative artificial intelligence (GenAI) services such as ChatGPT, we are just at the beginning of harnessing their potential to transform higher education. Yet, facing the large number of available options provided by cutting-edge technology, an imminent question on the mind of most educators is the following: how should I choose the technologies and integrate them into my teaching process so that they would best support student learning? We contemplate over these types of important and timely questions and share our reflections on evidence-based approaches to harnessing digital learning tools using a Self-regulated Engaged Learning Framework we have employed in our research in physics education that can be valuable for educators in other disciplines.more » « less
-
Introductory engineering courses at large universities often number over a hundred students, while online classes can have even larger enrollments, significantly constraining instructors’ ability to provide feedback on homework, including the free-body diagrams (FBDs). Most online homework systems do not provide feedback on FBDs if the systems even allow the submission, and instructors often lack time or resources to provide this. A few systems have been developed that use a menu-based system allowing students to creative FBDs. There is a growing concern amongst engineering educators that student lacks critical sketching skills and the ability to idealize a real-world system as a free body diagram (FBD). A sketch-recognition based tutoring system, Mechanix, allows learners to hand-draw solutions just as they would with pencil and paper, while also providing iterative real-time personalized feedback. Sketch recognition algorithms use artificial intelligence to identify the shapes, their relationships, and other features of the sketched student drawing. Other AI algorithms then determine if and why a student’s work is incorrect, enabling the tutoring system to return immediate and iterative personalized feedback facilitating student learning that is otherwise not possible in large classes. Preliminary results using Mechanix, a sketch-based statics tutoring system built at Texas A&M University suggest that a sketch-based tutoring system increases homework motivation in struggling students and is as effective as paper-and-pencil-based homework for teaching method of joints truss analysis. The current project implements Mechanix at five different universities obtaining Pre/Post Concept Inventory, homework, and exam scores. It is compared against either the university's current online system or paper-based homework. Focus groups provide further insight into the students’ perceptions about the impact of Mechanix on their learning.more » « less
-
Olney, AM; Chounta, IA; Liu, Z; Santos, OC; Bittencourt, II (Ed.)An advantage of Large Language Models (LLMs) is their contextualization capability – providing different responses based on student inputs like solution strategy or prior discussion, to potentially better engage students than standard feedback. We present a design and evaluation of a proof-of-concept LLM application to offer students dynamic and contextualized feedback. Specifically, we augment an Online Programming Exercise bot for a college-level Cloud Computing course with ChatGPT, which offers students contextualized reflection triggers during a collaborative query optimization task in database design. We demonstrate that LLMs can be used to generate highly situated reflection triggers that incorporate details of the collaborative discussion happening in context. We discuss in depth the exploration of the design space of the triggers and their correspondence with the learning objectives as well as the impact on student learning in a pilot study with 34 students.more » « less
-
Introductory engineering courses within large universities often have annual enrollments exceeding several hundreds of students, while online classes have even larger enrollments. It is challenging to achieve differentiated instruction in classrooms with class sizes and student diversity of such great magnitude. In such classes, professors assess whether students have mastered a concept through multiple-choice questions, marking answers as right or wrong with little feedback, or using online text-only systems. However, in these scenarios the feedback is of a mostly binary nature (right or wrong) with limited constructive feedback to scaffold learning. A growing concern among engineering educators is that students are losing both the critical skill of sketched diagrams and the ability to take a real system and reduce it to an accurate but simplified free-body diagram (FBD). A sketch-recognition based tutoring system, called Mechanix, allows students to hand-draw solutions just as they would with pencil and paper, while also providing iterative real-time personalized feedback. Sketch recognition algorithms use artificial intelligence to identify the shapes, their relationships, and other features of the sketched student drawing. Other AI algorithms then determine if and why a student’s work is incorrect, enabling the tutoring system to return immediate and iterative personalized feedback facilitating student learning that is otherwise not possible in large classes. To observe the effectiveness of this system, it has been implemented into various courses at three universities, with two additional universities planning to use the system within the next year. Student knowledge is measured using Concept Inventories based in both Physics and Statics, common exam questions, and assignments turned in for class. Preliminary results using Mechanix, a sketch-based statics tutoring system built at Texas A&M University, suggest that a sketch-based tutoring system increases homework motivation in struggling students and is as effective as paper-and-pencil-based homework for teaching method of joints truss analysis. In focus groups, students believed the system enhanced their learning and increased engagement. Keywords: sketch recognition; intelligent user interfaces; physics education; engineering educationmore » « less
