HBCUs in the United States are higher education institutions dedicated to providing educational opportunities and training, particularly for underprepared students, with a focus on African Americans [1]. Despite their critical role, HBCUs face persistent challenges, including lower retention and graduation rates compared to other institutions [2]. According to data from the National Center for Education Statistics (NCES), the six-year graduation rate at HBCUs is consistently lower than at other institutions, underscoring a significant disparity and emphasizing the need for targeted
The associate editor coordinating the review of this manuscript and approving it for publication was Martin Reisslein .
strategies to enhance student success [3]. In this context, early identification of students at risk of academic failure is crucial, as timely interventions can play a pivotal role in improving retention and increasing graduation rates.
DKT is a powerful method for modeling and predicting students' mastery of concepts and skills over time [4], offering significant potential for the early identification of students at risk of academic failure. Unlike conventional methods that rely solely on grades or assessments, DKT leverages deep learning algorithms to analyze sequential student interactions with learning materials, including exercises, quizzes, and assignments. By capturing the dynamic and temporal aspects of learning, DKT predicts future learning outcomes, aiding educators in understanding and addressing students' individual progress and needs. This provides a more intricate comprehension of students' knowledge acquisition processes, facilitating timely interventions and personalized support [5].
This study leverages students' historical data to predict their likelihood of passing or failing courses using DKT. This approach facilitates the early identification of students at risk of academic failure, enabling timely interventions to improve retention and graduation rates. First, the study constructs a comprehensive dataset focused on STEM education at HBCUs, with a particular emphasis on engineering disciplines. The dataset comprises 352, 148 learning records from 17, 181 undergraduate students across eight colleges at PVAMU, with a specialized focus on the College of Engineering. It addresses the unique challenges and opportunities in engineering education by detailing tailored data collection processes, model development strategies, and evaluation methodologies. Second, the study applies multiple SOTA DKT models-DKT [4], a variant (DKT+) [6], DKVMN [7], Self-Attentive Knowledge Tracing (SAKT) [8], and Knowledge Query Network (KQN) [9]-to assess their performance in predicting students' academic outcomes. Experimental results demonstrate the efficiency of these models, with DKVMN achieving superior accuracy and AUC scores. The findings have important implications for personalized adaptive learning and educational practices within HBCUs, potentially offering insights into leveraging predictive analytics to enhance student success and address retention challenges effectively.
The contributions of this study are summarized as follows:
• To the best of our knowledge, this is the first dataset to trace student learning outcomes at HBCUs using DKT by predicting their likelihood of passing or failing courses. It not only expands data availability for knowledge tracing through machine learning but also serves as a valuable resource for validating DKT models in the context of HBCUs. Additionally, this dataset provides a foundation for further research on adaptive learning systems, enabling the development of personalized interventions to support at-risk students. By facilitating more accurate predictions of academic performance, it contributes to the broader goal of enhancing student success and retention rates within HBCU institutions.
• This study provides a comprehensive evaluation of the effectiveness and limitations of SOTA DKT models in predicting SLO within the context of HBCUs. Notably, DKVMN outperforms other models, leveraging advanced mechanisms to better capture student learning patterns. These findings underscore the potential of DKT in identifying students at risk of underperformance, enabling timely and proactive interventions to support their academic progress and success at HBCUs. Furthermore, the strong predictive performance on SLO suggests promising applications of DKT techniques for enhancing graduation rates, particularly in STEM education at HBCUs.
This study aims to trace the student learning outcomes through predicting the passing or failing courses via DKT at HBCUs. Given an observed sequence of learning records C for a student, the goal is to predict the next learning record(s) (C t+1 , C t+2 , . . . ) based on historical learning records. Let C = (C 1 , C 2 , . . . , C t ) be an input sequence of the historical learning records with length t, where each C i is the i-th learning record in the sequence. C t+1 be the next learning record to be predicted. f θ be a function parameterized by θ that models the conditional probability of passing or failing the next learning record given the input sequence.
where C 1:t = (C 1 , C 2 , . . . , C t ) represents the history of learning records. Figure 1 illustrates an example scenario of this task, where the input includes the student's historical course information, comprising the SLO from course 1 to t and the corresponding skills. Here, skill refers to the knowledge required for the course. For example, course C 2 involves freshman-level mathematics knowledge (MATH for Mathematics and 1000 for the freshman level) as well as freshman-level general engineering knowledge (GNEG for General Engineering and 1000 for the freshman level). The output consists of the SLO for courses t + 1 to n, where n represents the total number of courses taken by the student.
To effectively implement SLO tracing, it is essential to collect comprehensive learning records from HBCUs. PVAMU, an HBCU, is Texas's second-oldest public university and a distinguished land-grant institution, currently serving 9,716 students. Nationally, it ranks #1 and #3 in producing African American males and females, respectively, who have earned bachelor's degrees in engineering disciplines. This study establishes the first dataset for tracing student learning outcomes at PVAMU.
A. DATA INTRODUCTION This study collected a real dataset of undergraduate students from the Student Information System at PVAMU. The data spans four years, from fall 2019 to summer 2023. The collected data includes five essential elements: Academic Year, Universal ID, Course Subject, Course Level, and Pass/Fail grades. Table 1 provides a few examples of learning records. Each student is assigned a unique Universal ID. The course subject specifies the knowledge area taught; for example, BIOL represents Biology. The course level indicates the academic level of the course, categorized as follows: 1000-1999 (Freshman Level), 2000-2999 (Sophomore Level), 3000-3999 (Junior Level), and 4000-4999 (Senior Level). The Pass/Fail grade indicates whether the student successfully completed the course. These attributes are strictly academic and relate to SLO in their courses over the academic years.
To prepare data for DKT, a few steps of data preprocessing are completed: 1) Data records lacking grades, incomplete courses, or non-gradable courses were excluded from the dataset; 2) The features comprise categorical data, which is incompatible with machine learning algorithms. Consequently, it is necessary to transform these categorical course data into a numerical format before inputting them into the machine learning model. In this study, we employed a technique known as label encoding to convert our categorical data into numerical values [10]. This method assigns an integer to each distinct nominal variable. Course subject and level features were transformed into integer numbers, representing the student's knowledge skill; 3) Courses sharing the same subject and level were treated as possessing the same knowledge skill. For example, courses like MATH 3201 and MATH 3800, both falling under the 3000-level math category, were considered part of the same skill. Table 2 shows a few examples of mapping course subject and level encoded to integer number as skill_id.
Over the four-year period, the total university (UNIV) dataset comprises 352, 148 records for undergraduate students, where Table 3 shows the statistics of the dataset and sub datasets.
TABLE 2. Examples of mapping course subjects and levels to integer values, encoded as skill_id. ACCT, ADMN, AFAM, SPED, SPMT refer to Account, Administration, Africana Studies, Special Education, and Sport Management. TABLE 3. Data distributions. COE and COAS stand for the college of engineering and the college of arts and sciences, respectively.
Following the exclusion of incomplete, non-gradable, and no-grade information, 326, 269 records remain. A total of 17, 181 students participated in 2, 124 courses, with courses sharing the same subject and level treated as possessing the same knowledge skill. The research identifies a total of 233 knowledge components (KCs) among all undergraduate students. In the subset of records from the College of Engineering (COE), the dataset comprises 52, 206 records. Following the exclusion of incomplete, non-gradable, and no-grade information, 46, 477 records remain. A total of 2, 387 students participated in 1, 027 courses. In the entirety of the university college system, the College of Arts and Sciences (COAS) stands as the largest academic entity. Consequently, we amalgamated student data from both COAS and COE disciplines to form another subset for training purposes. Within the subset of records originating from COE and COAS, the dataset encompasses 143, 982 entries. After removing incomplete, non-gradable, and no-grade data, 131, 857 records persist. Across 1, 797 courses, a collective of 7, 210 students took part in the study. This nuanced analysis not only provides insights into the overall predictive power of ML models but also delves into the domainspecific efficacy, showcasing the potential impact on targeted academic success initiatives. Table 4 represents the final dataset utilized for training and testing SLO predictions, where the DKT models discern correlations among courses, enabling the prediction of a student's likelihood of passing or failing a course based on historical records.
DKT has emerged as an advanced approach to enhancing higher education outcomes by enabling personalized learning
TABLE 4. Examples of samples for tracing SLO.
experiences and more accurate assessments of student knowledge mastery. Over time, several variants of DKT have been proposed, each introducing unique enhancements to address specific challenges. In this study, we delve into the methodologies of five prominent DKT models: DKT, DKT+, DKVMN, SAKT, and KQN to implement SLO tracing through predicting whether a student will pass or fail a course for HBUC education.
• DKT [4] is the first work to apply neural networks for knowledge tracing tasks. It uses recurrent neural networks (RNNs) to effectively capture the temporal dependencies inherent within a sequence of interactions comprising a student's questions and corresponding answers. This enables the model to predict a student's response to a new question based on their historical interactions.
A fundamental representation of a simple RNN network for DKT is defined as follows:
where tanh is the activation function, W hs is the input weights, W hh is the recurrent weights, W yt is the readout weights, and b h and b y are the bias terms. Within the DKT framework, the hidden state ht of the RNN is interpreted as the latent representation of the student's knowledge state. Moreover, h t is subjected to prediction through a Sigmoid-activated linear layer, denoted as y t . This layer is of the same length as the number of exercises, with each element representing the model's predicted probability of the student correctly answering the corresponding exercise. The empirical findings indicated that DKT surpassed traditional KT models across multiple benchmark datasets. This highlights the potential of deep learning models in predicting a student's response to new questions based on their past interactions, facilitating SLO tracing.
• DKT+ [6] presents a notable extension of the DKT framework, potentially enhancing SLO tracing. It introduces regularization terms that correspond to reconstruction and waviness to the loss function of the original DKT model to enhance the consistency in prediction.
The initial loss function is expanded by integrating three regularization terms, resulting in the subsequent regularized loss function:
where λ r , λ w1 , and λ w2 are regularization parameters. This refinement was devised to mitigate inherent constraints observed in DKT, particularly in its proficiency to accurately reconstruct input responses and diminish incongruities in predicting answers for questions associated with similar KCs.
• DKVMN [7] incorporates a key-value memory matrix, where each row represents an item or skill, and the columns correspond to key-value pairs associated with that item. The memory allows the model to store information about students' mastery of different skills and update this information as they interact with educational items. When a student responds to an item, the memory is updated to reflect their mastery of that item. The update process involves modifying the values associated with relevant keys in the memory, where the memory update for item i and key k is computed as:
where i is the index of the item, k represents the key associated with the item, and update k is the update value for key k, computed based on the student's response to the item. This study utilized a memory mechanism in DKVMN to improve SLO tracing.
• SAKT [8] tried to handle sparse data in which students interact with few KCs. It adopts a transformerbased architecture, replacing Long Short-Term Memory (LSTMs) with self-attention mechanisms to capture the relevance between the KCs and the students' historical interactions. SAKT employs a novel architecture that allows for the dynamic weighting of input sequences, enabling the model to focus on the most relevant interactions within a student's learning history. The self-attention mechanism computes attention scores for each pair of elements in a sequence and then calculates a weighted sum of the values based on these scores.
Here's the formula for computing the attention score for element i with respect to element j is computed as:
where Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimensionality of the key vectors. The self-attention mechanism in SAKT will benefit SLO tracing by assigning varying weights to historical learning records.
• KQN [9] combines memory-augmented structures with attention mechanisms, encoding each student's current knowledge state as a query vector. The attention mechanisms enable KQN to retrieve pertinent information from its memory, prioritizing the most relevant items for each student. The attention weights are computed based on the similarity between each node and the current knowledge query, ensuring that the model prioritizes nodes most relevant to the student's current knowledge state. The attention weights α ij for each node j in the graph based on its similarity to the knowledge query q i are computed as:
where h j represents the node embedding for node j, q i is the knowledge query for student i. Furthermore, KQN introduces a novel concept called probabilistic skill similarity, which relates pairwise cosine and Euclidean distances between skill vectors to the odds ratios of corresponding skills, making KQN interpretable and intuitive [9]. KQN will be employed to improve SLO tracing through networking weighted historical learning records.
This section aims to validate DKT-based SLO tracing across different institutional scales, specifically within the College of Engineering and at the university-wide level.
We adhere to conventional practices by allocating dataset resources primarily for training purposes, typically dedicating 80% to 90% for this phase, while reserving the remaining portion for testing. Leveraging a four years of authentic data, we aim to predict student achievement in contemporary courses by analyzing historical academic performance. Specifically, the initial three years of data serve as the training dataset, while the fourth year is designated for testing. representation, presenting challenges in model training for SLO tracing. For the testing phase, we exclusively utilize data from the five departments within the College of Engineering: Civil and Environment Engineering (CEE), Chemical Engineering (CHE), Computer Science (CSC), Electrical and Computer Engineering (ECE), and Mechanical Engineering (MCE). Detailed insights into these Engineering departments are depicted in Figure 3. Furthermore, Table 5 presents details of data distributions for training and testing of DKT. TABLE 5. Training and testing dataset statistics for SLO tracing. CEE, CHE, CSC, ECE, and MCE refer to departments of Civil & Environment, Chemical Engineering, Computer Science, Electrical and Computer Engineering, and Mechanical Engineering, respectively.
In this study, we employed the default settings and configurations of DKT models sourced from the GitHub repository. The original PyTorch implementation of these DKT models can be accessed at https://github.com/hcnoh/knowledge-tracing-collection-pytorch.
• Batch Size: The batch size denotes the number of instances processed in a single iteration during the training process. The default value is set to 256.
• Number of Epochs: Epochs represent the number of complete passes through the entire training dataset. The default number of epochs utilized in this research is 100.
• Learning Rate: This parameter governs the step size during the optimization process of the training algorithm.
The default learning rate employed in this study is 0.001.
• Optimizer: The optimizer determines the specific algorithm employed to optimize the model parameters during the training process. The available optimizers include Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam). For this research, the default optimizer chosen is Adam.
• Sequence Length: The sequence length represents the number of time steps or elements considered in the dataset for each training instance. The default sequence length utilized in this research is 100. These default settings serve as the foundational configuration for the DKT models applied throughout the experimental investigations conducted in this study. These DKT models were developed using PyTorch and trained on a workstation equipped with NVIDIA V-100 GPUs.
In terms of the task definition, accuracy, recall, precision, F1-score, and the Area Under the Receiver Operating Characteristic Curve (AUC) are employed as evaluation metrics.
Accuracy, a commonly employed metric in classification tasks, measures the ratio of correctly classified SLO (passing or failing courses), encompassing both true positives and true negatives, relative to the total SLO evaluated, thereby providing a straightforward assessment of prediction accuracy. Accuracy is ideal for SLO tracing on balanced datasets but can be misleading in imbalanced datasets, as a model that predicts only the majority class may still achieve high accuracy.
where True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN) come from confusion matrix terminology.
• TP: correct predictions of passing courses.
• TN: correct predictions of failing courses.
• FP: incorrect predictions of passing courses.
• FN: incorrect predictions of failing courses.
Precision measures the proportion of true positive predictions out of all positive predictions. Recall measures the proportion of true positive predictions out of all actual positive instances. The F1 score is a metric used to evaluate the performance of a classification model, particularly when dealing with imbalanced classes. It combines both precision and recall into a single measure.
Conversely, AUC serves as a valuable metric for assessing the performance of binary classifiers, particularly in scenarios characterized by class imbalance or variable importance of FP and FN. A higher AUC value signifies superior discrimination capabilities of the model in distinguishing between positive and negative classes [11].
The findings and discussions present the comprehensive prediction performance analysis of DKT models across various testing and training datasets, as delineated in Tables 6 through 13. These tables encapsulate the AUC and accuracy scores, providing insights into the model's efficacy. Recall, Precision, and F1 scores, further enrich our understanding of the model's capabilities under different training and testing conditions.
Table 6 and Table 7 illustrate the performance of SLO tracing when tested on five engineering departments, including CEE, CHE, CSC, ECE, and MCE, with training on the COE dataset. In Table 6, the DKVMN model consistently achieves the highest accuracy across all departments, with an average accuracy of 0.7892. This suggests that DKVMN is better at predicting SLO. The AUC scores show more variability across models and departments. While DKVMN has good AUC, it's not always the highest. This indicates that while DKVMN is good at overall prediction (accuracy), other models might be better at distinguishing between positive and negative cases (AUC). In Table 7, all models exhibit high recall scores, generally above 0.85, indicating that they are good at identifying positive cases (e.g., students who will pass the courses). DKVMN and SAKT generally achieve the best F1-scores, reinforcing their strong performance.
Table 8 and Table 9, which test the same engineering departments but with training on a larger dataset (COE + COAS dataset), show improved performance with average metrics of AUC 0.6820, accuracy 0.8092, and F1 score 0.8884. This highlights the impact of data diversity on model performance. Training on the combined COE + COAS dataset results in better performance in SLO tracing, suggesting that the models are sensitive to the characteristics of the training data. This implies that model selection and tuning may need to be adjusted based on the specific dataset used for training. Additionally, while departmental differences in SLO tracing performance are still evident, the patterns may differ slightly from those observed in Table 6 and Table 7.
To further improve SLO tracing across different departments within COE, the full dataset from PVAMU (UNIV data) is
TABLE 6. Performance comparison between engineering departments via Accuracy and AUC when DKT models are trained on COE data. EM denotes evaluation metrics. TABLE 7. Performance comparison between engineering departments via Recall, Precision and F1 scores between engineering departments when DKT models are trained on COE data. TABLE 8. Comparing AUC and Accuracy for each engineering department when DKT models are trained on COE and COAS data. TABLE 9. Comparing Recall, Precision and F1 scores for each engineering department when DKT models are trained on COE and COAS data. 10 and Table 11 present the comparison results. The performance differences between models are generally smaller when trained on the UNIV data compared to the COE + COAS dataset. This indicates that incorporating data from additional departments and colleges beyond COE and COAS does not significantly enhance model performance. DKVMN continues to be a strong performer on the UNIV dataset, though its advantage is slightly reduced compared to training on COE + COAS data. Furthermore, departmental differences in SLO tracing performance remain evident across all groups, emphasizing the importance of accounting for department-specific characteristics.
Table 12 and Table 13 illustrate the performance of these models when tested on COE and UNIV with training on the UNIV dataset. The consistent ranking of different DKT models across COE and UNIV testings indicates that the relative performance of the models remains largely unaffected by the testing datasets. Furthermore, the improved performance on the UNIV dataset highlights the generalization and applicability to a broader student population. Additionally, DKVMN consistently outperforms other models across all metrics and evaluation subsets, reinforcing its effectiveness in SLO tracing.
In summary, training models on different datasets (COEonly, COE+COAS, and UNIV) results in varying performance outcomes. Models trained on the full university dataset (UNIV) tend to achieve better generalization. Among the evaluated models, DKVMN consistently demonstrates strong performance across different datasets and evaluation metrics, including accuracy, AUC, recall, precision, and F1-score. However, other models, such as SAKT and KQN, also exhibit competitive performance, excelling in certain metrics or specific departments. Additionally, significant performance differences are observed across engineering departments (CEE, CHE, CSC, ECE, and MCE), emphasizing the need to consider department-specific characteristics. This suggests
TABLE 10. Comparing AUC and Accuracy for each engineering department when DKT models are trained on UNIV data. TABLE 11. Comparing Recall, Precision and F1 scores for each engineering department when DKT models are trained on UNIV data. TABLE 12. Comparing AUC and Accuracy for and UNIV when DKT models are trained on UNIV data. TABLE 13. Comparing Recall, Precision and F1 scores for COE and UNIV when DKT models are trained on UNIV data.
the potential benefit of developing tailored models or features for different departments.
Knowledge tracing involves modeling students' learning progress based on their activity sequences, making it a challenging task due to the need for accurate performance prediction and understanding of students' mastery levels. Researchers have tackled this challenge through various approaches. There are two most representative models: Bayesian Knowledge Tracking (BKT) and DKT. Early efforts focused on BKT models [12], wherein each student's knowledge state is represented as a binary variable. These models employ probabilistic techniques like the Hidden Markov Model (HMM) to assess students' grasp of concepts [13]. It characterizes a student's progress in tackling problems associated with a specific concept by utilizing a binary indicator (either correct or incorrect) and consistently refines its assessment of the student's understanding of that concept [14].
DKT [4], a pioneering algorithm designed to model students' learning states through recurrent neural networks (RNN), has demonstrated significant enhancements in predictive accuracy. It grasps the temporal nuances within a sequence of student-question interactions. Empirical findings demonstrated DKT's superiority over traditional knowledge tracing models across various benchmark datasets [15].
The DKT models can be categorized into the following groups [16]:
• DKT and its Variants. Early deep learning KT models often used RNNs architectures like LSTM or Gated Recurrent Unit (GRU). RNNs are well-suited for sequential data, making them appropriate for modeling students' learning trajectories over time. e.g., DKT [4] and DKT+ [6].
• Memory Network based models. Memory networks introduce an external memory component that the model can read from and write to at each time step. This memory serves as a repository for storing past interactions, knowledge concepts, or context information. By accessing this memory, the model can maintain a richer representation of the student's learning history and better infer their current knowledge state. e.g., DKVMN [7], and SKVMN [17].
• Attention Mechanism based models. To improve the modeling of long-term dependencies and focus on relevant parts of the input sequence, attention mechanisms were introduced in KT models. Attention mechanisms allow the model to dynamically weigh the importance of different time steps or features, enhancing its predictive power. e.g., SAKT [8], DKVMN [7] and SAINT [18].
• Graph Structure based models. Another direction of development involves using graph neural networks to represent the knowledge structure and student interactions as graphs. Graph Neural Networks (GNNs) allow models to capture complex relationships between knowledge concepts and how they influence each other's mastery. e.g., GKT [19]. The pioneering use of DKT in online exercises has significantly advanced the prediction of students' performance. By leveraging past student records, the original DKT model successfully forecasts whether a student will pass or fail their current academic exercises. This research extends beyond mere prediction, utilizing past grades to anticipate students' future course outcomes. The research findings underscore the effectiveness of DKT models in accurately predicting students' academic trajectories, thereby highlighting their potential for enhancing educational outcomes.
This study investigated SLO tracing at PVAMU by leveraging students' historical data to predict course pass/fail likelihood using DKT models. First, it constructed the first dataset for SLO tracing, encompassing eight colleges at PVAMU. Then, five DKT models were applied to evaluate SLO tracing performance across five metrics. Experimental results demonstrate that DKT models effectively trace SLO across various departments within the College of Engineering, with DKVMN consistently achieving strong performance across different training datasets and evaluation metrics.
Future research directions include: 1) Expanding the dataset to cover additional HBCUs, such as Alabama A&M University, and research-intensive institutions like Texas A&M University. Additionally, more advanced DKT models, such as DGEKT [20] and DMKT [21], will be employed to further validate the generalizability and robustness of SLO tracing; 2) Enhancing SLO tracing by incorporating individual student differences, including learning styles, cognitive abilities, and cultural backgrounds, to provide more personalized learning support; 3) Implementing near real-time SLO tracing to support online education by deploying models on multi-GPU platforms. The effectiveness of DKT-based SLO tracing will also be indirectly assessed through its impact on learning path recommendation [22] for developing personalized adaptive learning systems; 4) Improving the interpretability of DKT-based SLO tracing by integrating item response theory to provide insights into student learning abilities and course difficulty levels; 5) Further enhancing SLO tracing by incorporating personalized adaptive learning pedagogies and leveraging data augmentation techniques, such as diffusion models, to improve model robustness and generalization.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of NSF or the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. Additionally, they acknowledge the use of AI-based tools, such as ChatGPT, for assistance in editing, grammar enhancement, and spelling checks during the preparation of this manuscript.
The Authors. This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/ VOLUME 13, 2025
VOLUME 13, 2025