More Like this

1. Instructional Activity Detection Using Deep Neural Networks

   https://doi.org/10.1109/DSP58604.2023.10167935  

   Korban, Matthew; Youngs, Peter; Acton, Scott T (June 2023, IEEE)

   Korban, Matthew; Youngs, Peter; Acton, Scott T (Ed.)

   Analyzing instructional videos via computer vision and machine learning holds promise for several tasks, such as assessing teacher performance and classroom climate, evaluating student engagement, and identifying racial bias in instruction. The traditional way of evaluating instructional videos depends on manual observation with human raters, which is time-consuming and requires a trained labor force. Therefore, this paper tests several deep network architectures in the automation of instruc- tional video analysis, where the networks are tailored to recognize classroom activity. Our experimental setup includes a set of 250 hours of primary and middle school videos that are annotated by expert human raters. We present several strategies to handle varying length of instructional activities, a major challenge in the detection of instructional activity. Based on the proposed strategies, we enhance and compare different deep networks for detecting instructional activity. 

   more » « less
2. Robust Activity Recognition for Adaptive Worker-Robot Interaction using Transfer Learning

   Shahnavaz, Farid; Tavassoli, Riley; Akhavian, Reza (January 2023, 2023 ASCE International Conference on Computing in Civil Engineering)

   Turkan, Yelda; Louis, Joseph; Leite, Fernanda; Ergan, Semiha (Ed.)

   Human activity recognition (HAR) using machine learning has shown tremendous promise in detecting construction workers’ activities. HAR has many applications in human-robot interaction research to enable robots’ understanding of human counterparts’ activities. However, many existing HAR approaches lack robustness, generalizability, and adaptability. This paper proposes a transfer learning methodology for activity recognition of construction workers that requires orders of magnitude less data and compute time for comparable or better classification accuracy. The developed algorithm transfers features from a model pre-trained by the original authors and fine-tunes them for the downstream task of activity recognition in construction. The model was pre-trained on Kinetics-400, a large-scale video-based human activity recognition dataset with 400 distinct classes. The model was fine-tuned and tested using videos captured from manual material handling (MMH) activities found on YouTube. Results indicate that the fine-tuned model can recognize distinct MMH tasks in a robust and adaptive manner which is crucial for the widespread deployment of collaborative robots in construction. 

   more » « less
3. Automated Classification of Elementary Instructional Activities: Analyzing the Consistency of Human Annotations

   https://doi.org/10.18608/jla.2024.8323  

   Foster, Jonathan K; Youngs, Peter; Van_Aswegen, Rachel; Singh, Samarth; Watson, Ginger S; Acton, Scott T (April 2024, Journal of Learning Analytics)

   Khosravi, H (Ed.)

   Despite a tremendous increase in the use of video for conducting research in classrooms as well as preparing and evaluating teachers, there remain notable challenges to using classroom videos at scale, including time and financial costs. Recent advances in artificial intelligence could make the process of analyzing, scoring, and cataloguing videos more efficient. These advances include natural language processing, automated speech recognition, and deep neural networks. To train artificial intelligence to accurately classify activities in classroom videos, humans must first annotate a set of videos in a consistent way. This paper describes our investigation of the degree of inter-annotator reliability regarding identification of and duration of activities among annotators with and without experience analyzing classroom videos. Validity of human annotations is crucial for research involving temporal analysis within classroom video research. The study reported here represents an important step towards applying methods developed in other fields to validate temporal analytics within learning analytics research for classifying time- and event-based activities in classroom videos. 

   more » « less
4. A multi-modal transformer network for action detection.

   https://doi.org/10.1016/j.patcog.2023.109713  

   Korban, M.; Youngs, P.; Acton, S (October 2023, Pattern recognition)

   Hancock, E. (Ed.)

   This paper proposes a multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a novel multi-modal attention mechanism that captures the correlations between different combinations of spa- tial and motion modalities. Exploring such correlations for actions effectively has not been explored before. We also suggest an algorithm to correct the motion distortion caused by camera movements. Such motion distortion severely reduces the expressive power of motion features represented by optical flow vectors. We also introduce a new instructional activity dataset that includes classroom videos from K-12 schools. We conduct comprehensive ex- periments to evaluate the performance of different approaches on our dataset. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet, and our instructional activity dataset. 

   more » « less
5. Investigating racial and skin tone biases in automated classification of teachers’ activities in classroom videos.

   Drimalla, J; Korban, M; Foster, J (July 2025, Springer Nature Switzerland)

   Previous studies have shown that artificial intelligence can be used to classify instruction-related activities in classroom videos. The automated classi- fication of human activities, however, is vulnerable to biases in which the model performs substantially better or worse for different people groups. Although algo- rithmic bias has been highlighted as an important area for research in artificial intelligence in education, there have been few studies that empirically investigate potential bias in instruction-related activity recognition systems. In this paper, we report on an investigation of potential racial and skin tone biases in the automated classification of teachers’ activities in classroom videos. We examine whether a neural network’s classification of teachers’ activities differs with respect to teacher race and skin tone and whether differently balanced training datasets affect the performance of the neural network. Our results indicate that, under ordinary class- room lighting conditions, the neural network performs equally well regardless of teacher race or skin tone. Furthermore, our results suggest the balance of the training dataset with respect to teacher skin tone and race has a small—but not necessarily positive—effect on the neural network’s performance. Our study, how- ever, also suggests the importance of quality lighting for accurate classification of teacher-related instructional activities for teachers of color. We conclude with a discussion of our mixed findings, the limitations of our study, and potential directions for future research. 

   more » « less