More Like this

1. Instructional Activity Detection Using Deep Neural Networks

   https://doi.org/10.1109/DSP58604.2023.10167935  

   Korban, Matthew; Youngs, Peter; Acton, Scott T (June 2023, IEEE)

   Korban, Matthew; Youngs, Peter; Acton, Scott T (Ed.)

   Analyzing instructional videos via computer vision and machine learning holds promise for several tasks, such as assessing teacher performance and classroom climate, evaluating student engagement, and identifying racial bias in instruction. The traditional way of evaluating instructional videos depends on manual observation with human raters, which is time-consuming and requires a trained labor force. Therefore, this paper tests several deep network architectures in the automation of instruc- tional video analysis, where the networks are tailored to recognize classroom activity. Our experimental setup includes a set of 250 hours of primary and middle school videos that are annotated by expert human raters. We present several strategies to handle varying length of instructional activities, a major challenge in the detection of instructional activity. Based on the proposed strategies, we enhance and compare different deep networks for detecting instructional activity. 

   more » « less
2. Robust Activity Recognition for Adaptive Worker-Robot Interaction using Transfer Learning

   Shahnavaz, Farid; Tavassoli, Riley; Akhavian, Reza (January 2023, 2023 ASCE International Conference on Computing in Civil Engineering)

   Turkan, Yelda; Louis, Joseph; Leite, Fernanda; Ergan, Semiha (Ed.)

   Human activity recognition (HAR) using machine learning has shown tremendous promise in detecting construction workers’ activities. HAR has many applications in human-robot interaction research to enable robots’ understanding of human counterparts’ activities. However, many existing HAR approaches lack robustness, generalizability, and adaptability. This paper proposes a transfer learning methodology for activity recognition of construction workers that requires orders of magnitude less data and compute time for comparable or better classification accuracy. The developed algorithm transfers features from a model pre-trained by the original authors and fine-tunes them for the downstream task of activity recognition in construction. The model was pre-trained on Kinetics-400, a large-scale video-based human activity recognition dataset with 400 distinct classes. The model was fine-tuned and tested using videos captured from manual material handling (MMH) activities found on YouTube. Results indicate that the fine-tuned model can recognize distinct MMH tasks in a robust and adaptive manner which is crucial for the widespread deployment of collaborative robots in construction. 

   more » « less
3. A multi-modal transformer network for action detection.

   https://doi.org/10.1016/j.patcog.2023.109713  

   Korban, M.; Youngs, P.; Acton, S (October 2023, Pattern recognition)

   Hancock, E. (Ed.)

   This paper proposes a multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a novel multi-modal attention mechanism that captures the correlations between different combinations of spa- tial and motion modalities. Exploring such correlations for actions effectively has not been explored before. We also suggest an algorithm to correct the motion distortion caused by camera movements. Such motion distortion severely reduces the expressive power of motion features represented by optical flow vectors. We also introduce a new instructional activity dataset that includes classroom videos from K-12 schools. We conduct comprehensive ex- periments to evaluate the performance of different approaches on our dataset. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet, and our instructional activity dataset. 

   more » « less
4. How do students’ achievement goals relate to learning from well-designed instructional videos and subsequent exam performance?

   https://doi.org/10.1016/j.cedpsych.2023.102162  

   Kuhlmann, Shelbi L.; Bernacki, Matthew L.; Greene, Jeffrey A.; Hogan, Kelly A.; Evans, Mara; Plumley, Robert; Gates, Kathleen; Panter, Abigail (April 2023, Contemporary Educational Psychology)

   Well-designed instructional videos are powerful tools for helping students learn and prompting students to use generative strategies while learning from videos further bolsters their effectiveness. However, little is known about how individual differences in motivational factors, such as achievement goals, relate to how students learn within multimedia environments that include instructional videos and generative strategies. Therefore, in this study, we explored how achievement goals predicted undergraduate students’ behaviors when learning with instructional videos that required students to answer practice questions between videos, as well as how those activities predicted subsequent unit exam performance one week later. Additionally, we tested the best measurement models for modeling achievement goals between traditional confirmatory factor analysis and bifactor confirmatory factor analysis. The bifactor model fit our data best and was used for all subsequent analyses. Results indicated that stronger mastery goal endorsement predicted performance on the practice questions in the multimedia learning environment, which in turn positively predicted unit exam performance. In addition, students’ time spent watching videos positively predicted practice question performance. Taken together, this research emphasizes the availing role of adaptive motivations, like mastery goals, in learning from instructional videos that prompt the use of generative learning strategies. 

   more » « less
5. Instructional Activity Recognition Using A Transformer Network with Multi-Semantic Attention

   https://doi.org/10.1109/SSIAI59505.2024.10508634  

   Korban, Matthew; Acton, Scott T; Youngs, Peter; Foster, Jonathan (March 2024, IEEE)

   Korban, Matthew; Acton, Scott T; Youngs, Peter; Foster, Jonathan (Ed.)

   Instructional activity recognition is an analytical tool for the observation of classroom education. One of the primary challenges in this domain is dealing with the intri- cate and heterogeneous interactions between teachers, students, and instructional objects. To address these complex dynamics, we present an innovative activity recognition pipeline designed explicitly for instructional videos, leveraging a multi-semantic attention mechanism. Our novel pipeline uses a transformer network that incorporates several types of instructional seman- tic attention, including teacher-to-students, students-to-students, teacher-to-object, and students-to-object relationships. This com- prehensive approach allows us to classify various interactive activity labels effectively. The effectiveness of our proposed algo- rithm is demonstrated through its evaluation on our annotated instructional activity dataset. 

   more » « less