Evaluation of classroom activities is essential for instructors to revise and improve their teaching skills [1]. As a result, accurate feedback from classroom evaluation drastically affects students' classroom engagement and enhances the quality of education [2]. Traditionally, providing such feedback requires considerable labor and manual rating from trained experts, which is expensive and time-consuming. Using deep learning models is an efficient and effective solution to this issue as they can automate the evaluation of teacher activities, reducing manual labor work and errors caused by humans [3]. Therefore, this paper compares several deep network architectures based on their ability to detect instructional activities. Several strategies have also been presented to enhance the effectiveness of deep models in detecting instructional activities. The new strategies include: (1) an adaptive sampling algorithm for selecting critical frames in classroom videos; (2) a new loss function incorporating frame-level and sequence-level prediction; (3) a post-processing algorithm for detecting the start and end frames of long actions; (4) a motion enhancement algorithm to make the motion features insensitive to camera movements. The first three strategies address the issue of varying lengths of instructional activities, a significant challenge in detecting instructional activities. The fourth strategy makes the pipeline more reliable under camera movements during capturing instructional videos.
Earlier methods used hand-crafted features such as dense trajectory features [4] and bag-of-words (BOW) histogram of motion tubelests [5] combined with traditional classification algorithms including fisher kernels [4] and Support Vector Machine (SVM) [5] to detect actions in untrimmed videos. With new advances in deep learning, detecting action has become more effective. [6] p suggested an algorithm to localize actions based on the maximum sum of frame-wise classification scores in different temporal segments that are processed through a deep Convolutional Neural Network (CNN). [7] improved it by adding a recurrent mechanism that can better than CNN model the temporal dependencies in action frames. [8] proposed a more effective approach than [6], [7] using a long-short term transformer that can process longer videos without any bias against older temporal inputs.
There have been several approaches to detect activities in classroom videos. [9] suggested using hand-crafted features including elbow angles and movements in the face and hands combined with Primitive-based Coupled Hidden Markov Model (PCHMM) to recognize seven teacher activities. [1] presented a more effective deep model with a multimodal attention layer to capture long-term semantic dependencies in instructional videos. [10] suggested that skeleton pose is a more effective modality than RGB images used by others, as the skeleton pose better represents teacher and students' actions in classrooms.
Fig. 1 shows the overall pipeline of our instructional action detection. Given a sequence of RGB frames, I R = {I r t 2 R H⇥W ⇥3
, t = 0, 1, ..., T }, the goal is to find the action class scores, P , and the start of the end of action, and ✏, respectively. Here, T is the size of the action sequence; and H and W are the height and width of the image, respectively. We first select the keyframes
, t = 0, 1, ..., T 0 }, which include important frames in the sequence. Here, T 0 is the number of keyframes. Next, the optical flow fields,
, t = 0, 1, ..., T 0 }, are extracted using a state-of-the-art optical flow estimation algorithm [11]. Using the motion enhancement algorithm, the optical flow fields are enhanced, making them insensitive to camera movements. The enhanced optical flow fields then are converted to optical flow images
, t = 0, 1, ..., T 0 } using a color-coding technique [12]. I K and I K 0 are then converted to RGB features, I F = {I f t 2 R m , t = 0, 1, ..., T 0 } and motion features
, t = 0, 1, ..., T 0 }, using a pretrained I3D network [13], a widely used model for action recognition. Here, m is the size of features. The enhanced baseline model with a revised loss function process I F and I F 0 produces the action class prediction scores. A post-processing algorithm is also utilized to generate the start and end of action instances. .The pipeline of the presented instructional activity detection algorithm.
The instructional videos are long and often the important events occur sparsely. So, a keyframe selection algorithm is presented the choose the important frames, making the proposed pipeline more efficient and effective in handling long videos. The keyframes are selected when there is a desirable change in videos that are evaluated by measuring the difference between two consecutive frames as follows:
1
where T r is a threshold value.
Camera movement is inevitable in videos captured from classrooms. Such camera movements distort the optical flow fields and reduce the quality of motion features. To solve this we utilize a motion enhancement algorithm. To do such, first Gaussian mixture models (GMMs) are used to model the background motion as P ( ) =
and K are mean. covariance, mixing coefficient, and the number of distributions, respectively. The background is modeled by optimizing the GMMs parameters using maximum likelihood estimation [14]. With the assumption that the background is only affected by the camera movements, the camera-insensitive foreground then is recovered by subtracting from the corresponding background parts with respect to the Gaussian models.
Three baselines are selected in this paper, including the background suppression network [15], multi-label action dependencies [16], and long short-term transformer [8]. All of these networks performed exceptionally well on the THU-MOUS [17] and ActivityNet [18] datasets. Our instructional activity datasets share several characteristics with these two datasets, including (1) videos that are long in length; (2) videos are continuous streams, which means they are not segmented;
(3) several instances of class labeling can occur simultaneously with co-occurring labels in the annotation data. The background suppression network produces weighted scores for background and foreground frames. Such a weighting strategy handles the crowded scenes in classroom videos that may include a significant amount of irrelevant information. The multi-label action dependencies model captures the multi-class dependencies between different action classes. This is useful in our experimental setup since multiple instructional activities may co-occur. The long short-term transformer can capture short and long-term dependencies in action videos. This will help to include the critical temporal dependencies regardless of their temporal distances. This is important in classroom activity detection, where the temporal locations of critical frames may vary based on the size of video sequences.
The instructional videos include both long and short action instances, such as "teacher sitting" and "student raising hand", respectively. To accommodate the videos with varying lengths, we suggest adding a new loss function to enhance the baseline models:
where y, ŷ, Y , and Ŷ are the ground truth per frame, predicted values per action frame, ground truth per action sequence, and predicted values per action sequence, respectively. Moreover, c, C, ↵, and are predicted class, the number of classes, and the loss adjustment parameters for frame-level, and loss adjustment parameters for sequence-level, respectively.
Many instructional activities such as "teaching sitting" are significantly longer than standard actions. So, they cannot be entirely processed within the standard deep learning models to regress the start and end frames as these deep models have limited temporal receive fields. So, we propose a postprocessing algorithm to find the start and end frames of action instances after the frame-level prediction stage.
Given the action detection scores Ŷ 2 R T ⇥C , the goal is to find the start and end frames of action instances,
is the number of class instances. Our post-processing algorithm consists of two phases of action scores thresholding and the start and end frames detection as shown in Algorithm 1. We used our post-processing algorithm to visualize the start and end of actions for teachers based on a developed teacher dashboard.
The size of RGB and motion features in our experiments for all baselines is 1024. For the background suppression model, six convolutional layers are used. The learning rate is 1e 5 which is decayed by 0.1, for every 1500 iterations. The multi-label dependencies network includes five layers. The temporal length is 132, and the initial learning rate is 1e 4 . The long-short term transformer consists of four layers and 16 heads. Moreover, the learning rate is increased from zero to 5e 5 for half of the training iterations that took 50 epochs. All the experiments are conducted using PyTorch 1.7 on a server PC with dual Nvidia RTX 3090 GPUs (24GB VRAM), AMD Ryzen Threadripper 3990X 64-Core Processor, and 256GB of RAM.
An elementary school dataset was collected to analyze instructional activities. 250 hours of instructional activity videos were annotated by a team of nine professional annotators. Fig. 2 shows the labels for our 24 instructional activity classes. In our experiments, 80% of the data is used for training and 20% for testing. In our experiments, we used the F1 score metric based on frame-level prediction as
where T P , F P , and F N are true positive, false positive, and false negative predicted frames, respectively.
Fig. 3 shows the comparative results of our pipeline based on three baseline models. The average per class performance for the background impression, multi-label dependencies, and long-short term transformer models are 0.47, 0.49, and 0.4, respectively. The average per sample performance for the background impression, multi-label dependencies, and long-short term transformer models are 0.52, 0.57, and 0.47, respectively if Ŷt,c ✓ then . ✓ is the detection threshold 6:
Ŷt,c = 1
else if Ŷt,c < ✓ then 8:
Ŷt,c = 0
9: end if 10: c c + 1 11: end while 12: t t + 1 13: end while Phase 2 -Start and end frames detection
Fig. 4 illustrates the impact of our proposed strategies, Keyframe Selection (KS), Motion Enhancement (MH), and Enhanced Loss Function (EL) on the total performances of two baseline models. Using our proposed strategies, the average performance for background suppression model has a significant improvement of 0.2, from 0.37 to 0.57. Similarly, for the long-short term transformer, the performance has been increased from 0.52 to 0.65.
This paper proposes several strategies to improve the performance of multiple state-of-the-art action detection networks on instructional activity videos. The presented strategies mainly focus on improving the network in dealing with varying activity sequences and camera movements. Such a enhanced deep learning framework will facilitate teachers to receive feedback more effectively and efficiently than using manual labor. The experimental results have been promising when the enhanced deep models are evaluated on 250 hours of our annotated instructional videos.