Introduction

This paper investigates the effectiveness of end-to-end training for real-time weakly supervised video anomaly detection (wVAD). As opposed to the unsupervised VAD problem [15], in which only nominal videos are used in training, anomalous videos with video-level labels are also available for training in wVAD [16]. The lack of framelevel anomaly labels differentiates wVAD from supervised VAD. Recent wVAD approaches extract features from a video using large pre-trained video models and process them using elaborate deep neural networks [10], [17], [19], [6]. These feature aggregation networks are trained together with a shallow model using a deep metric learning loss, such as the multiple-instance learning (MIL) loss [16] to detect anomalous events. These models process a full video together during inference to refine the features across the entirety of the video. Hence, they focus on offline anomaly detection after observing the entire video with decision period typically being equal to the video length. Figure 1 shows the impact of reducing the decision period during inference on three recent state-of-the-art methods [17], [19], [6] on the UCF-Crime dataset [16]. Results indicate that the performance of state-of-the-art models is proportional to the number frames processed at a time, with significantly reduced performance for real-time or near-real-time performance as shown by the dashed rectangle in Figure 1 1 . In this paper, we show that a much improved real-time detection performance is possible (shown by star in Figure 1.) An important premise of video anomaly detection in many applications, including video surveillance, is realtime or near-real-time detection to enable timely response. This poses a question: Can wVAD models be trained with a short decision period to enable real-time inference?

In this paper, we show that the answer to this question is positive through end-to-end training of modern transformerbased models using a novel self-supervised learning network based on kNN (k-nearest-neighbors) distances and uniform frame sampling. We show that using a decision period of 6.4 seconds the proposed method, called RE-WARD, outperforms state-of-the-art methods which use decision periods 43 times ours.

End-to-end training is not feasible with the existing wVAD methods since the commonly used metric learning losses require uploading two batches of nominal and anomalous videos in a single GPU together. The batch sizes necessary for gradient estimates of sufficient quality cause a memory bottleneck problem. Hence, as summarized in Figure 2, the existing methods limit the training to the feature encoder for feature refinement and do not train the feature extractor, which is typically pre-trained on a large action recognition dataset, such as Kinetics-600.

In the proposed method, the end-to-end trained video model is directly used for inference, greatly simplifying the wVAD pipeline, as shown in Figure 2. Our contributions can be summarized as follows:

• We introduce a novel end-to-end solution, called RE-WARD, for wVAD systems.

• It enables real-time anomaly detection on test videos with 6.4 sec decision period.

• It outperforms state-of-the-art methods which use 273 sec decision period on popular wVAD datasets such as UCF-Crime and XD-Violence.

Related Work

Video Anomaly Detection: The motivation for wVAD is that in some applications it may be possible to roughly label videos as anomalous, without specifying where and when the anomaly happens, to obtain a representative set of anomalies of interest.

Sultani et al. [16] first introduced a deep MIL ranking loss framework to detect anomalous segments. MIST [10] used an encoder-based method that fine-tunes a feature encoder based on the generated pseudo-labels. RTFM [17] uses feature magnitude with the multi-scale temporal scenario from the video to select the top-k segments to determine the abnormality of a segment in a video. This paper also introduced the first feature aggregation MTN network. MTN has since been a staple backbone for current research. S3R [19] uses dictionary-based self-supervised learning to generate en-normal and de-normal features, which also uses the MTN network [17] to retrieve enhanced features to create pseudo-anomaly video features. MGFN [6] introduces a Glance-and-Focus module along with Magnitude Contrastive loss to increase the separability of normal and abnormal features.

There has been some works on real-time anomaly detection in an unsupervised setting [8,9]. However, in the wVAD setting, the focus has been on offline anomaly detection. Recent works share the popular pipeline which can be summarized as in Figure 2. We propose an end-to-end trained method that outperforms existing methods in realtime detection while providing a more computationally efficient solution to the wVAD problem.

Feature Extractors: Deep neural networks have made substantial progress in action recognition and feature extraction from videos. Two main categories of these models are 3D convolutional neural networks (3D-CNN) and more recent Transformer models. 3D-CNNs leverage 3D convolution to capture spatial and temporal information. Notable examples of 3D-CNNs include C3D [18] and I3D [5], which extract video features effectively. The former employs both RGB images and optical flow in two-stream networks, while the latter uses raw video frames directly, operating as a 3D network. wVAD literature has predominantly focused on utilizing I3D as a feature extractor.

Transformer-based video models, such as TimeSformer [2], Video Swin Transformer [13], ViViT [1], and Uniformer [12] have shown significant improvement in video understanding.

While TimeSformer is a completely attention-based model, Uniformer combines attention and convolution to further improve video understanding. Recently, we also see the use of transformer-based models in wVAD research, such as MGFN [6], which reports results using a Swin Transformer [13] model as a feature extractor.

Proposed Method

For end-to-end training large video models on wVAD data, we propose a self-supervised learning method called REWARD (Real-Time End-to-End Weakly Supervised Video Anomaly Relevance Detector). As illustrated in Figure 3, it consists of incremental training of a stronger classifier with the help of a weaker classifier in three steps. In the first step, a kNN classifier network is trained on features from a pretrained video model. In the second step, the predicted anomalous frame indexes from the first step are used to train a Multi-Layer Perceptron (MLP) classifier and in the final step, the raw video frames of anomalous frame indexes predicted by MLP are used to end-to-end train a large video model such as a Transformer. Before explaining the details of REWARD, we first discuss the background and motivation.

Motivation

End-to-End Training refers to the process of training all the neural network parts together to perform a task directly from the raw input to the final output, without any intermediate, separately trained feature refinement steps. This approach can bring about significant advantages. It can simplify the overall machine learning pipeline by eliminating the need for feature refinement. Moreover, it can lead to better real-time performance as the model can learn taskspecific representations directly from the raw input data without the need for feature aggregation. It has been applied successfully in various domains, including computer vision, natural language processing, and speech recognition [3,7].

Memory Bottleneck of Metric Learning Loss: Metric learning losses are a family of loss functions used in machine learning in cases where the output labels for training instances are not sufficient for the use of loss functions used in supervised learning, such as binary cross entropy and mean squared error [11]. These losses are specifically designed to learn a metric space, where distances between samples in the learned space correspond to their semantic dissimilarity. The aim of metric learning losses is to minimize the distance between samples of the same class and maximize the distance between samples of different classes, Figure 4. To train the feature encoder and classifier using a metric learning loss, Fn and Fa are processed together to promote the separability between normal and anomalous segments. Nevertheless, training a feature extractor end-to-end using a metric learning loss is not feasible due to two reasons: processing together large number of video clips in a single GPU entails huge memory requirements and feature extractor has orders of magnitude more parameters than feature encoder.

thereby improving the quality of features.

In recent works, MIL ranking loss, a metric learning loss, has played a key role in wVAD systems [16]. This loss function has a requirement that is impracticable for end-toend training of feature extractors in a wVAD setting (Figure 4). MIL loss requires an equal number of anomaly videos and nominal videos to be trained together at once.

In the existing works utilizing metric learning loss, while training only the feature encoder and classifier with multiple bags of anomalous and normal features together is feasible on a single processing unit, training the feature extractor end-to-end using a metric learning loss would require raw video clips to be processed together (Figure 4). This is infeasible because the memory requirement of processing a large number of raw video clips at once is inordinate. Moreover, the number of trainable parameters in feature extractor is orders of magnitude higher than that of feature encoder.

End-to-End Training via REWARD

We propose a solution to the memory bottleneck problem by converting the wVAD problem with video-level labels into a classification problem with frame-level pseudolabels. Specifically, we generate segment-level pseudolabels and replace metric learning loss functions with the more memory-friendly loss functions such as BCE (binary cross entropy).

To find the anomalous segments and perform end-to-end training, we propose the REWARD network with architec- ture shown in Figure 5. It consists five types of modules: kNN distance calculation, smoothing, selection, multi-layer perceptron (MLP), and a large video model for feature extraction and classification.

We first divide each video into T segments by uniformly sampling at a rate of 5 frames/sec and extract features using a pretrained feature extractor to get feature representations for nominal and anomalous videos as F n ∈ R T ×P and F a ∈ R T ×P respectively, where P is the dimension of the feature vector. All features from nominal and anomalous videos are collected into the sets

where N n and N a are the numbers of nominal and anomalous videos in the training set, and j denotes the anomalous video index.

Distance Calculation: The distance calculation module consists of four layers. In the distance layer, each segment F t aj , t = 1, . . . , T, in an anomalous video is compared to F n by computing the Euclidean distance δ ti between F t a and all

which integrates the temporal information in the anomalous video. Then, in the sorting layer, the N distances

in each row t of ∆ are sorted in the ascending order. Finally, an average pooling layer is applied to each row only by taking the average of the first k elements (i.e., distances to the nearest k neighbors), resulting in the T -dimensional kNN distance vector δ t . Each kNN distance δ t gives us an estimate of the similarity of each segment t within an anomaly video to the nominal segments. Small values of δ t mean a higher degree of similarity to normal action and vice-versa. Smoothing: Considering the temporal continuity of actions in video, a smoothing step is applied to avoid spurious dissimilarities. We first normalize the distance values around zero by subtracting the average value

and then apply the cumulative sum operation followed by the ReLU activation function in a recurrent way to compute the anomaly evidence for each segment:

Initial Selection: After smoothing, the anomaly evidence time series D t takes values around zero for segments similar to the nominal segments and positive values for segments not so similar to the nominal ones. Since it is known that some segment(s) in each anomalous video are anomalous, the ones with largest D t values are good candidates. We select a small percentage of segments by focusing on the segments with largest anomaly evidence D t . After normalizing D t by its largest value,

we pick the segments with normalized evidence greater than λ ∈ (0.5, 1) as an initial anomalous set:

where λ is a threshold close to 1 (e.g., 0.8 used in the experiments) and j denotes the anomalous video index. Combining the initial sets from all anomalous videos we form the set

where denotes the union of sets and M is the number of anomalous videos.

MLP and Final Selection: To generalize the selection of anomalous segments for the entire video, we train an MLP using the segments with strong anomaly evidence in the initial selection set I 1 as label 1 and nominal segments as label 0. Once the MLP is trained using BCE loss, all the segments in the anomalous videos are passed through the trained MLP to estimate their probability of being anomalous, p t , t = 1, . . . , T for each anomalous video. Then, a final smoothing and selection procedure is applied for each anomaly video. Using Eq. ( 1), p t is normalized to the zeromean pt . Then, its rectified cumulative sum R t is obtained using Eq. ( 2). Finally, the normalized version Rt of R t (Eq. ( 3)) is used to make the final selections. Since the anomaly scores in Rt are more reliable than the evidences Dt in the initial selection module thanks to the trained MLP classifier, we use a lower threshold here, the average value of Rt :

End-to-End Training: With the pseudo-labelled set I 2 of anomalous segments available, we can now turn the wVAD task into a binary classification problem which lends itself to end-to-end training of the feature extractor. By labeling the video segments V a (I 2 ) as 1 and the nominal video segments V n as 0, we train a feature extractor in an end-to-end fashion using the BCE loss. We use a single neuron with the logistic sigmoid activation function at the classification layer to compute the anomaly relevance probability.

Real-Time Inference: After training, the feature extractor with the single-neuron classifier is used in an online fashion with a small decision period (6.4 sec) for real-time inference (see Figure 2). The details of real-time inference and computational efficiency can be found in Sections 3.3 and 4.4.

Implementation Details

Since we do not use feature aggregation, instead of I3D, we used a more modern transformer-based feature extractor in the experiments, Uniformer-32, that can benefit from end-to-end training. In recent works, consecutive 16 frames are sampled, and the features are extracted using the I3D feature extractor. We also sample consecutive 32-frame segments for the state-of-the-art methods to obtain results with the new Uniformer features for a fair comparison. Then, during training, each video is condensed into 32 segments through linear interpolation. In testing, existing methods can produce an anomaly score for each individual 16-frame segment, but effective feature aggregation (e.g., MTN [6,17,19]) requires all or most segments to be passed together at once. The significant drop in the detection performance with smaller number of segments for inference (i.e., decision period) is shown in Figure 1.

To address this, we first uniformly sample each video at 5 frames/sec. Leveraging Uniformer-32, which require an input of 32 frames, a mere 6.4 seconds suffice to accumulate 32 frames, yielding a 6.4-second contextual scope. We set the number of neighbors as k = 20, and the initial selection threshold as λ = 0.80. When training the MLP, which consists of 2 hidden layers each with 1000 neurons, we set the learning rate as 5e-5 with a weight decay of 0.001. Adam optimizer is used for training all networks.

During the calculation of D t using kNN distances (Eq. ( 2)), we observed that a considerable number of anomaly training videos in the UCF-Crime dataset exhibit banners (Figure 6). Typically, these banners appear at the start or end of a video and are static images. In comparison to the nominal video data, banners are relatively uncommon, which leads to the feature representations F a for such segments being markedly distinct when compared to the nominal feature set F n . Consequently, this produces anomalously high D t scores. To mitigate this problem we drop the first and last 20% of total segments from all anomaly videos in the training set for UCF-Crime. Note that this is done during selfsupervision in training to identify an initial set of anomalous video frames, hence it does not violate fair comparison with other methods during testing.

For end-to-end training using Uniformer-32 [12], we start with the pretrained model on the Kinetics-600 dataset [4]. It was observed that utilizing all available layers in Uniformer-32 causes the model to overfit rapidly to the training data, owing to the relatively smaller data sizes of UCF-Crime and XD-Violence as compared to Kinetics-600. In view of this, all but the last block, which comprises 7 attention modules, are frozen in Uniformer-32. The learning rate of 5e-5 is opted for UCF-Crime, and 5e-6 for XD-Violence, with a weight decay of 0.005 for both datasets.

Experiments

Datasets

We evaluate our approach against the state-of-the-art methods using two datasets: UCF-Crime [16], and XD-Violence [20]. We adopt the datasets to fit the evaluation criteria by following the procedures outlined in previous studies [10], [17], [19], [6]. UCF-Crime [16] involves 1900 surveillance videos depicting 13 different types of real-world anomalous events such as abuse, robbery, arson, explosion, and road accidents. The training data consists of 1610 videos with 810 videos labelled as nominal and 800 videos labelled as anomalous. The dataset also includes 290 videos for testing with a mix of nominal and anomalous videos. The XD-Violence [20] dataset is more recent and larger than UCF-Crime. It consists of 4754 untrimmed videos with accompanying audio. The videos cover a diverse range of sources, including surveillance footage, movies, dash-cam recordings, and video games. The training set for the wVAD setting includes 3954 videos with over 1900 anomalous videos and over 2000 nominal videos. To fairly evaluate the performance of our model, we only utilized video information and discard audio information.

Metrics

Following the common practice in the literature, we use the Area Under ROC Curve (AUC) metric for evaluating our performance on the UCF-Crime dataset and the Average Precision (AP) metric, which is the area under the precisionrecall curve, on the XD-Violence dataset. By putting more emphasis on number of anomalies, AP presents a more suitable metric for the inherently class-imbalanced anomaly detection problems than AUC. Hence, it has been preferred over AUC in the literature for the more recent XD-Violence dataset. The results are reported by averaging over four trials. We compute these metrics with window sizes (i.e., number of segments used for inference) ranging from 4 to 256 for three state-of-the-art methods, RTFM [17], S3R [19], MGFN [6], to evaluate the change in their detection performance when they are made to perform in real-time. We also present computational efficiency results for our algorithm in terms of frames per second (fps) to evaluate its real-time inference capability. Method feature AUC % Decision Period (sec) S3R [19] I3D 81.34 8.533 S3R [19] Uniformer-32 82.16 8.533 RTFM [17] I3D 80.63 8.533 RTFM [17] Uniformer-32 81.22 8.533 MGFN [6] I3D 81.76 8.533 MGFN [6] Uniformer-32 80.99 8.533 REWARD Uniformer-32 86.94 6.4 Table 1. Real-time detection performance of the proposed method is 4.78% higher than the state-of-the-art methods on UCF-Crime.

Results

Figures 7 and 8 present the performances of S3R, RTFM, and MGFN as a function of decision period using I3D and Uniformer-32 features on the UCF-Crime and XD-violence datasets, respectively. Tables 1 and 2 summarize the realtime detection performance of state-of-the-art methods together with our method on both datasets. As seen in the re- sults, the proposed approach outperforms the state-of-theart methods by a wide margin, especially on UCF-Crime, where REWARD with 6.4 sec decision period improves upon S3R with a decision period of 273 sec by more than 1% AUC. Note that for a fair comparison we evaluated the state-of-the-art methods with both I3D and Uniformer feature extractors. XD-Violence imposes some challenges the action recognition models. As in Figure 9, XD-Violence consists of dynamic camera views with constant scene changes, as opposed to the static camera views in UCF-Crime. Also, the heterogeneity in video sources is much larger in XD-Violence (movies, video games, dash-cam recordings, etc.) than UCF-Crime (only surveillance videos). It should be noted that, the performance of RE-WARD is highly dependent on the used feature extractor since no feature refinement step is involved in the current version. This aspect will be further analyzed in Section 4.5.

Computational efficiency

By eliminating the need for feature encoding neural networks, the proposed REWARD method significantly reduces the number of parameters and processing time during inference, leading to better real-time performance compared to the existing methods. Table 3 presents the inference computation rate of 63.3 fps for our method based on Uniformer-32 using a Nvidia RTX-2070 GPU. With 63.3 fps our method is able to process the 32 frames within 0.5 sec, resulting in a total decision delay of 6.9 sec. The kNN computations during training took around 5 hours for Uniformer-32 on an Intel Core i7 8700K CPU for the UCF-Crime dataset.

Ablation Study

In this section, we discuss the impact of several factors on the performance of REWARD.

End-to-End Training: To understand the contribution of end-to-end (E2E) training, we compare the wVAD performance of REWARD-E2E with a non-E2E version. Specifically, using the pre-trained features from Uniformer-32 we trained an MLP with the same configuration as the one used in self-supervision using the same labels provided by the self-supervision mechanism, i.e., V a (I 2 ) and V n in Figure 5. This transfer learning (TL) approach based on REWARD's self-supervision is called REWARD-TL. On both datasets, end-to-end training improves the performance by a wide margin, 1.64% on UCF-Crime and 2.46% on XD-Violence. Offline Performance: Since the existing methods focus on offline detection, we additionally assess the offline anomaly detection performance of our approach by adopting a segmented video analysis strategy. In contrast to a temporal sampling rate of 5 frames per second, we partition both the training and testing videos into 32 segments. Each segment consists of 32 frames, resulting in a consistent total of 1024 frames across videos of varying lengths. The experimental process is reiterated while adhering to the identical implementation details outlined in Section 3.3, with the additional implementation of Savitzky-Golay filter [14] on predictions with a window size of four and a polynomial order of one.

Table 5 and Table 6 demonstrate REWARD's offline detection performance on UCF-Crime and XD-Violence datasets, respectively. For long videos, partitioning the video into 32 segments may result in long segment duration (i.e., decision period), which may be prohibitive for online (real-time) detection. Note that longer segments with more contextual information improved RE-WARD's performance by 0.54% on UCF-Crime and by 2.59% on XD-Violence compared to the online version with a fixed 6.4 sec decision period (cf. Tables 1 and 2).

Method

Feature Year AUC(%) Sultani et al. [ Impact of feature aggregation: Feature aggregation step (e.g., MTN) in existing methods is mainly what prevents high real-time detection performance. Hence, we investigate the impact of feature aggregation on the wVAD performance of existing methods in relation to Uniformer-32 and I3D. Table 7 underscores the substantial deterioration in wVAD performance of existing methods when the critical feature aggregation step is omitted, irrespective of the employed model -I3D or Uniformer-32. Notably, the results suggests a more effective synergy between feature aggregation and I3D.

Conclusion

We propose an end-to-end training approach, RE-WARD (Real-Time End-to-End Weakly Supervised Video Anomaly Relevance Detector), for real-time video anomaly detection in the weakly supervised setting. REWARD trains a large video model for the wVAD task, as opposed to

Method

Feature Year AP(%) Wu et al. [20] I3D 2020 75.41 RTFM [17] I3D 2021 77.81 S3R [19] I3D 2022 80.26 MGFN [6] I3D 2022 80.1 REWARD-E2E Uniformer-32 2023 80.30 Table 7. Impact of feature aggregation on wVAD performance.

the existing methods based on refining the feature extractors pretrained on action recognition datasets. The novel technique enabling end-to-end training is a self-supervision method based on kNN distance calculations. The experimental results demonstrated that in a delay sensitive setting where real-time decision is important, the proposed endto-end solution with a decision periof of 6.4 sec outperforms the state-of-the-art methods with a similar decision period of 8.5 sec by 4.78% and 4.54% AUC on the commonly used UCF-Crime and XD-Violence datasets. Moreover, REWARD provides a more computationally efficient pipeline for real-time inference by eliminating the popular feature aggregation/refinement step in the literature.