Fueled by the growing availability of video-and-text data [2,8,9,24,41,43,48] and advances in the Transformer model design [12,54], the last few years have witnessed incredible progress in video-and-language (VidL) understanding [26,31,40,64,75,80]. Since the initial transformerbased models for VidL, such as ClipBERT [26], the text-tovideo retrieval accuracy has improved from 22.0%, 22.4%, and 21.3% on MSR-VTT [65], DiDeMo [1], and Activi-Figure 1. We present a recipe for effective video-language pretraining. Our recipe starts with image and text transformer encoders trained on video-text pairs using a contrastive objective (VTC). We then progressively add more components to our framework while also studying the importance of each component along the way. Our final recipe includes the steps for (1) adding temporal attention, (2) injecting a multimodal fusion encoder, (3) incorporating masked modeling pretraining objectives, (4) jointly training on images and videos, (5) using more frames during fine-tuning and inference, and lastly, (6) scaling up the data and the model. tyNet [23] to > 45% R@1 accuracy on all three of these datasets, thus, marking an extraordinary relative improvement of more than 100% in less than 2 years.
At the same time, the model architectures and pretraining/finetuning protocols used by modern VidL approaches
Method Model Design Pretraining Data #Frames Temporal Modeling Multimodal Fusion Pretraining Objectives Dataset Size Modality PT FT Eval
UniVL [39] Joint Att. [5] 2-layer TR VTC+VTM+MLM+MFM+LM HT 136M V 48 48 48 VideoCLIP [64] 1D-Conv+TR ✗ VTC HT 136M V 32 32 32 ClipBert [26] Mean Pooling BERT MLM+VTM COCO+VG 0.2M I 1 16 16 Frozen [2] Temp. Attn [5] ✗ ITC C5M 5M I+V 1 → 4 4 4 MERLOT [75] Joint Attn RoBERTa VTC+MLM+FOM YT 180M V 16 16 16 VIOLET [16] Window Attn [37] BERT VTC+VTM+MLM+MVM YT+C5M 185M I+V 4 5 5 MV-GPT [47] Joint Attn 2-layer TR MLM+LM HT 136M V ---ALL-in-one [55] Token Rolling [55] ViT VTC+VTM+MLM HT+W2 172M V 3 3 9 Singularity [25] Late Temp. Attn 3-layer TR VTC+VTM+MLM C17M 17M I+V 1 → 4 4 12 LAVENDER [32] Window Attn [37] BERT MLM C17M+IN 30M I+V 4 5 5 OmniVL [57] Temp. Attn 2×BERT VTC+VTM+LM C17M 17M I+V 1 → 8 8 8 ATP [6] ✗ ✗ VTC CLIP 400M I 1 16 16 CLIP4Clip [40] Late TR ✗ VTC CLIP 400M I 1 12 12 ECLIPSE [34] Late TR ✗ VTC CLIP 400M I+A 1 32 32 CLIP2TV [18] CLIP 4-layer TR VTC+VTM CLIP 400M I 1 12 12 CLIP-Hitchhiker [3] Late Attn ✗ VTC CLIP 400M I 1 16 120 CLIP-ViP [66] Prompt Attn [66] ✗ VTC CLIP 500M I+V 1 → 12 12 12 TR: Transformer; Late: Late fusion; Attn: Attention. V: Video; I: Image; A: Audio; 1 → 4: 1 frame for stage-1 training and 4 frames for stage-2.
VTC: Video-text contrastive; VTM: Video-text matching; MLM: Masked language modeling; MFM: Masked frame modeling; LM: Language modeling. HT: HowTo100M [41]; C5M, C17M: see supplementary; YT: YT-Temporal [75]; W2: WebVid-2M [2]; COCO: [33], VG: Visual Genome [24]; IN: An internal dataset.
Table 1. An overview of the existing VidL methods. Significant differences exist among these methods, making it challenging to reproduce, analyze and compare these methods. This motivates us to answer the question "What are the key steps to build a highly performant VidL framework" by investigating various components in the VidL framework design.
have become significantly more complex and specialized over the last several years. As a result, it is increasingly difficult to reproduce, analyze and compare most recent VidL frameworks. For example, several recent approaches [25,32,66] propose new architectures, new initialization strategies, pretraining objectives, pretraining datasets, and optimization protocols. Due to the large computational cost of ablating all these factors, it is difficult to understand which components are critical to the success of the proposed frameworks. Similarly, the key success factors of many other recent VidL approaches [6,16,32,57] are also often obfuscated, which hinders future research.
In Table 1, we illustrate the complexity of modern VidL frameworks by dissecting them along multiple dimensions, including temporal modeling schemes, multimodal fusion modules, pretraining objectives, the source of the pretraining data, and the number of frames for pretraining, finetuning and inference. Based on this analysis, we observe that there exist significant differences among these VidL methods. Unfortunately, it's not clear which differences are important for the overall VidL performance and which are not.
The recent METER [13] work studies a subset of these components in the context of image-language modeling. However, their analysis is limited to images and, thus, ignores various aspects related to video modeling, such as spatiotemporal architecture design, video pretraining ob-jectives, video pretraining data, and video-specific finetuning/evaluation protocols such as the number of frames. As we will show in our experimental section, many of the findings presented in the image-based studies [13] do not hold for video. Beyond image-based analysis, we note that the concurrent work in [17] conducts an empirical study of VidL transformers. However, unlike our work, which covers a broad range of VidL design factors, their analysis is focused predominantly on masked visual modeling objectives, which we also study in this work.
Our main objective in this work is to answer the question "What are the key steps needed to build a highly performant VidL framework?" To do this, we conduct a thorough empirical study that demystifies the importance of various VidL design choices and ultimately leads to a VidL framework that achieves state-of-the-art results on various VidL benchmarks. Using our empirical insights, we then develop a step-by-step recipe for effective VidL pretraining. Our recipe, dubbed VINDLU (VIdeo aND Language Understanding), starts from a standard Vision Transformer (ViT) [12] and uses a simple progressive expansion scheme where at each step, we investigate a particular aspect of VidL framework design (e.g., architecture, pretraining objective, pretraining data, etc.), and choose the best performing option. In particular, we study the following VidL design components: (i) the spatiotemporal architecture design,
70.4 71.2 73.2 70.2 Image Transformer Temporal Modeling Mean Pooling Late Temp. Att. Temp. Att. Temp. Conv 50.4 Multimodal Fusion Text-to-Video Bidirectional Video-to-Text 49.8 50.2 54.6 56.7 55.4 60.3 60.3 62.1 MVM MLM 66.5 67.5 67.5 1 68.5 70.2 70.2 69.6 69.5 1 4 frames 1 4 8 frames 4 8 MVM + MLM 64.8 30 50 70 72.4 25M 17M 73.6 + 6.3% + 3.6% + 7.2% + 2.2% + 1.2% Pretraining Objectives Pretraining Data # Pretraining Frames Image Video Image + Video Multi-stage Pretraining + 2.7% Scaling Up Data (ViTbase + BERTbase ) Averaged Acc (%) of R{1,5,10} on MSR-VTT, DiDeMo, ActivityNet-Captions Scaling Up Model (5M Corpus) BERTlarge ViTlarge + 1.0% + 3.0% 4 8 frames
Figure 2. We progressively expand an image transformer baseline (e.g., ViT) to a performant video-and-language (VidL) model. We do so by investigating the importance of many VidL design choices such as (i) temporal modeling, (ii) multimodal fusion modules, (iii) pretraining objectives, (iv) the source of the pretraining data, (v) the number of pre-training frames, (vi) multi-stage pretraining, and (vii) scaling of the data and model. Each bar depicts an average text-to-video retrieval Recall@1,5,10 accuracy across MSR-VTT [65], DiDeMo [65], ActivityNet [23]. The red bars denote the best-performing design choice in each subgroup. Our final VidL framework, dubbed VINDLU, outperforms our initial image Transformer baseline by 23.2%. The figure was inspired by [36].
(ii) the multimodal fusion schemes, (iii) the pretraining objectives, (iv) the source of the pretraining data, (v) finetuning/inference protocols, and (vi) scaling of the data and model. We present our recipe in Fig. 1.
The key findings of our empirical study include: • Contrary to the conclusions of several prior works [6,25] that a single frame is sufficient for VidL modeling, we discover that temporal modeling using multiple frames leads to a significant improvement over the spatial-only baselines (+6% averaged video retrieval accuracy on MSR-VTT, DiDeMo, and ActivityNet). • Multimodal fusion module incorporating video features into text is critical for good VidL performance (+3.6%). Conversely, adding text features to the video representation is not useful.
• Masked language modeling objective significantly improves performance (+6.2%) while masked video modeling objective brings an additional +1% improvement. • Pretraining jointly on images and videos is beneficial (+2.7%). Also, contrary to prior methods [2,57], we find multi-stage training unnecessary. • Pretraining with a small number of frames (e.g., 4) is sufficient and it can significantly reduce the computational cost of large-scale pretraining. Pretraining with more frames does not lead to a substantial performance boost. • Compared to many recent CLIP-based [45] VidL approaches [3,40,66], our recipe achieves comparable or even better performance with 20× less pretraining data. Our final model, trained using our VINDLU recipe, achieves state-of-the-art results on several VidL benchmarks. Specifically, on the video retrieval task, our method achieves 46.5%, 61.2%, 55.0% R@1 accuracy on MSR-VTT, DiDeMo, and ActivityNet outperforming the state-of-the-art by 7.8% and 6.1% on the latter two datasets. Also, our approach obtains state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC and TVQA, where we achieve top-1 accuracy of 44.7%, 44.6%, 97.1%, and 79.0% respectively.
We want to make it clear that, in this paper, we do not claim technical novelty behind any of the individual design choices (i.e., different subsets of these design choices were already used by prior VidL methods as shown in Table 1). Instead, our main contribution, which we believe might be equally if not more important than proposing yet another specialized or obfuscated VidL model, is to investigate these components collectively and validate their importance. We also do not claim superiority over previous methods (despite better results). Due to the implementation complexities of each method, fair and complete comparisons are difficult and not our intent. Instead, we hope that our recipe for building an effective VidL framework will provide useful insights for future research on VidL understanding. To enable the VidL community to build on our work, we release our code and pretrained models.
Image-and-Language Pretraining. Recent years have witnessed remarkable progress in image-and-language pretraining [7, 10, 20, 22, 29, 38, 45, 49, 51, 59-61, 69, 70, 74, 76-79]. However, most modern methods such as ViL-BERT [38], UNITER [10], CoCa [71], LEMON [20], BEiT-3 [61] employ complex transformer architectures and pretraining objectives. Thus, it is difficult to decipher which components are critical for good performance. A recent empirical study on image-language modeling METER [13] studies a variety of components. However, since their analysis is done exclusively on images, it is unclear whether these findings generalize to video. In comparison, our work thoroughly investigates various video-specific design choices for effective video-language pretraining.
Video-and-Language Pretraining. In recent years, the large-scale VidL pretraining [6,16,26,32,57,58] has shown strong transfer learning ability to downstream VidL tasks such as text-to-video retrieval [1,23,26,35,40,65,72], video question answering [63,72,73], video captioning [21,23,50,56,81], etc. Several methods [3,18,40,66] achieve impressive results by building on the popular image-language pretrained model CLIP [45]. Additionally, several recent approaches [25,32,57] propose more sophisticated VidL frameworks to achieve comparable performance as CLIPbased methods without large-scale CLIP pretraining. However, with the impressive results, these methods also require more complex architectures and specialized video pretraining protocols (as shown in Table 1). The complexity of these frameworks and the large computational cost of VidL pretraining makes it challenging to decipher which VidL framework components are truly needed for good performance. Moreover, unlike in the image-language domain, there are few empirical studies investigating various VidL design components collectively. For instance, the concurrent work of Fu [17] only studies masked video modeling pretraining objectives and is based on a slightly older VIO-LET [16] method. Furthermore, the recent works [6,25] focus predominantly on spatial biases in modern VidL benchmarks. In contrast to these approaches, our work investigates the importance of various factors in VidL framework design. We then use our empirical insights to provide a detailed step-by-step recipe for effective VidL pretraining.
In this section, we describe our recipe for video-andlanguage (VidL) pretraining. We begin with a standard image transformer (e.g., ViT [12]) and progressively expand it to a model that achieves state-of-the-art results on various VidL datasets and tasks. At each step of our recipe, we study how various design choices affect VidL performance. In particular, we are interested in answering the following questions about the VidL pretraining design:
• Does a VidL model need temporal modeling, especially since most VidL benchmarks are spatially biased [6,25]?
If so, what is the best temporal modeling scheme? • What is the most effective way to do multimodal fusion? Some approaches [16,32,55] use bidirectional while others [25,57] employ unidirectional multimodal fusion modules. Which of these schemes works the best? • Which pretraining objectives are most useful for VidL representation learning? Prior methods use video-text contrastive (VTC) [28], video-text matching (VTM) [28,31,39], masked-language-modeling (MLM) [11], or masked-video-modeling (MVM) [52]. Are all of these objectives needed for the best performance? • What pretraining data is most useful for training VidL models (e.g., video-only or images and videos)? Is it necessary to use curriculum learning [2,55,57] or is singlestage pretraining sufficient? • How many frames are needed for pretraining, fine-tuning, and inference? Several approaches [6,25] claimed that single frame pretraining is sufficient while others [57,66] pretrained their models with 8 or even more frames. Should we finetune the pretrained VidL models using the same number of frames as during pretraining or is it helpful to use more frames during fine-tuning and inference?
Motivated by these questions, we next present our recipe while also studying these questions in more detail.
Step 0: Starting Ingredients Image Transformer Baseline. We start with a standard ViT-B/16 [12] transformer trained on single frames of WebVid-2M [2]. We use BERT [11] as our text encoder for all experiments. Formally, given the paired video and text input (v, t), The image transformer randomly selects a single frame from the video as input to extract the video embeddings. A text encoder encodes the text t to extract the text embeddings. We then use a video-text contrastive (VTC) loss to maximize the agreement between the paired video and text embeddings as in [2,45]. Following [25], we use BEiT [4] initialization for our image transformer, whereas the text encoder is initialized with BERT base .
Experimental Setup. As our initial pretraining data, we use WebVid-2M [2] unless noted otherwise. We then finetune and evaluate our pretrained model on the three popular text-to-video retrieval datasets: MSR-VTT [65], DiDeMo [1], and ActivityNet-Captions [23], which include short and long videos. We report the averaged top-1, top-5, and top-10 text-to-video retrieval accuracies across these datasets as our evaluation metric. As shown in Fig. 2, our Image Transformer baseline achieves an average accuracy of 50.4%.
Over the next several subsections, we progressively expand this baseline by adding more components of increasing complexity. In particular, we start by incorporating (i) temporal modeling blocks, (ii) a multimodal fusion encoder, and (iii) additional pretraining objectives. Afterward, we investigate the choice for the (iv) pretraining data, (v) finetuning and inference protocols, and (vi) dataset and model scaling schemes. We would like to note that due to the large computational cost, we cannot ablate the order of the steps in our recipe. Thus, the order of the steps is primarily determined by the computational cost (i.e., the steps that can be implemented most efficiently are studied first then, moving to the more computationally costly steps).
In the first step of our recipe, we extend our initial image transformer to video via a temporal modeling mechanism, which enables training our model on multiple frames. We experiment with several temporal modeling schemes: • Mean Pooling (MP). In this variant, the visual encoder processes input frames independently and averages their frame-wise scores for the video-level score as in [40]. • Late Temporal Attention (L-TA). Following [25,40,42] we use a late temporal modeling scheme by attaching 2 Transformer layers to an image encoder, which then aggregates temporal information across all input frames. • Temporal Convolution (TC). Many prior methods [14,44,62] used 3D convolutions for temporal modeling. To validate its effectiveness, we inject 3D convolution [53] before the spatial attention to each Transformer Layer. • Temporal Attention (TA). Inspired by TimeSformer [5],
we experiment with divided space-time attention, which we insert before spatial attention as in [5]. As shown in the upper part of Fig. 2 and the Table below, the temporal modeling capability is critical for good VidL performance. This is indicated by a +6.3% accuracy boost of our temporal attention variant (TA) over the spatial-only baseline. We also observe that late temporal modeling (L-TA) has nearly no effect. We conjecture that this is due to the limited temporal modeling capacity (i.e., only two layers) and the lack of temporal fusion in the early layers. Lastly, our results suggest that TA outperforms TC by 2.1%, which might indicate that long-range temporal attention is more useful than local 3D convolutions.
Mean Pooling L-TA TC TA acc.(%) 49.8 50.2 54.6 56.7
Interestingly, we note that our findings contradict the conclusions of several recent methods [6,25], claiming that temporal modeling is not needed for many VidL tasks. We hypothesize that even on the spatially-biased datasets, temporal modeling is useful for resolving spatial ambiguities caused by appearance variations across different frames.
Takeaway #1: We adopt Temporal Attention (TA) as our temporal modeling mechanism and pretrain our model with 4-frame inputs unless otherwise noted.
Building on the model from Step 1, we next analyze the role of multimodal fusion modules. The multimodal fusion encoder aims to fuse multimodal cues from video and language for a more discriminative VidL feature representation. As shown in Fig. 3, we experiment with several variants of multi-modal fusion encoders: • Video-to-Text Multimodal Fusion (V2T-MF). As illustrated in Fig. 3a, V2T-MF injects relevant video cues into the textual features using Cross-Attention. For a fair V2T-MF VTM loss VTC loss Cross Attention Video Encoder Text Encoder T2V-MF VTM loss VTC loss Cross Attention (a) V2T-MF (b) T2V-MF Figure 3. An illustration of (a) video-to-text (V2T-MF), and (b) text-to-video (T2V-MF) multimodal fusion schemes. The videotext matching (VTM) loss is attached to the multimodal fusion encoder, whereas video-text contrastive (VTC) loss is added to the video and text encoders.
comparison with previous baselines [2,25], we do not add any extra layers but instead re-purpose the last m layer of our text encoder for V2T fusion. Specifically, a crossattention operation is inserted into each of the m last layers in the text encoder between Self-Attention and MLP. This scheme was also previously used by [25,57]. • Text-to-Video Multimodal Fusion (T2V-MF). Similar to V2T-MF, we build T2V-MF (Fig. 3b) by re-purposing the last m layers of the vision encoder and using crossattention to incorporate text cues into the video features.
proaches [16,32,55,75] feed the concatenated visual and textual features to a m-layer Transformer. However, this is often computationally infeasible in the video domain due to many input frames. Instead, we implement B-MF by combining T2V-MF and V2T-MF. To train each variant, we add the video-text matching (VTM) loss (see Sec. 3) as in [16,55,57]. In the table below and Figure 2, we report that the V2T-MF scheme performs the best (i.e., +3.6% improvement). Surprisingly, the reverse T2V-MF scheme substantially decreases performance (-1.3%). We conjecture that predicting the matching videotext pairs using a pretrained language rather than a visual representation is easier. We also note that the B-MF scheme yields no improvement compared to V2T-MF. Takeaway #2: For our remaining experiments, we use V2T-MF as our multimodal fusion encoder.
Step 3: Pretraining Objectives Building on Step 2, we next study the following pretraining objectives: • Visual-Text Contrastive Learning (VTC). VTC aims to learn independent representations for video and text by maximizing the agreement between positive (visual, text) pairs while minimizing the agreement between negative pairs. Note that this objective is already used in previous steps, and thus, not included in Figure 2.
• Visual-Text Matching (VTM). VTM objective is implemented as a standard cross-entropy loss that encourages a VidL model to produce binary predictions indicating whether a given video-text pair matches. Following [30], we attach this loss to our multimodal fusion encoder and use hard negative mining during training as in [25]. The VTM objective is already used in Step 2 (i.e., the multimodal fusion step) and thus, not included in Figure 2. • Masked Language Modeling (MLM). MLM objective aims to predict the masked words by leveraging information from both visual and textual features. We mask 50% text tokens using the same masking strategy as in BERT [80] and attach a linear layer to our multimodal fusion encoder (T2V-MF) to predict the masked words. • Masked Video Modeling (MVM). The MVM objective aims to recover the masked video tokens [15,19,37,52].
To implement MVM, we apply a linear layer on the vision encoder and predict the masked tokens as in [46]. In the Table below and Fig. 2, we report that the MLM pretraining objective leads to a substantial boost in performance (+6.2%). Furthermore, adding MVM loss further improves the accuracy by 1%. However, adding the MVM objective slows the training by about 40% (due to additional forward and backward passes). Thus, to speed up the training, we don't use MVM loss in our remaining experiments.
objectives acc.(%) VTC (Step 1) 56.7 VTC+VTM (Step 2) 60.3 VTC+VTM+MLM 66.5 VTC+VTM+MLM+MVM 67.5
Takeaway #3: For the remaining experiments, we use VTC, VTM, MLM as our pretraining objectives.
In this section, we analyze the effect of (i) the pretraining data, and (ii) pretraining protocols.
Datasets. Recent methods [2,16] suggest that jointly pretraining on images and videos leads to better performance. To investigate this, we consider an additional image-based CC3M [48] consisting of 3M image-text pairs. Specifically, we experiment with pretraining our model on the (i) imageonly (CC3M), (ii) video-only (WebVid2M), and (iii) joint image and video (CC3M + WebVid2M) datasets. When pretraining on images, we replace our previously introduced temporal attention module with an identity connection. As shown in the Table below and Fig. 2, training on videos is more beneficial than training on images (+2.7%). Furthermore, jointly pretraining on images and videos leads to an additional 2.7% boost, which suggests that a stronger spatial representation is useful for VidL modeling.
Images Videos Images+Videos acc.(%) 64.8 67.5 70.2 The Number of Input Frames for Pretraining. Prior approaches [2, 16, 57, 75] use a different number of input frames for pretraining (i.e., from 1 to 16). Thus, we next study how many frames are needed for effective VidL pretraining. From the Table below and Fig. 2, we observe that multi-frame pretraining using 4 frames leads to 1.7% improvement compared to a single-frame pretraining. However, we also observe that the performance saturates with 4-frame inputs while the computational cost of pretraining with more frames increases significantly, i.e., pre-training with 4 frames is 2.5× faster than pretraining with 16 frames.
1 frame 4 frames 8 frames 16 frames acc.(%) 68.5 70.2 70.2 70.2 speedup 4.6× 2.5× 1.7× 1×
Multi-stage Curriculum Pretraining. Lastly, we validate the necessity of multi-stage curriculum pretraining, which was used in several prior VidL approaches [2,57]. Specifically, we experiment with two different pretraining protocols: (i) a two-stage pretraining that first trains a model for 10 epochs using single frames, and then for 5 additional epochs using 4-frame inputs, and (ii) a three-stage pretraining that builds on (i) by adding a third stage where the model is trained for additional 3 epochs using 8-frame inputs. Our results in the Table below and Figure 2, indicate that multi-stage pretraining does not lead to any significant performance boost, contrary to the findings of prior approaches [2,57]. We believe that this happens because prior approaches [2,57] train their model for only several epochs at each stage, whereas we train it until convergence. We also note that compared to the 4-frame one-stage pretraining, the two-stage 1 → 4 has a comparable pretraining cost as the latter model is trained for more epochs.
frames 4 1→ 4 1 → 4 → 8 4 → 8 acc.(%) 70.2 69.6 69.5 70.4 speedup 1.7× 1.7× 1.2× 1×
Takeaway #4: We adopt a single-stage pretraining on joint image and video datasets while using 4-frame inputs.
Existing methods typically use the same number of frames either between pretraining and finetuning [2,25,75] or finetuning and inference [16,57,75]. Here, we study using a different number of frames at different phases.
Finetuning. We experiment with finetuning our 4-frame pretrained model with K = 1, 4, 8, 12, 24, 32-frame inputs while using M frames during inference. We use M = 12 for all K ≤ 12 and M = K for K > 12. Based on the results in the Table below, we observe that while finetuning with more frames leads to higher accuracy (70.5%) the performance saturates with about 12 frames. We also note that finetuning with a single-frame input is 22.4× faster than with 32 frames but has a 5% lower accuracy. On the other hand, finetuning with 12 frames yields only 0.3% lower accuracy but 2.6× speedup compared to finetuning with 32 frames. Therefore, due to the favorable accuracy-cost tradeoff, we finetune most of our models with 12-frame inputs.
# frames 1 4 8 12 24 32 acc.(%) 65.5 68.1 69.2 70.2 70.1 70.5 speedup 22.4× 7.1× 3.9× 2.6× 1.5× 1.0× Inference. Next, we experiment with 12, 24, 32, 64 frames for testing our 4-frame pretrained and 12-frame finetuned model. We report the averaged accuracies on DiDeMo (D) / ActivityNet (A), which contain longer videos. Using more frames for inference helps, but the accuracy saturates quickly, and the inference cost becomes large. # frames 12 24 32 64 D/A acc.(%) 73.4/70.4 73.0/72.1 72.7/72.6 73.8/72.8 speedup 10.6× 3.1× 2.1× 1×
Takeaway #5: Considering the trade-off between computational cost and accuracy, we use 12 frames for finetuning and inference on all datasets except ActivityNet. On Activi-tyNet, we use 12 and 32 frames for finetuning and inference.
Step 6: Scaling Up Lastly, we scale up the pretraining data and the model. Table below and in Fig. 2 indicate that scaling our corpus from 5M → 17M improves the performance by 2.2%. Furthermore, scaling the corpus from 17M → 25M leads to an additional boost of 1.2%. # corpus 5M 17M 25M acc.(%) 70.2 72.4 73.6 Model Size. We also experiment with scaling the video encoder (ViT base → ViT large ) or text encoder (BERT base → BERT large ). Due to the large computational cost, we only conduct these experiments on the 5M corpus. We report that scaling the vision encoder brings larger improvement ( +3.0%) than scaling the text encoder (+1.0%). encoders base ViT large BERT large acc.(%) 70.2 73.2 71.2
Final Takeaway: Our final scaled-up VINDLU model improves the initial image transformer baseline by 23.2%.
We validate our VINDLU recipe on two mainstream VidL tasks. See implementation details and dataset descriptions in the supplementary material.
Text-to-Video Retrieval. We compare our results with existing methods on three spatially-biased datasets MSR-VTT, DiDeMo, and ActivityNet and two temporally-heavy datasets, SSv2-label, and SSv2-template as shown in Tab. 2 and Tab. 3 respectively. Our method outperforms previous methods by a large margin on multiple datasets, achieving averaged accuracies of 79.3% (+5.6%), 75.4% (+4.7%), 84.6% (+4.6%) on DiDeMo, ActivityNet-Captions and SSv2 respectively. Our results on MSR-VTT are worse (66.5% vs. 68.6%) than OmniVL [57] but our pretraining framework is significantly cheaper (i.e., 82 vs. 169 V100 GPU days). We also note that our method is significantly cheaper than other top-performing approaches including LAVENDER [32], All-in-one [55], and CLIP-ViP [66] (82 vs. 640, 448, 984 V100 GPU days for pretraining respectively). Additionally, our cheapest VINDLU variant requires only 15 V100 GPU days for pre-training, which is the second cheapest model among all listed approaches, and it still achieves competitive results on all three benchmarks. Furthermore, compared to the other leading VidL approaches such as OmniVL and Singularity, which rely on a multi-stage curriculum pretraining, our framework is simpler since it can be trained in a single stage. We also include the results of our scaled up variant VINDLU-L that uses ViT large as its video encoder, and report that it achieves 74.5% averaged retrieval accuracy, thus, outperforming all other approaches. Lastly, our results on the SSv2 dataset in Table 3 indicate that VINDLU performs well not only on spatially-biased datasets but also on temporally-heavy datasets, which require sophisticated temporal modeling capabilities. For fairer comparisons, we de-emphasize CLIPbased methods since they use a lot more pre-training data.
Video Question-Answering.
In Table 4, we also present our results for the video question-answering task on ActivityNet-QA [73], MSRVTT-QA [63], MSRVTT-MC [72] and TVQA [27]. Our results indicate that compared to prior state-of-the-art approaches, VIN-DLU achieves competitive results across all four of these datasets. In particular, our method outperforms existing approaches by 0.6% on ActivityNet-QA, 0.3% on MSRVTT-QA, 3.4% on MSRVTT-MC and 0.3% on TVQA. For fair comparison, we de-emphasize FrozenBiLM [68], since it is a lot larger than our model (1.2B vs. 201M parameters) and uses a lot more pretraining data (400M vs. 25M).
In this work, we demystify the importance of various components used in modern VidL framework design. Throughout our empirical study, we find that temporal modeling, multimodal fusion, masked modeling pretraining objectives, and joint training on images and videos are critical for good performance on the downstream VidL under- Table 3. Comparison with state-of-the-art text-to-video retrieval methods on the temporally-heavy SSv2-Label [25] and SSv2-Template datasets [25]. #PT denotes the amount of pretraining data. Averaged numbers are the average of Recal@{1,5,10} on these two datasets. CLIP-based models are de-emphasized for fairer comparisons. We observe that VINDLU achieves the best performance, which demonstrates its ability to reason about complex temporal dependencies in the video data.
standing tasks. Our empirical insights enable us to develop a step-by-step recipe for effective video-language (VidL) pretraining, which leads to a highly performant VidL model, dubbed VINDLU. Compared to the existing VidL approaches, our method achieves competitive or even better results on 9 VidL benchmarks while also being simpler and more efficient. While our paper does not provide any novel individual contributions, we believe that our empirical insights and our VidL pretraining recipe will be useful and help advance further research in the VidL domain.
Method #PT ANet MSR-QA MSR-MC TVQA ClipBERT [26] 0.2M -37.4 88.2 -ALPRO [28] 5M -42.1 --JustAsk [67] 69M 38.9 41.5 --VideoCLIP [64] 136M --92.1 -All-in-one [55] 138M -44.3 92.0 -MERLOT [75] 180M 41.4 43.1 90.9 78.7 VIOLET [16] 138M -43.9 91.9 -Singularity [25] 17M 44.1 43.9 93.7 -OmniVL [57] 17M -44.1 --HERO [31] 7.5M ---74.2 FrozenBiLM [68] 400M 43.2 47.0 -82.0 VINDLU 5M 44.2 43.6 95.4 79.0 17M 44.6 43.8 93.8 78.8 25M 44.7 44.6 95.5 79.0 Table 4.
Comparison with state-of-the-art video questionanswering methods on ActivityNet-QA (ANet), MSRVTT-QA (MSR-QA), MSRVTT-MC (MSR-MC) and TVQA. #PT denotes the amount of pretraining data. We gray out FrozenBiLM [68] as it is much larger than our model (1.2B vs 207M parameters). VIN-DLU achieves competitive results across all four datasets.
Authorized licensed use limited to: N.C. State University Libraries -Acquisitions & Discovery S. Downloaded on March 01,2024 at 19:31:46 UTC from IEEE Xplore. Restrictions apply.