More Like this

1. Progress-Aware Video Frame Captioning

   Xue, Z; An, J; Yang, X; Grauman, K (March 2025, https://doi.org/10.48550/arXiv.2412.02071)

   While image captioning provides isolated descriptions for individual images, and video captioning offers one single narrative for an entire video clip, our work explores an important middle ground: progress-aware video captioning at the frame level. This novel task aims to generate temporally fine-grained captions that not only accurately describe each frame but also capture the subtle progression of actions throughout a video sequence. Despite the strong capabilities of existing leading vision language models, they often struggle to discern the nuances of frame-wise differences. To address this, we propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence. Alongside, we develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality. The results demonstrate that ProgressCaptioner significantly surpasses leading captioning models, producing precise captions that accurately capture action progression and set a new standard for temporal precision in video captioning. Finally, we showcase practical applications of our approach, specifically in aiding keyframe selection and advancing video understanding, highlighting its broad utility. 

   more » « less
2. Capturing Video Frame Rate Variations via Entropic Differencing

   https://doi.org/10.1109/LSP.2020.3028687  

   Madhusudana, Pavan C.; Birkbeck, Neil; Wang, Yilin; Adsumilli, Balu; Bovik, Alan C. (January 2020, IEEE Signal Processing Letters)

   null (Ed.)
3. Active Learning for Video Classification with Frame Level Queries

   https://doi.org/10.1109/IJCNN54540.2023.10191348  

   Goswami, Debanjan; Chakraborty, Shayok (June 2023, IEEE International Joint Conference on Neural Networks (IJCNN))
4. Frame-Recurrent Video Inpainting by Robust Optical Flow Inference

   https://doi.org/https://arxiv.org/pdf/1905.02882.pdf  

   Ding, Y; Wang, C; Huang, H; Liu, J; Wang, J; Wang, L (May 2019, ArXivorg)

   In this paper, we present a new inpainting framework for recovering missing regions of video frames. Compared with image inpainting, performing this task on video presents new challenges such as how to preserving temporal consistency and spatial details, as well as how to handle arbitrary input video size and length fast and efficiently. Towards this end, we propose a novel deep learning architecture which incorporates ConvLSTM and optical flow for modeling the spatial-temporal consistency in videos. It also saves much computational resource such that our method can handle videos with larger frame size and arbitrary length streamingly in real-time. Furthermore, to generate an accurate optical flow from corrupted frames, we propose a robust flow generation module, where two sources of flows are fed and a flow blending network is trained to fuse them. We conduct extensive experiments to evaluate our method in various scenarios and different datasets, both qualitatively and quantitatively. The experimental results demonstrate the superior of our method compared with the state-of-the-art inpainting approaches. 

   more » « less
5. CNN Based Key Frame Extraction for Face in Video Recognition

   https://doi.org/10.1109/ISBA.2018.8311477  

   Qi, Xuan; Liu, Chen; Schuckers, Stephanie (January 2018, IEEE 4th International Conference on Identity, Security, and Behavior Analysis (ISBA))