Current state-of-the-art approaches to video understanding adopt temporal jittering to simulate analyzing the video at varying frame rates. However, this does not work well for multirate videos, in which actions or subactions occur at different speeds. The frame sampling rate should vary in accordance with the different motion speeds. In this work, we propose a simple yet effective strategy, termed random temporal skipping, to address this situation. This strategy effectively handles multirate videos by randomizing the sampling rate during training. It is an exhaustive approach, which can potentially cover all motion speed variations. Furthermore, due to the large temporal skipping, our network can see video clips that originally cover over 100 frames. Such a time range is enough to analyze most actions events. We also introduce an occlusion-aware optical flow learning method that generates improved motion maps for human action recognition. Our framework is end-to-end trainable, runs in real-time, and achieves state-of-the-art performance on six widely adopted video benchmarks.
To handle multirate videos, there are two widely adopted approaches. One is to train multiple models, each of them corresponding to a different fixed frame-rate. This is similar to using image pyramids to handle the multi-scale problem in image analysis. The other is to generate sliding windows of different lengths for each video (a.k.a, temporal jittering), with the hope of capturing temporal invariance. However, neither of these approaches is exhaustive, and they are both computationally intensive. @cite_27 is the most similar work to ours since they deal with motion speed variance. However, our work differs in several aspects. First, we aim to explicitly learn the transitions between frames while @cite_27 uses past and future neighboring video clips as the temporal context, and reconstruct the two temporal transitions. Their objective is considerably harder to optimize, which may lead to sub-optimal solutions. Second, our random skipping strategy is easy to implement without computational overhead whereas the image reconstruction of @cite_27 will lead to significant computational burden. Third, their proposed multirate gated recurrent unit only works in RNNs, while our strategy is generally applicable.
@cite_27
2952550003
Despite the recent success of neural networks in image feature learning, a major problem in the video domain is the lack of sufficient labeled data for learning to model temporal information. In this paper, we propose an unsupervised temporal modeling method that learns from untrimmed videos. The speed of motion varies constantly, e.g., a man may run quickly or slowly. We therefore train a Multirate Visual Recurrent Model (MVRM) by encoding frames of a clip with different intervals. This learning process makes the learned model more capable of dealing with motion speed variance. Given a clip sampled from a video, we use its past and future neighboring clips as the temporal context, and reconstruct the two temporal transitions, i.e., present-past transition and present-future transition, reflecting the temporal information in different views. The proposed method exploits the two transitions simultaneously by incorporating a bidirectional reconstruction which consists of a backward reconstruction and a forward reconstruction. We apply the proposed method to two challenging video tasks, i.e., complex event detection and video captioning, in which it achieves state-of-the-art performance. Notably, our method generates the best single feature for event detection with a relative improvement of 10.4 on the MEDTest-13 dataset and achieves the best performance in video captioning across all evaluation metrics on the YouTube2Text dataset.