Most existing 3D CNN structures for video representation learning are clip-based methods, and do not consider video-level temporal evolution of spatio-temporal features. In this paper, we propose Video-level 4D Convolutional Neural Networks, namely V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, as well as preserving 3D spatio-temporal representations with residual connections. We further introduce the training and inference methods for the proposed V4D. Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by a large margin. 3D convolutional neural networks (3D CNNs) and their variants (Ji et al., 2010; Tran et al., 2015; Carreira & Zisserman, 2017; Qiu et al., 2017; Wang et al., 2018b ) provide a simple extension from 2D counterparts for video representation learning. However, due to practical issues such as memory consumption and computational cost, these models are mainly used for clip-level feature learning instead of training from the whole video. In this sense, during training, the clip-based methods randomly sample a short clip (e.g., 32 frames) from the video for representation learning. During testing, they uniformly sample several clips from the whole video in a sliding window manner and calculate the prediction scores for each clip independently. Finally the prediction scores of all clips are simply averaged to yield the video-level prediction. Although achieving very competitive accuracy, these clip-based models ignore the video-level structure and long-range spatio-temporal dependency during training, as they only sample a small portion of the entire video. In fact, sometimes it could be very hard to recognize action class only with partial observation. Meanwhile, simply averaging the prediction scores of all clips could be also sub-optimal during testing. To overcome this issue, Temporal Segment Network (TSN) uniformly samples multiple clips from the entire video and uses their average score to guide back-propagation during training. Thus TSN is a video-level representation learning framework. However, the inter-clip interaction and video-level fusion in TSN is only performed at very late stage, which fails to capture finer temporal structures. In this paper, we propose a general and flexible framework for video-level representation learning, called V4D. As shown in Figure 1 , to model long-range dependency in a more efficient and principled way, V4D is composed of two critical design: (1) holistic sampling strategy and (2) 4D convolutional interaction. We first introduce a video-level sampling strategy by uniformly sampling a sequence of short-term units covering the holistic video. Then we model long-range spatio-temporal dependency by designing a unique 4D residual block. Specifically, we present a 4D convolutional operation to capture inter-clip interaction, which could enhance the representation power of the original cliplevel 3D CNNs. The 4D residual blocks could be easily integrated into the existing 3D CNNs to perform long-range modeling more earlier and hierarchically than TSN. We also design a specific video-level inference algorithm for V4D. Specifically, we verify the effectiveness of V4D on three video action recognition benchmarks, Mini-Kinetics (Xie et al., 2018) , Kinetics-400 (Carreira & Zisserman, 2017) and Something-Something-V1 (Goyal et al., 2017) . V4D structures achieve very competitive performance on these benchmarks and obtain evident performance improvement over their 3D counterparts. In this section, we will show that the proposed V4D can be considered as a 4D generalization of a number of recent widely-applied methods, which may partially explain why V4D works practically well on learning meaningful video-level representation. Temporal Segment Network. Our V4D is closely related to Temporal Segment Network (TSN). Although originally designed for 2D CNN, TSN can be directly applied to 3D CNN to model video-level representation. It also employs a video-level sampling strategy with each action unit named "segment". During training, each segment is calculated individually and the prediction scores after the fully-connected layer are then averaged. Since the fully-connected layer is a linear classifier, it is mathematically identical to calculating the average before the fully-connected layer (similar to our global average pooling) or after the fully-connected layer (similar to TSN). Thus our V4D can be considered as 3D CNN + TSN if all parameters in 4D Blocks are assigned zero. Dilated Temporal Convolution. One special form of 4D convolution kernel, k × 1 × 1 × 1, is closely related to Temporal Dilated Convolution (Lea et al., 2016) . The input tensor V can be considered as a (C, U × T, H, W ) tensor when all action units are concatenated along the temporal dimension. In this case, the k × 1 × 1 × 1 4D convolution can be considered as a dilated 3D convolution kernel of k × 1 × 1 with a dilation of T frames. Note that the k × 1 × 1 × 1 kernel is just the simplest form of our 4D convolutions, while our V4D architectures utilize more complex kernels and thus can be more meaningful for learning stronger video representation. Furthermore, our 4D Blocks utilize residual connections, ensuring that both long-term and short-term representation can be learned jointly. Simply applying the dilated convolution might discard the short-term fine-grained features. We have introduced new Video-level 4D Convolutional Neural Networks, namely V4D, to learn strong temporal evolution of long-range spatio-temporal representation, as well as retaining 3D features with residual connections. In addition, we further introduce the training and inference methods for our V4D. Experiments were conducted on three video recognition benchmarks, where our V4D achieved the state-of-the-art results. A APPENDIX
A novel 4D CNN structure for video-level representation learning, surpassing  recent 3D CNNs.
{"src": "scitldr-AIC_train", "paper_id": "SJeLopEYDH"}