Efficient architectures. cnn provide different levels of feature abstractions at different layers of the hierarchy. It has been found that the bottom layer features are less useful for extracting discriminative motion cues [1]}, [2]}, [3]}. In [1]} it is proposed to apply 1D convolution layers on top of a 2D cnn for video action recognition. The works of [2]} and [3]} show that it is more effective to apply full 3D and separable 3D convolutions at the top layers of a 2D cnn for extracting spatio-temporal features. These approaches resulted in performance improvement over full 3D architectures with less parameters and computations. Static features from individual frames represent scenes and objects and can also provide important cues in identifying the action. This is validated by the improved performance obtained with two-path structures that apply a parallel 2D convolution in addition to the 3D convolution [7]}, [8]}. MiCT [7]} is designed by adding 3D convolution branches in parallel to the 2D convolution branches of a BN-Inception-like cnn. GST [8]} makes use of the idea of grouped convolutions for developing an efficient architecture for action recognition. They separate the features at a hierarchy across the channel dimension and separately perform 2D and 3D convolutions followed by a concatenation operation. In this way, the performance is increased while reducing the number of parameters. MVFNet [11]} extends the idea of GST by replacing the 3D convolution with three 1D convolutions applied to each of the two spatial dimensions and temporal dimension. STM [12]} proposes two parallel blocks for extracting motion features and spatio-temporal features. Their network rely only on 2D and 1D convolutions and feature differencing for encoding motion and spatio-temporal features. TSM [13]} proposes to shift the features across the channel dimension as a way to perform temporal interaction between the features from adjacent frames of a video. This parameter-less approach has resulted in similar performance to 3D cnn. RubiksNet [14]} replaces the convolution operation in 3D cnn with learnable spatio-temporal shifting. TEA [15]} develops a novel ResNet based architecture that weights channels with motion information while suppressing those without relevant information and that can encode long range temporal information in a hierarchical aggregation fashion. MSNet [16]} develops a plug in module that can estimate the correspondences between adjacent frame level features and convert these into effective motion features that can be injected into the backbone cnn for efficient spatio-temporal modeling.
