Video as a space-time volume. Unconstrained modeling and learning of action features is possible when considering video in space-time. Since video can be seen as a temporally dense sampled sequence of images, expanding 2D convolution operation in 2D-CNNs to 3D convolution is a most intuitive approach to spatio-temporal feature learning [1]}, [2]}, [3]}. The major drawback of 3D cnn is the huge number of parameters involved. This results in increased computations and the requirement of large scale datasets for pre-training. Carreira and Zisserman [3]} addressed this limitation by inflating video 3D kernels with the 2D weights of a cnn trained for image recognition. Several other approaches focused on reducing the number of parameters by disentangling the spatial and temporal feature extraction operations. P3D [5]} proposes three different choices for separating the spatial and temporal convolutions and develops a 3D-ResNet architecture whose residual units are a sequence of such three modules. R(2+1)D [6]} and S3D-G [7]} also show that a 2D convolution followed by 1D convolution is enough to learn discriminative features for action recognition.
CoST [8]} performs 2D convolutions, with shared parameters, along the three orthogonal dimensions of a video sequence. CT-Net [9]} proposes to tensorize the channel dimension of features as a multiplication of K sub-dimensions, followed by 3D separable convolutions along each of the K channel sub-dimensions. This allows the model to gradually increase the spatio-temporal receptive field with reduced computational complexity while maintaining interaction of features across both spatio-temporal and channel dimensions. MultiFiber [10]} uses multiple lightweight networks, the fibers, and multiplexer modules that facilitate information flow using point-wise convolutions across the fibers. CIDC [11]} performs channel independent directional convolution by applying 2D convolution independently on individual feature channels while considering the temporal dimension as channels. Other works involve Neural Architecture Search (NAS) for developing an optimum architecture [12]}, [13]}, [14]}. X3D [15]} develops a progressive algorithm for expanding the spatio-temporal resolution of inputs, network depth, number of channels, etc., of 3D cnn architectures. The final architecture developed with the search algorithm results in improved performance with less computational complexity compared to hand-designed 3D cnn.
