Modeling long-term dependencies. Majority of existing approaches rely on the network depth to combine the information present in frames that are further apart in time. [1]} develops a feature bank that stores features from the entire video which can then be integrated as contextual information to a video recognition model for understanding the present. Non-local [2]} is a plug and play module that computes the features at a specific position in space-time as the weighted sum of features at all other positions, thereby capturing long-range dependencies. Timeception [3]} introduces multi-scale separable temporal convolutions to reason about videos of longer duration. Eidectic 3D LSTM [4]} integrates 3D convolution into rnn and introduces a gated controlled self-attention module to enable the memory to interact with its past values. The former enables short-term temporal modeling while the latter allows the model to encode long-term temporal information. VideoGraph [5]} constructs a unidirectional graph whose nodes represent the latent action concepts. A novel graph embedding layer is then developed to learn the relationship between these concepts. Zhang et al.  [6]} introduces 4D convolution to perform long-term modeling hierarchically. Inspired by the performance improvement obtained by Transformers in Natural Language Processing, Vision Transformer (ViT) [7]} is developed for image recognition. TimeSformer [8]} extends ViT by developing different variants of self-attention schemes for spatio-temporal feature learning. However, even the efficient variant of TimeSformer suffers from increased computational complexity and larger parameter count.
