
Attention Mechanisms.
Since attention showed impressive results in natural language processing tasks [1]}, using attention for video action recognition has become popular. Typically, these attention mechanisms are used to incorporate temporal information in 2D backbone networks (e.g, ResNets and MobileNets) which otherwise process the frames independently. One of the first works to use attention for video action recognition was Non-Local Networks [2]} which proposed using a self-attention block across the frames to improve temporal modeling. TAM [3]} further extended this idea by using dynamic convolutions and attention. Recently, TimeSformer [4]} processed the frames with a transformer network using a patch-based approach. Though these methods have shown impressive results, they do not incorporate any motion information, which is a key feature of videos.
