Inspired from the improved performance of 2D cnn on image recognition tasks, [1]} explores several approaches to extend 2D cnn for action recognition by considering videos as a set of frames. This includes using a single frame from the video to stacking multiple frames and applying it to a 2D cnn. However it is found that both of these configurations perform similarly, indicating that the learned video representation is not capable of capturing motion information in the video. Two-Stream networks [2]}, [3]} address this problem by stacking optical flow images to capture short-term motion features and incorporate them with the appearance features obtained from a single frame. Such approaches require costly optical flow computations and ignore valuable appearance features from the video. TSN [4]} and ActionVLAD [5]} propose to extract features from multiple frames instead, followed by a late temporal aggregation via mean pooling for improved appearance feature extraction. Such approaches still relied on optical flow images for capturing motion information. The works of [6]}, [7]}, [8]} formulate late temporal aggregation as a sequence learning problem by encoding the frame-level features using variants of rnn. All these approaches perform temporal modelling in a shallow manner.
<FIGURE>