Action Recognition by Learning Deep Multi Granular Spatio Temporal Video Representation
Recognizing actions in videos is a challenging task as video is an information intensive media with complex variations. Most existing methods have treated video as a flat data sequence while ignoring the intrinsic hierarchical structure of the video content. In particular, an action may span different granularities in this hierarchy including, from small to large, a single frame, consecutive frames (motion) a short clip, and the entire video. In this paper, we present a novel framework to boost action recognition by learning a deep spatio temporal video representation at hierarchical multi granularity. Specifically, we model each granularity as a single stream by 2D (for frame and motion streams) or 3D (for clip and video streams) convolutional neural networks (CNNs) The framework therefore consists of multi stream 2D or 3D CNNs to learn both the spatial and temporal representations. Furthermore, we employ the Long Short Term Memory (LSTM) networks on the frame, motion, and clip streams to exploit long term temporal dynamics. With a softmax layer on the top of each stream, the classification scores can be predicted from all the streams, followed by a novel fusion scheme based on the multi granular score distribution. Our networks are learned in an end to end fashion. On two video action benchmarks of UCF101 and HMDB51, our framework achieves promising performance compared with the state of the art.
ICMR