MOTIVATION Attention based models have been shown to achieve promising results on several challenging tasks, including caption generation [9], machine translation [1], game-playing and tracking [4]. Attention based models can potentially infer the action happening in videos by focusing only on the relevant places in each video frame. Soft-attention models are deterministic and can be trained using backpropagation. We propose a soft-attention based model for action recognition in videos. We use multi-layered Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM). Our model tends to recognize important elements in video frames based on the activities it detects.
Most of the existing approaches also tend to have CNNs underlying the LSTMs and classify sequences directly or do temporal pooling of features prior to classification . LSTMs have also been used to learn an effective representation of videos in unsupervised settings by using them in an encoder-decoder framework. More recently, @cite_0 have proposed to use 3-D CNN features and an LSTM decoder in an encoder-decoder framework to generate video descriptions. Their model incorporates attention on a video level by defining a probability distribution over frames used to generate individual words. They, however, do not employ an attention mechanism on a frame level (i.e. within a single frame).
@cite_0
2133564696
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.