In the general RNN framework for sequence-to-sequence problems, the information of input is treated uniformly without discrimination on the different parts. This will result in fixed length of intermediate features and subsequent sub-optimal system performance. On the other hand, the practice is in sharp contrast to the way humans accomplish sequence processing tasks. Humans tend to selectively concentrate on a part of information and at the same time ignores other perceivable information. The mechanism of selectively focusing on relevant contents in the representation is called attention. The attention based RNN model in machine learning was successfully applied in natural language processing (NLP), and more specifically, in neural translation [1]}. For many visual recognition tasks, different portions of an image or segments of a video have unequal importance, which should be selectively weighted with attention. Xu et al. [2]} systematically analyzed the stochastic hard attention and deterministic soft attention models and applied them in image captioning tasks, with improved results compared with other RNN-like algorithms. The hard attention mechanism requires a stochastic neuron which is hard to train using conventional back propagation algorithm. They applied REINFORCE [3]} as an estimator to implement hard attention for image captioning.
