Review Networks for Caption Generation
We propose a novel extension of the encoder decoder framework, called a review network. The review network is generic and can enhance any existing encoder decoder model: in this paper, we consider RNN decoders with both CNN and RNN encoders. The review network performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a thought vector after each review step; the thought vectors are used as the input of the attention mechanism in the decoder. We show that conventional encoder decoders are a special case of our framework. Empirically, we show that our framework improves over state of the art encoder decoder systems on the tasks of image captioning and source code captioning.
NIPS