Encode, Review, and Decode: Reviewer Module for Caption Generation
We propose a novel module, the reviewer module, to improve the encoder decoder learning framework. The reviewer module is generic, and can be plugged into an existing encoder decoder model. The reviewer module performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a fact vector after each review step; the fact vectors are used as the input of the attention mechanism in the decoder. We show that the conventional encoder decoders are a special case of our framework. Empirically, we show that our framework can improve over state of the art encoder decoder systems on the tasks of image captioning and source code captioning.
ArXiv