Very recently, <cite>[8]</cite> proposed a multi-modal <cite>encoder-decoder framework</cite> that, given an image caption, jointly predicts another caption and the features of associated image. <cite>The work</cite> showed promising results for further improving general sentence representations by grounding them visually. However, according to <cite>the model</cite>, visual association only occurs at the final hidden state of the encoder, potentially limiting the effect of visual grounding.