Current attention model attends to image regions based on current hidden state [1]}, [2]}, which contains the information of past generated words.
As a result, the attention model has to predict attention weights without knowing the word it should ground.
Figure REF  illustrates a generated caption and the attended image regions from a state-of-the-art captioning system [3]}.
As we can see, the attended image regions are more grounded on current input word than the output one. For example, at the time step to generate the 5th word yellow, the attended image region is the woman instead of the umbrella. As a result, the incorrect adjective yellow is generated rather than the correct adjective red. This is mainly due to the “focus” of the attention is “deviated” several steps backwards and the conditioned words are woman and holding; Another example is at the time step to generate the 7th word wearing, the attended image region should be the woman instead of the umbrella. Although the generated word is correct, the unfavorable attended image region impairs the grounding performance [4]} and ruins the model interpretability,
because the attended image region often serves as a visual interpretation to qualitative measurement of the captioning model [4]}, [6]}, [7]}, [8]}, [9]}.
