
With Eq. (REF ), the CPA grounds the output word at current time step to the attended image regions.
However, in image captioning, when the output word \(y_{t}\)  is an attribute word that can be used to describe multiple objects in the image, the attention model may produce confusing attended image regions. For example, when there are a black shirt and black pants in the image, if we only adopt the word black to calculate the attention weights, the Prophet Attention model will be confused which image region it should attend to, as there are more than one proper image regions, i.e., shirt and pants.
In addition, when the \(y_{t}\)  is a non-visual word, e.g., of and the, there is no suitable visual information at all [1]}, [2]}, so we should also remove (i.e., mask) the Prophet Attention to prevent it from affecting the learning of the captioning model.
