Visual attention mechanisms have been widely used in image captioning models. In this paper, to better link the image structure with the generated text, we replace the traditional softmax attention mechanism by two alternative sparsity-promoting transformations: sparsemax and Total-Variation Sparse Attention (TVmax). With sparsemax, we obtain sparse attention weights, selecting relevant features.   In order to promote sparsity and encourage fusing of the related adjacent spatial locations, we propose TVmax.   By selecting relevant groups of features, the TVmax transformation improves interpretability. We present results in the Microsoft COCO and Flickr30k datasets, obtaining gains in comparison to softmax.   TVmax outperforms the other compared attention mechanisms in terms of human-rated caption quality and attention relevance. The goal of image captioning is to generate a fluent textual caption that describes a given image (Farhadi et al., 2010; Kulkarni et al., 2011; Vinyals et al., 2015; Xu et al., 2015) . Image captioning is a multimodal task: it combines text generation with the detection and identification of objects in the image, along with their relations. While neural encoder-decoder models have achieved impressive performance in many text generation tasks Vaswani et al., 2017; Chorowski et al., 2015; Chopra et al., 2016) , it is appealing to design image captioning models where structural bias can be injected to improve their adequacy (preservation of the image's information), therefore strengthening the link between their language and vision components. State-of-the-art approaches for image captioning (Liu et al., 2018a; b; Anderson et al., 2018; Lu et al., 2018) are based on encoder-decoders with visual attention. These models pay attention either to the features generated by convolutional neural networks (CNNs) pretrained on image recognition datasets, or to detected bounding boxes. In this paper, we focus on the former category: visual attention over features generated by a CNN. Without explicit object detection, it is up to the attention mechanism to identify relevant image regions, in an unsupervised manner. A key component of attention mechanisms is the transformation that maps scores into probabilities, with softmax being the standard choice . However, softmax is strictly dense, i.e., it devotes some attention probability mass to every region of the image. Not only is this wasteful, it also leads to "lack of focus": for complex images with many objects, this may lead to vague captions with substantial repetitions. Figure 1 presents an example in which this is visible: in the caption generated using softmax (top), the model attends to the whole image at every time step, leading to a repetition of "bowl of fruit." This undesirable behaviour is eliminated by using our alternative solutions: sparsemax (middle) and the newly proposed TVMAX (bottom). In this work, we introduce novel visual attention mechanisms by endowing them with a new capability: that of selecting only the relevant features of the image. To this end, we first propose replacing softmax with sparsemax (Martins & Astudillo, 2016) . While sparsemax has been previously used in NLP for attention mechanisms over words, it has never been applied to computer vision to attend over image regions. With sparsemax, the attention weights obtained are sparse, leading to the selection (non-zero attention) of only a few relevant features. Second, to further encourage the weights of related adjacent spatial locations to be the same (e.g., parts of an object), we introduce a new attention mechanism: Total-Variation Sparse Attention (which we dub TVMAX), inspired by prior work in structured sparsity (Tibshirani et al., 2005; Bach et al., 2012) . With TVMAX, sparsity is allied to the ability of selecting compact regions. According to our human evaluation experiments, Figure 1 : Example of captions generated using softmax (top), sparsemax (middle) and TVMAX attention (bottom). Shading denotes the attention weight, with white for zero attention. The darker the green is, the higher the attention weight is. The full sequences are presented in Appendix C. this leads to better interpretability, since the model's behaviour is better understood by looking at the selected image regions when a particular word is generated. It also leads to a better selection of the relevant features, and consequently to the improvement of the generated captions. This paper introduces three main contributions: • We propose a novel visual attention mechanism using sparse attention, based on sparsemax (Martins & Astudillo, 2016) , that improves the quality of the generated captions and increases interpretability. • We introduce a new attention mechanism, TVMAX, that encourages sparse attention over contiguous 2D regions, giving the model the capability of selecting compact objects. We show that TVmax can be evaluated by composing a proximal operator with a sparsemax projection, and we provide a closed-form expression for its Jacobian. This leads to an efficient implementation of its forward and backward pass. • We perform an empirical and qualitative comparison of the various attention mechanisms considered. We also carry out a human evaluation experiment, taking into account the generated captions as well as the perceived relevance of the selected regions. We propose using sparse and structured visual attention, in order to improve the process of selecting the features relevant to the caption generation. For that, we used sparsemax and introduced TVMAX. Results on the image captioning task, show that the attention mechanism is able to select better features when using sparsemax or TVMAX. Furthermore, in the human assessment and attention analysis we see that the improved selection of the relevant features as well as the ability to group spatial features lead to the generation of better captions, while improving the model's interpretability. In future work, TVMAX attention can be applied to other multimodal problems such as visual question answering. It can also be applied in other tasks for which we have prior knowledge of the data's stucture, for instance graphs or trees. Summing up the Eq. 17 over all j ∈ G, we observe that for any k ∈ G, the term λt jk appears twice with opposite signs. Thus, Dividing by |G| gives exactly Eq. 8. This reasoning applies to any group G i .
We propose a new sparse and structured attention mechanism, TVmax, which promotes sparsity and encourages the weight of related adjacent locations to be the same.
{"src": "scitldr-AIC_train", "paper_id": "r1e8WTEYPB"}