Visual attention mechanisms have been widely used in image captioning models. In this paper, to better link the image structure with the generated text, we replace the traditional softmax attention mechanism by two alternative sparsity-promoting transformations: sparsemax and Total-Variation Sparse Attention (TVmax). With sparsemax, we obtain sparse attention weights, selecting relevant features. In order to promote sparsity and encourage fusing of the related adjacent spatial locations, we propose TVmax. By selecting relevant groups of features, the TVmax transformation improves interpretability. We present in the Microsoft COCO and Flickr30k datasets, obtaining gains in comparison to softmax. TVmax outperforms the other compared attention mechanisms in terms of human-rated caption quality and attention relevance. The goal of image captioning is to generate a fluent textual caption that describes a given image (; ; ;). Image captioning is a multimodal task: it combines text generation with the detection and identification of objects in the image, along with their relations. While neural encoder-decoder models have achieved impressive performance in many text generation tasks;; ), it is appealing to design image captioning models where structural bias can be injected to improve their adequacy (preservation of the image's information), therefore strengthening the link between their language and vision components. State-of-the-art approaches for image captioning (a; b; ;) are based on encoder-decoders with visual attention. These models pay attention either to the features generated by convolutional neural networks (CNNs) pretrained on image recognition datasets, or to detected bounding boxes. In this paper, we focus on the former category: visual attention over features generated by a CNN. Without explicit object detection, it is up to the attention mechanism to identify relevant image regions, in an unsupervised manner. A key component of attention mechanisms is the transformation that maps scores into probabilities, with softmax being the standard choice. However, softmax is strictly dense, i.e., it devotes some attention probability mass to every region of the image. Not only is this wasteful, it also leads to "lack of focus": for complex images with many objects, this may lead to vague captions with substantial repetitions. Figure 1 presents an example in which this is visible: in the caption generated using softmax (top), the model attends to the whole image at every time step, leading to a repetition of "bowl of fruit." This undesirable behaviour is eliminated by using our alternative solutions: sparsemax (middle) and the newly proposed TVMAX (bottom). In this work, we introduce novel visual attention mechanisms by endowing them with a new capability: that of selecting only the relevant features of the image. To this end, we first propose replacing softmax with sparsemax . While sparsemax has been previously used in NLP for attention mechanisms over words, it has never been applied to computer vision to attend over image regions. With sparsemax, the attention weights obtained are sparse, leading to the selection (non-zero attention) of only a few relevant features. Second, to further encourage the weights of related adjacent spatial locations to be the same (e.g., parts of an object), we introduce a new attention mechanism: Total-Variation Sparse Attention (which we dub TVMAX), inspired by prior work in structured sparsity . With TVMAX, sparsity is allied to the ability of selecting compact regions. According to our human evaluation experiments, Figure 1: Example of captions generated using softmax (top), sparsemax (middle) and TVMAX attention (bottom). Shading denotes the attention weight, with white for zero attention. The darker the green is, the higher the attention weight is. The full sequences are presented in Appendix C. this leads to better interpretability, since the model's behaviour is better understood by looking at the selected image regions when a particular word is generated. It also leads to a better selection of the relevant features, and consequently to the improvement of the generated captions. This paper introduces three main contributions: • We propose a novel visual attention mechanism using sparse attention, based on sparsemax , that improves the quality of the generated captions and increases interpretability. • We introduce a new attention mechanism, TVMAX, that encourages sparse attention over contiguous 2D regions, giving the model the capability of selecting compact objects. We show that TVmax can be evaluated by composing a proximal operator with a sparsemax projection, and we provide a closed-form expression for its Jacobian. This leads to an efficient implementation of its forward and backward pass. • We perform an empirical and qualitative comparison of the various attention mechanisms considered. We also carry out a human evaluation experiment, taking into account the generated captions as well as the perceived relevance of the selected regions. Attention mechanisms have the ability to select the relevant features, in this case spatial locations. This requires a mapping from importance scores to a distribution,, p 0 denotes the simplex (the set of all probability distributions over k values). The standard choice for this mapping is softmax, defined as: However, as softmax is strictly positive, its output is dense. Thus, the model must pay some attention to the whole image and, consequently, assign lower attention weights to the relevant regions. This motivates our proposed selective visual attention mechanisms, which, by being sparse, are able to better isolate the relevant image regions. To achieve selective capabilities, we propose the use of sparsemax , a sparse mapping consisting in the Euclidean projection of z onto the probability simplex: which allows to obtain sparse outputs with a small increase in complexity. Output sparsity is an attractive property for attention mechanisms, since some features do not provide relevant information for the current prediction. In the image captioning case, using sparsemax allows focusing only on the spatial locations of the image that are relevant to the word being generated, assigning zero attention weight to all other regions. To generate descriptive captions, the model should identify the objects present in the image. Thus, when generating object-related words, the attention mechanism should assign high weights to the regions of the image containing the object. However, sparsemax is unstructured and index-invariant, leading it to select discontinuous regions. To overcome this, we propose a new visual attention mechanism, TVMAX. TVMAX is a non-trivial generalization of fusedmax , a transformation based on fused lasso, to the 2D case. To this end, we first extend fusedmax even more generally, to arbitrary graphs. Let w ∈ R k, and let I = {1, . . ., k}. Consider a graph over I defined by its edges E ⊆ I × I, where an edge between i and j means we want to encourage w i to be close to w j. For simplicity we use i ∼ j as shorthand for (i, j) ∈ E. The generalized fused lasso penalty is defined as: Minimizing Ω E encourages "fused" solutions, i.e., it encourages w i = w j for i ∼ j. In particular, its proximal operator 1 can be seen as a fused signal approximator, seeking a vector w that approximates z well (in terms of Euclidean distance) and that is encouraged to be fused: Computing the value of prox λΩ E is non-trivial in general , but for certain edge configurations, described below, efficient algorithms exist. • If E forms a chain, i.e. i ∼ j ⇐⇒ i = j − 1, the problem is called 1D total variation and can be solved in O(k) time using the taut string algorithm . We use the quasilinear algorithm of , which is very fast in practice. • If the indices are aligned on a 2D grid, as in an image, and i ∼ j holds iff. j is to the right or immediately below i, the problem is called 2D total variation. Unlike the 1D case, exact algorithms are not available. However, for an input of size a × b, it is possible to split the penalty into a column-wise and b row-wise 1D problems. We may then apply a number of iterative methods, for instance proximal Dykstra . TVMAX combines 2D total variation (TV2D) regularization with sparsemax. This way it promotes sparsity and encourages the attention weights of adjacent spatial locations to be the same, selecting contiguous regions of the image. TVMAX is defined as follows: Definition 1 (TVMAX). Let z ∈ R k, such that z's indices can be decomposed into rows and columns. The TVMAX transformation is defined as where λ is an hyper-parameter controlling the amount of fusion (λ = 0 recovers sparsemax) and Ω T V 2D is a 2D total variation penalty. Note that Eq. 5 differs from Eq. 4 in which the variable p is further constrained to lie in the probability simplex. We show next how the forward and backward passes can be efficiently computed. To construct generalized fused sparse attention, we follow and define This can be seen as a constrained fused lasso approximator, because the solution p must be a probability distribution vector. While the optimization function is very similar to Eq. 4, the additional constraint that p ∈ increases complexity. Fortunately, the following holds: Proposition 1 (Computing generalized fusedmax). The proof is given in Appendix A.2. Proposition 1 also provides a shortcut for deriving the Jacobian of generalized fusedmax via the chain rule: denoting by J F the Jacobian of prox λΩ E, we have As we already know how to compute J sparsemax (Appendix A.1), we may concentrate our effort on deriving the simpler J F (Eq. 9). Proposition 2 (Group-wise characterization of prox λΩ E). Let w:= prox λΩ E, and denote by G i the set of indices fused to w i in the solution, G i may be defined recursively: Define s ij = sign(w i − w j). Then, the solution has the expression Proposition 2 shows how to easily compute a generalized Jacobian of gfusedmax: since small perturbations in z never change the groups G i nor the signs of across-group differences s ij, differentiating Eq. 8 yields This generalizes Lemma 1 of to generalized fused lasso, with a simpler proof, given in Appendix A.3. As we show in Proposition 1, computing TVMAX's forward pass can be done by chaining efficient algorithms for TV2D and sparsemax. From Eq. 7 we have that TVMAX's Jacobian can be computed as J TVMAX = J sp (prox λΩ T V (z))J tv (z), where J sp is the sparsemax's Jacobian and J tv is the Jacobian of the Total Variation proximal operator. 3 As derived in Proposition 2, (J tv) i,j = 1 /nij if i and j are fused in a group with n ij elements, and 0 otherwise. The backward pass intuitively involves "spreading" the credit assigned to one image location evenly across all locations fused with it. This can be implemented by Algorithm 1 in O(k+N g log k) where N g is the number of groups of fused positions. In the worst case, when there are no positions fused, the complexity is O(k + k log k). This algorithm is inspired by flood filling algorithms . Algorithm 1 TVMAX backward pass (Jacobian-vector products) 1 Input: p = TVMAX(z), dp ∈ R k. 2 Output: dz = J TVMAX (dp) dp # Eqs. 14 and 15 of §A.1 8 while |V | < k do # check if all positions have been visited if (i, j) ∈ V then push (i, j) to N # add not visited neighbours of (i, j) to the stack 17 if G not empty then: To compare the proposed attention mechanisms, we use a straight-forward simple encoder-decoder model with visual attention, inspired by Liu et al. (2018a). The model is sketched in Figure 2. Given an image, we use a residual CNN pretrained on ImageNet to get a feature map with spatial dimension of size 8 × 8 and channel dimension of size 2048, that go through a fine-tuned feedforward layer yielding g = 512 feature maps. The visual feature matrix V = [v 1, v 2, . . ., v k], with v i ∈ R g and k = 64 = 8 × 8, contains the image information used to generate the corresponding caption. Following Liu et al. (2018a), we use input and output attention to select the relevant features for the current generation. To generate the word at position t, the input attention, α t, is computed using the LSTM's previous hidden state, h t−1 ∈ R d. First, a similarity score z t,i, i ∈ {1, . . ., k}, is computed between h t−1 and the i th image cell via a feedforward transformation, as z t,i = w tanh(affine([v i ; h t−1])), for all k image cells. Then, α t is obtained by normalizing the k-dimensional vector of scores z t with softmax, α t = softmax(z t). Using these attention weights, a vector representation of the image to be used as input of the LSTM, is obtained, s t = V α t. The output attention α t, is computed in the same way as above, but applied to the current LSTM hidden state h t, instead of h t−1, and normalized with the different proposed transformations. This produces output visual features s t = V α t, which are passed through a feedforward layer to yield the image representation r t = tanh(affine( s t)). Finally, the predictive probability of the next word is: Settings. The input images are resized to 256 × 256 before going through the residual CNN and the feature maps obtained have a size of 8 × 8. We use an LSTM hidden size of d = 512 and a word embedding size of 256, for all models. The models were trained for 50 epochs using the Adam optimizer with a learning rate of 0.0001 and a decay of 0.8 and 0.999 for the first and second momentum, respectively. After the 10 th epoch, the learning rate starts decaying with a decay factor of 0.99. For TVMAX, we set λ = 0.01. Datasets and Metrics. We report our on the Microsoft COCO (MSCOCO) and Flickr30k datasets. MSCOCO is composed of 113,287 images of common objects in context while Flickr30k consists in 31,000 pictures of people involved in everyday activities and events. Each image is annotated with 5 captions. We use the split proposed by , which stipulates equal validation and test sizes of 5,000 images (MSCOCO) and 1,000 (Flickr30k). The metrics we report are SPICE , CIDEr , longest common subsequence ROUGE, (denoted ROUGE L ;), 1-to 4-gram BLEU (denoted BLEU 4 ;), and METEOR . To investigate whether selective attention alleviates repetition, we also measure the n-gram repetition metric REP . Automated metrics. As can be seen in table 1, overall sparsemax and TVMAX attention mechanisms achieve better when compared with softmax, indicating that the use of selective attention leads to better captions. This improvement does not come at a high computational cost: at Figure 3: Example of captions generated using softmax (top), sparsemax (middle) and TVMAX attention (bottom). Shading denotes the attention weight, with white for zero attention. The darker the green is, the higher the attention weight is. The full sequences are presented in Appendix C. inference time, models using TVMAX and sparsemax are only 1.3x and 1.1x slower than softmax. Moreover, for TVMAX, automatic metrics are slightly worse than sparsemax but still superior to softmax on MSCOCO and similar on Flickr30k. We show next that this is compensated with fewer repetitions and higher scores in the human evaluation of the captions and attention relevance. Human rating. The caption evaluation consisted in attributing a score from 1 to 5 to the caption of each model while the attention evaluation consisted in scoring the relevancy of the attended areas, from 1 to 5, when generating the non stop words of the captions. A full description of the human assessment can be found in Appendix B. Despite performing slightly worse than sparsemax under automated metrics, TVMAX outperforms sparsemax and softmax in the caption human evaluation and the attention relevance human evaluation, reported in Table 2. The superior score on attention relevance shows that TVMAX is better at selecting the relevant features and its output is more interpretable. Additionally, the better caption evaluation demonstrate that the ability to select compact regions induces the generation of better captions. We next explore possible explanations for the TVMAX superior . Repetition. Figure 1 illustrates that softmax attention is prone to spuriously repeating references to the same object. Selective attention mechanisms like sparsemax and especially TVMAX reduce repetition, as measured by the REP metric reported in Table 1. This expected success can be attributed to the sparsity of the attention weights distribution and to the ability to select compact regions exclusively and can be one of the causes of the human evaluation . This happens even though TVMAX generates longer sentences than sparsemax and softmax (9.5 against 9.0 words on average) and shows the benefit of promoting structured and sparse attention simultaneously. To corroborate our intuition that sparsity leads to less repetition, we measured the Jensen-Shannon divergence (JS) between the attention distributions for each step of the generation of the captions correspondent to the images of the MSCOCO test set. The mean JS values are 0.12, 0.29, and 0.34 for softmax, sparsemax, and TVmax, respectively. This shows that sparsity leads to less similar attention distributions along the generation of the captions and, consequently, to less repetitions. Object detection. Using the MSCOCO object detection ground truth, we compared the percentage of objects present in the image that are referred to in the captions, using each attention mechanism. With TVMAX 28.2% of the reference objects are referred, against 27.5% and 27.4% for sparsemax and softmax, repectively. This shows that promoting high attention to groups of spatial locations of the image leads to a more precise identification of the objects. Sparsity. The average image area that receives zero attention is 34% for sparsemax and 25% for TVMAX. To illustrate where the models attend to, we display the output attention in Figures 1 and 3. As expected, softmax weights are spread widely across the image, ending up missing the relevant regions. In contrast, sparsemax and TVMAX weights are zero for the non-relevant spatial locations. Qualitative comparison. As the image of Figure 1 contains various similar objects, the softmax model (top) generates a incoherent, repetition-laden caption. In contrast, the sparsemax (middle) and TVMAX (bottom) models better identify the relevant parts of the image, generating coherent and descriptive captions. Moreover, the groups obtained with TVMAX are clearly visible and more aligned to object boundaries, offering better interpretability, as revealed by human attention assessment. In Figure 3 it can also be noticed that with TVMAX (bottom) the model correctly identified "a group of people" instead of "a soccer player" as with sparsemax (middle) and softmax (top). This indicates its superior ability to correctly define the relevant groups of features and that this ability leads to improved captions. Image captioning. In the last years, neural models with visual attention mechanisms have been receiving increased interest. Several researchers have been studying diverse attention mechanisms in order to refine visual information for image captioning. proposed the use of hard attention, which only attends to one region at each step. However, to generate descriptive captions the model should, often, focus on more than one region. In addition, hard attention is non-differentiable, requiring imitation learning or Monte Carlo policy gradient approximations. proposed bottom-up attention, using an object detection model designed to identify bounding boxes of objects, and top-down attention, selecting the relevant bounding-boxes. proposed an hierarchical attention network composed by a patch detector, object detector, and concept detector. Using object detection models is less demanding on the attention mechanism, since it only has to select the boxes the model should attend to. However, such models are limited by the bounding boxes position's accuracy. introduced a deliberate attention network to refine the attended visual features. Yet, the attention distribution remained dense. Sparse attention. In several tasks only a few features are relevant for the current prediction. This can be attained when using sparse attention. Various prior works have proposed sparse attention mechanisms with promising , (; ; ;). proposed 1D fusedmax, which incorporates the fused lasso, so that adjacent words are encouraged to have the same attention weight. In this work, the authors were able to improve interpretability without sacrificing performance, obtaining superior on textual entailment and summarization. We derive a generalized fused attention mechanism, extending 1D fusedmax. We propose using sparse and structured visual attention, in order to improve the process of selecting the features relevant to the caption generation. For that, we used sparsemax and introduced TVMAX. Results on the image captioning task, show that the attention mechanism is able to select better features when using sparsemax or TVMAX. Furthermore, in the human assessment and attention analysis we see that the improved selection of the relevant features as well as the ability to group spatial features lead to the generation of better captions, while improving the model's interpretability. In future work, TVMAX attention can be applied to other multimodal problems such as visual question answering. It can also be applied in other tasks for which we have prior knowledge of the data's stucture, for instance graphs or trees. Summing up the Eq. 17 over all j ∈ G, we observe that for any k ∈ G, the term λt jk appears twice with opposite signs. Thus, Dividing by |G| gives exactly Eq. 8. This reasoning applies to any group G i. To perform the human evaluation firstly 100 images were randomly selected from the test set of the MSCOCO dataset (using the split proposed by). For each of the selected images, the human evaluators selected a score from 1 to 5 for the captions generated by the models using softmax attention, sparsemax attention, and TVMAX attention. They were also asked to evaluate whether the models attend to the relevant regions of the image when generating a certain word. For that they observed the attention plots corresponding to the non stop words of the caption of each of the models. While in Figures 1 and 3, 4, and 5 we emphasized sparsity with a hard white mask, for the human evaluation the sparse regions of the attention plots were simply fully transparent, to avoid biasing the evaluators. The possible scores were also between 1 and 5. The 100 images were judged by 6 persons both for the captions evaluation and attention evaluation. The order of the captions and attention plots was randomly chosen for each image. With these scores, we computed the mean of the captions evaluation scores and the mean of the attention relevance evaluation scores. The are reported in Table 2. C ADDITIONAL ATTENTION VISUALIZATION Figure 4: Example generated captions using softmax attention (top), sparsemax attention (middle) and TVMAX attention (bottom). The captions are "A bowl of fruit and a bowl of fruit", "A bowl of fruit and oranges on a table" and "A bowl of oranges and a banana on a table". Figure 5: Example generated captions using softmax attention (top), sparsemax attention (middle) and TVMAX attention (bottom). The captions are "A soccer player is running to the base", "A soccer player is running to the field" and "A group of people playing soccer on a field".
We propose a new sparse and structured attention mechanism, TVmax, which promotes sparsity and encourages the weight of related adjacent locations to be the same.