Deep Bayesian Active Learning for Multiple Correct Outputs

Typical active learning strategies are designed for tasks, such as classification, with the assumption that the output space is mutually exclusive. The assumption that these tasks always have exactly one correct answer has resulted in the creation of numerous uncertainty-based measurements, such as entropy and least confidence, which operate over a model's outputs. Unfortunately, many real-world vision tasks, like visual question answering and image captioning, have multiple correct answers, causing these measurements to overestimate uncertainty and sometimes perform worse than a random sampling baseline. In this paper, we propose a new paradigm that estimates uncertainty in the model's internal hidden space instead of the model's output space. We specifically study a manifestation of this problem for visual question answer generation (VQA), where the aim is not to classify the correct answer but to produce a natural language answer, given an image and a question. Our method overcomes the paraphrastic nature of language. It requires a semantic space that structures the model's output concepts and that enables the usage of techniques like dropout-based Bayesian uncertainty. We build a visual-semantic space that embeds paraphrases close together for any existing VQA model. We empirically show state-of-art active learning results on the task of VQA on two datasets, being 5 times more cost-efficient on Visual Genome and 3 times more cost-efficient on VQA 2.0.


Introduction
Active Learning is an approach for data annotation in supervised learning problems that selectively seeks labels for informative examples [68,60]. It has proven to be a potential solution [68] to maximize learning while reducing the data annotation cost for many problems, including visionlanguage generation tasks like Visual Question Answering (VQA) [52,11,44] or image captioning [63] that rely on large, expensive, curated datasets [3,39,78,21]. Nonetheless, applying existing active learning strategies to language generation tasks does not result in performance increase OUTPUT Figure 1: Many vision tasks, such as image captioning and visual question answering, have multiple correct answers for the same input. In this example, multiple correct answers exist: "Glasses", "A pair of Glasses", etc. Existing active learning strategies overestimate uncertainty as they fail to account for paraphrases. We propose a new uncertainty estimation framework built over a visual-semantic embedding space that extends Monte-Carlo dropout-based Bayesian uncertainty for VQA and demonstrate that it is more sample efficient across multiple datasets.
compared to randomly annotating examples [14,48,62]. Traditionally designed for classification and regression [60], popular active learning strategies use some notion of model output based uncertainty to select informative data [29,1,9,19,33]. There are two main challenges with existing approaches when they are adapted for vision-language tasks. First, they are founded on the assumption that no two output categories can be associated with any given input, assuming that the model's output space is a set of mutually exclusive labels. Unfortunately, this assumption is invalid for numerous vision-language tasks, such as image captioning and VQA, as one image or image-question pair can have multiple correct captions or answers [21,5,73]. This paraphrastic nature of language causes these measures to fail. Figure 1 illustrates an instance of this phenomenon where the question "what is the dog wearing?" can be answered by "Glasses", "Spectacles", "Eye-wear" and countless other paraphrases. Measures like entropy [64], margin [10], or least confidence [64] confound multiple correct answers to imply high uncertainty in the model's output.
Second, extending existing uncertainty functions for language generation models quickly renders them computationally intractable. Consider a VQA task with a model that can generate answers up to a length of N with each word belonging to a vocabulary of size |V |. The output space is of the order of N |V | . So, measuring uncertainty in model outputs grows exponentially with the vocabulary size. To circumvent intractability, Monte-Carlo simulations or similar approximations are often used to reduce the output space [60,61]. However, there is no clear extension of such approximations for discrete tokens, like the words in an answer or a caption.
In this paper, we propose a novel active learning uncertainty sampling paradigm that estimates uncertainty in an embedding space instead of an output probability space. Our proposed method enables the use of dropout-based Bayesian Monte-Carlo uncertainty estimation [18] for sequence generation. We claim that our strategy is effective, even in the presence of multiple correct outputs with discrete words. We showcase the efficacy of our approach in a VQA setting, as it represents a natural manifestation of the phenomenon of having multiple correct candidates for a given input, but insist that our solution is extendable to any language generation task, provided that a semantic space can be created. Instead of immediately generating answers, we impose a semantic structure on the hidden representation of existing VQA models. Our semantic space is designed to embed paraphrases close together. It allows us to abstract away from the discrete tokens generated by VQA models and estimate uncertainty in the embedding space instead.
We show the utility of our uncertainty sampling strategy for VQA generation on the Visual Genome v1.4 [39] and the VQA v2.0 [3] datasets in an active learning setting. We observe that our solution achieves state-of-the-art results in multiple language generation metrics compared to existing uncertainty sampling strategies [64,10,58], while being 5 times more cost-efficient on Visual Genome and 3 times more cost-efficient on VQA 2.0.

Related Work
We explore the field of active learning and comment on how existing uncertainty measurements fail in environments where we have multiple correct outputs. We place our work in context with the task of VQA, as we believe it represents a highly representative manifestation of these environments. Next, we dive into visual-semantic embedding, a practical solution to construct semantic spaces using visual priors. Finally, we explore Bayesian uncertainty measurement techniques.
Active learning. A typical active learning setting starts with an initial small set of labeled data D L and a large set of unlabeled data D U . Knowing that labeling data is costly and time consuming, the task is to minimize the number of samples to annotate from D U while maximizing performance on a given task that requires those labels [68]. Active learning strategies have been successfully applied to a wide number of machine learning tasks, including image recognition [29,59], information extraction [58,15,28,10], named entity recognition [64,22] and text categorization [40,24]. Active learning strategies have included uncertainty-based sampling [29,1,9] and information gain [25]. Others have introduced a theoretical dropout-based framework to measure uncertainty [19,33]. Similar solutions have been adapted to NLP tasks like Named Entity Recognition [64] and Neural Semantic Parsing by adding Gaussian noise to the network weights [13]. In this paper, we empirically show how previous uncertainty sampling strategies do not perform better than random sampling in a VQA setting and propose a novel strategy that uses Bayesian uncertainty in a semantically structured embedding space.
Visual question answering. VQA systems expect an input image and a natural language question and attempt to output the correct answer [3]. VQA has received a considerable amount of attention in recent years with the development of several datasets, proposed as benchmarks [3,50,27,20,39,57,78], and of various models [3,17,45,77,75,78,26,49,71]. To tackle the task, many proposed architectures encode and then merge the visual and textual information in order to classify answers [50,46,26]. Attention mechanisms have proved to be successful in this task [65,72,45,2]. Other work focuses on designing effective multi-modal feature fusion schemes [4,17,34]. While the performance has been encouraging, such approaches require a large amount of labelled data with a predefined, mutually exclusive set of answer categories. We explore a more realistic variant of VQA where models generate natural language answers, resulting in the generation of paraphrases.
Semantically structured embeddings. Our key insight lies in moving uncertainty estimation from the model's output space to a semantically structed internal embedding space. Since we specifically study vision-language tasks, we build a visual-semantic space that combines both visual and textual information in order to create a unified latent representation. This problem has been studied extensively in the last few years with work that involves jointly embedding images and text, at the word level [16,38,30,51,37,66], and sentence level [79,31,32,8,36,56]. Visually grounded text representations have been applied to different tasks, including caption generation [36,31], image retrieval [42], and visual question answering [50]. Our approach is inspired by previous work [32], where we use image regions with their associated captions and adapt a margin objective to build the semantic space. We use the semantic space to enable a new active learning strategy.
Uncertainty. As uncertainty estimation represents the most popular sampling strategy for active learning, we explore uncertainty estimation techniques developed for deep learning models. Even though Deep Learning systems are performant, they remain uninterpretable, poor at quantifying predictive uncertainty, and overconfident in their predictions. To mitigate this problem, Bayesian models have been proposed by placing a prior distribution over model weights [70,53,47]. Yet, even though the problem is simple to formulate, deriving a posterior distribution for Deep Bayesian Neural Networks (BNN) is intractable which makes Bayesian inference difficult [18]. Therefore, focus has shifted to approximating BNNs with variational inference. Bayesian modeling of stochastic processes introduced new techniques into the field such as sampling-based and stochastic variational inference [6,7].
On the other hand, Monte-Carlo Dropout [19,18] has empirically demonstrated comparable uncertainty estimation quality to variational inference [19]. A model trained with dropout can be used as a Bayesian model by making multiple predictions while sampling different dropout masks for each forward propagation. Estimating the posterior amounts to computing the mean and variance of the predictions. While previous work has showed promise in classification and regression tasks [74,18,60], such methods have not been extended to language generation. In fact, most language-based uncertainty work adapts classification-based uncertainty techniques to language tasks, transforming the problem from an open language problem to a constrained closed problem [64,22,40,24,13]. We propose a new sampling strategy that extends Monte-Carlo dropout uncertainty [18] to measure uncertainty in a semantic space.

Method
We design an uncertainty measurement for active learning even when questions have multiple correct answers. In this section, we formulate our overall active learning framework, and then describe our uncertainty measurement approach. Our approach depends on two components: a semantic space that structures the model's output representations and a denoiser that refines our representations to get an accurate dropout-based Bayesian uncertainty estimation.

Visual question answer generation
We specifically study the scenario with multiple correct outputs using the VQA task, which expects an imagequestion pair (q, i) as input and natural language answer a as output. The goal is to train a modelâ ∼ f (a|q, i), whereâ is the generated answer and f (·) is the trained VQA model. Since different sentences can have the same meaning, we define S a , the set of all sentences semantically similar to a. When evaluating the performance of the VQA model, we consider any answerâ ∈ S a as a correct candidate for an answer. For example, in Figure 1, the question "What is the dog wearing?" can have multiple correct answer candidates, including "Glasses", "Spectacles", etc. Traditionally, VQA classifies the answer in a mutually exclusive set of 1000 correct answers [3]. However, we study a more realistic variant of VQA where our model generates the answer in natural language. So, in all our experiments, we use the popular bottom-up attention model [2] and replace its classifier with a long short term memory network (LSTM) [67] to generate answers instead of classifying them. Although we report results with the bottom-up attention model, our results are consistent with other existing VQA models [75]. Almost all VQA models follow a traditional encoder-decoder architecture, where (q, i) are encoded to generate a hidden representation z. This representation is then decoded to produce the answerâ.

Active learning for VQA
We follow a traditional pool-based active learning setting [61,60]. The model is initially trained with a small bootstrapping training set {(q, i, a)} ∈ D train to produce a starting model f 0 . At every time step t, we receive a new pool P t of N question-image pairs (q, i) and must choose K pairs to annotate using an oracle. The K pairs are chosen using an uncertainty measurement {(q 1 , i 1 ) . . . (q k , i k )} ∼ U (P t ). Once the oracle answers the chosen questions, they are added to D train and the model is re-trained to f t .
The process of incorporating new annotations by choosing data points from a pool continues for T iterations. Our final model is f T . In our experiments, we compare and evaluate different uncertainty measurements by the performance of their resulting {f 0 , . . . , f T } models.  Figure 3: (a) We train existing VQA models, which almost always follow an encoder-decoder architecture, with an additional embedding loss that structures the output according to the visual-semantic space. (b) We define an embedding denoiser, by combining the VQA decoder with the pretrained visual-semantic encoder. The denoiser refines these representations to get an accurate uncertainty estimation.

Our contribution: Bayesian uncertainty as variance in semantic space
As discussed earlier, traditional uncertainty estimation techniques for language generation are based on the assumption that all outputs must be mutually exclusive, and thus, fail when paraphrases exist. Also since the existing methods take exponentially long to compute with respect the size of the vocabulary [60], these methods require approximations that lead to further errors [60].
Our solution tackles both of these challenges. First, we build a visual-semantic space that captures semantic language similarly and maps similar paraphrases close together, and second, uses Monte-Carlo dropout-based Bayesian uncertainty [18] with a denoiser to measure the model's uncertainty.
Monte-Carlo dropout [18] is an approximate inference approach to Bayesian Neural Networks that approximates a Gaussian Process by enforcing model dropout in both training and test time. At test time, due to the randomness induced by dropout, our model inference becomes stochastic which makes our output potentially variable. Applying a Monte-Carlo process with m simulations, equivalent to applying m forward passes each with a different dropout mask, results in a probabilistic distribution that can be used as a Bayesian interpretation of a neural network and allows the use of variance as an uncertainty measure.
Instead of dealing with intractable output spaces, we measure uncertainty within a visual-semantic space that is computationally independent of the vocabulary size. More specifically, given an input (q, i), we encode the pair into a hidden representation h using m forward passes, each with a randomly sampled dropout mask, resulting in m hidden representations. We assume that the representations embed into a visual-semantic space, a space where similar visual and language concepts lie close together. We measure uncertainty as the sum of variances across all the dimensions of the representation vectors: The (q, i) inputs with the K highest uncertainty scores are chosen and sent to an oracle for annotation. Intuitively, our measurement outputs a small uncertainty score when the m forward passes all produce representations that occupy a small volume in the embedding space, implying that all the answers the model produces are paraphrases or at least semantically similar. Similarly, our measurement outputs a large uncertainty score when the m representations occupy a large volume in the embedding space, implying that the answers produced by the model are different.

Training the visual-semantic space
Since we are demonstrating our method using a visionlanguage task, we build a visual-semantic space to structure language outputs. Similar to previous work, we design the embedding space using a dataset of image regions and their corresponding natural language captions [36,16,38,30,51,37,66,79,31,32,8,56]. Concretely, given a dataset of (i, c) pairs of images and captions, we first embed the images using a pretrained ResNet50 [23] h i = CN N (i) and the caption using a visual-semantic encoder h c = enc vs (c). Next, we use contrastive loss, often called triplet loss, to ensure that similar image regions and similar captions are embedded nearby: where (i j c j ) and (i k , c k ) are two distinct image-captions pairs and δ m is a hyper-parameter representing our loss margin. d(·, ·) is the euclidean distance function though any other distance function can also be used.
[·] + lower bounds all values to zero. To ensure that our embedding space does not collapse, we add a cross entropy loss H(·) which reconstructs the caption from its embedding. λ 1 , λ 2 are hyperparameters we optimize using a validation set of imagecaption pairs. Figure 2 visualizes this training process. Figure 4: Active learning strategies' performance on Visual Genome [39] measured using multiple language generation metrics. Performance of the initial model is not shown as it is the same for all strategies. We outperform all existing strategies. Additional metrics with exact numbers are included in the appendix.

Training VQA using visual semantic space
Finally, with the visual-semantic space trained, we can use it to train a VQA model and structure its outputs. Almost all VQA models proposed in Computer Vision follow an encoder-decoder architecture with an encoder which embeds the question-image pair h ∼ enc(q, i) and then a decoder that converts the embedding into an answer a ∼ dec(h) [75,2]. We train our model in a traditional way by imposing the following reconstruction cross entropy loss along with a visual-semantic embedding loss: where n is the size of the dataset and h aj = enc vs (a j ) is the projected answer embedding in the visual-semantic space. Figure 3(a) visualizing the training objective.

Denoising the representations for uncertainty estimation
Our approach depends on two central components: a semantic space that structures the model's output representations and a denoiser that refines our representations to get an accurate uncertainty estimation. As discussed previously, we use the L embed distance loss with respect to a visual-semantic space to semantically structure our VQA encoder's hidden representation. In practice, the resulting representation is an approximation of the reference visualsemantic space. We find that embeddings for some concepts often result in a noisy representations. In our dropout-based framework, we find that variance measured on these noisy representations does not guarantee an accurate model uncertainty estimation and results in random sampling behavior.
To refine the image-question pair representations to be similar to the answer representation, we introduce an embedding denoiser module. The denoiser consists of components we have already introduced during training: the pretrained VQA decoder and the pretrained visual-semantic encoder. As the VQA decoder is trained to map the imagequestion representations to text, it learns the inherent noise in the embedding space generated by the VQA encoder. By decoding these concepts to answer outputs and then using the visual-semantic encoder to re-project them back, we semantically correct the embeddings. Figure 3(b) visualizes the denoiser. As the denoiser does not expect the VQA model's hidden representation to be semantically structured, we might expect that it renders the training VQA models with L embed loss unnecessary. However, we find in practice that the semantic information provided by L embed combined with the denoiser, not only increases the quality of the samples but also improves model performance (see Section 4.3 for ablations).

Implementation Details
We implement our visual-semantic model by combining a Resnet50 [23] encoder for the image, with a bidirectional LSTM encoder for the question, with a hidden size of 512 and 2 layers. For our decoder, we use a 2-layer LSTM decoder with a hidden size of 512. We train our the model using 70k image region and corresponding description pairs from Visual Genome [39]. We also feed VQA imageanswer pairs from the VQA training set 10% of all iterations ensure that answer concepts are mixed together with the captions. The model is optimized using Adam [35] with Figure 5: Active learning strategies' performance on VQA [3] measured using multiple language generation metrics. Performance of the initial model is not shown as it is the same for all strategies. We outperform all existing strategies. Additional metrics with exact numbers are included in the appendix. a learning rate of 1e −3 with zero weight decay. We initialize our LSTM embeddings using GLoVe [55] and train the model for a total of 27k iterations with a batch size of 128.
When training our VQA model [2], we find that a dropout rate of 0.5 results in good performance along with reasonable hidden representations for our uncertainty framework. To estimate our uncertainty, we simulate m = 20 forward passes with dropout 0.5 to generate our embedding distribution. If we increase m beyond 20, we find that the change in the uncertainty value is negligible.

Experiments
In our experiments, we empirically demonstrate that Bayesian uncertainty, measured as variance in the visualsemantic space, is a better sampling strategy for active learning than existing strategies. Furthermore, we show that all existing uncertainty measures perform better when the output of the VQA model is structured with a visualsemantic space. Datasets. We study our approach using two existing VQA datasets: Visual Genome [39] and VQA [3]. Visual Genome contains 108k images densely annotated with scene graphs containing objects, attributes and relationships, as well as region descriptions and visual question answers. We use 70k of 5M randomly sampled region descriptions to create our visual-semantic space. For the VQA model, we use the 1.18M question answers. We split our data into a train set of 69k images with 853k question answers and a validation set of 27k images with 335k (q, a) pairs. VQA is a dataset of 123k MSCOCO [43] images with 658k visual question answers. We use the default train/val split with a training set of 83k images with 443k question answers and a test set of 40k images with 215k (q, a) pairs. Active learning setup. In order to showcase the advantage of usin g our uncertainty measurement, we test its efficiency against widely used uncertainty sampling strategies [10,58,64]. We setup our active learning pipeline as follows: we randomly initialize our bootstrapping set D train with size representing 5% of the original training set, and pretrain our model f 0 . f 0 is then used as the starting point for all our experiments. For each active learning iteration, we sample a pool P t with size representing 15% of the original dataset size. Using our sampling strategies, we update D train with the K best scoring data points such that we have an increase of 5% of the entire dataset in each iteration. We retrain our model using the resulting train set and repeat the procedure for 5 iterations, resulting in a final D train size of 30% of the original dataset. We train the visual-semantic space with the same amount of image-caption pairs as the question-answer pairs the initial f 0 uses.
Baselines. We compare our uncertainty measure against the most popular uncertainty-based strategies. All baselines use a traditional encoder-decoder architecture without imposing a visual-semantic embedding loss. Random sampling (Random) is a passive learning strategy that queries points following a uniformly random distribution. Least confidence (LC) [10] queries the instances for which the model has the least confidence in its most likely generated sequence: u(q, i) = 1 − P (a * |q, i), where P (a * |q, i) is the probability of the most confident model response. Margin Sampling (Margin) [58]: queries the instances with the least margin between the output probabilities for the two most likely generated answers of our model: u(q, i) = 1 + P (a 2 |q, i) − P (a 1 |q, i), where P (a j |Q, I) is the probability of the j th most confident model response, j ∈ {1, 2}. Maximum Entropy Sampling (Entropy) [64] queries the instances that maximize the entropy of our models output: We use a beam size of b = 5 to get our uncertainty score.
While other active learning strategies like information gain and density based methods exist, some require an explicit enumeration over the output space, which is intractable, while others are not scalable for large datasets and adapting them in language generation models is still an open research problem [60]. Evaluation. To evaluate answer quality, we use the standard automatic evaluation metrics, namely CIDEr [69], METEOR [12], and BERT (precision and recall) [76]. We report additional metrics, like Bert F1 [76], Bleu [54], ROUGE-L [41] and accuracy in the appendix.

Active learning performance
Setup. We report quantitative results from our active learning experiments in Table 4, as well as in Figure 5. To ensure that our results are not a product of noise, we run all experiments 5 times for each strategy on both datasets. Each run is accompanied with a different random seed to alter how the weights of the model are initialized and which pool of image-question pairs arrives at every step. We present the average scores from all 5 separate experiments.
Results. Our solution out-performs all traditional active learning strategies in all metrics by a large margin for both Visual Genome (see Table 4) and VQA 2.0 (see Table 5). The increase in performance is larger for Visual Genome, which has more unique answers than VQA 2.0, resulting in the model learning more sequences to express the same answer. On Visual Genome, our measure is over 5 times more cost efficient than most existing sampling strategies, performing at 20% of the annotation budget better than what other strategies manage after using the whole budget across all metrics. Similarly for VQA 2.0, we find that our solution is around 3 times more cost efficient, performing at 60% of the annotation budget better than what other strategies manage after using the whole budget across all metrics. This demonstrates our framework's ability to maximize performance when multiple correct answers are present.

What questions are sampled?
Setup. Next, we dive into what types of questions are sampled by Ours versus Margin, which is the best performing baseline. We report the sampling statistics per question category, i.e the percentage of a specific type of question the strategy chooses at every step. We also report the performance of the models on the test set for those specific categories. We only report CIDEr scores as we see the same trend across the other metrics. Results are reported in Table 1. We look at "what", "where", "who", "how" and "how many (counting)" questions in particular as they make up more than 90% of questions in Visual Genome. We also report how well the models perform at every iteration on the test split containing only that type of questions.
Results. Not only do we outperform Margin in cases where we sample more points for a specific type of question but also in cases where we sample less. In order to explain this behavior, we study samples from the 3 rd iteration of active learning, in which we sample significantly less "where" questions and more "what" questions and yet perform better than Margin on both types. We find that Margin samples 25.3% more answers of length ≥ 4 on "where" questions. We also look at the average length of answers in Visual Genome [39] and find that "what" questions have an average length of 1.9(±1.2) while "where" questions have an average length of 3.1(±1.5). Longer answers typically have a higher potential of having paraphrases. We find this assumption to be true qualitatively: Margin samples questions with ground truth answers like "next to the large brick buildings", "near the tall buildings" and "by the buildings" when it already has 6 paraphrases already in its training set. Furthermore, Ours samples 27% more new concepts on "what" question that might be less prone to redundancy since they are generally shorter answers (refer to Appendix for more details). We also see that our method samples very few counting questions as the model picks up on the dataset bias where the answer is usually "2" and therefore has a lower uncertainty.

Ablations
Setup. Our solution involves three components: a visualsemantic space (VS), applying dropout-based Bayesian uncertainty within that space (Baye), and utilizing a denoiser to make the measurements more accurate (Deno). The visual-semantic space, however, can be used during training for any VQA model to structure the outputs and doesn't need to specifically rely on dropout-based Bayesian uncertainty. So, we can add L embed as a loss and still utilize existing uncertainty functions. Here, we perform experiments by ablating the three components of our model as well as adding the visual-semantic space loss and reporting how they impact existing methods. Results. We report the impact of the structure provided by the visual-semantic space when still using existing uncertainty measure in Table 2. We notice that even though our proposed strategy outperforms the extended traditional strategies, all existing uncertainty methods achieve a considerable boost in performance over iterations. This leads to two important conclusions: (1) structuring the output space of problems that have multiple correct answers can improve uncertainty estimations by all existing measures and (2) measuring uncertainty in the embedding space instead of model outputs proves to be more robust with multiple correct answers. Our ablations also demonstrate the importance of the denoiser and visual-semantic space when using Bayesian uncertainty estimation. We show that Baye by itself performs just as well as the existing baselines but combined with Deno or VS increases its performance and together Baye+VS+Deno performs the best.

Discussion
While promising, it is important to note that our method does have a few limitations. First, constructing a semantic space requires additional data. In our case, the availabil-ity of Visual Genome's [39] region descriptions made constructing a visual-semantic space feasible. However, in different tasks where such data might not be already available, such an approach can increase the overall cost. Second, it is possible for semantic embeddings for some concepts to occupy a larger convex hull than others. Therefore, variance as a measure might overestimate uncertainty for concepts that cover a larger convex hull while underestimating uncertainty for those placed within a smaller hull. Measuring uncertainty with respect to the density of specific concepts is an open research question we leave to future work. Third, as we measure the variance on m sampled embeddings as our uncertainty, our solution needs multiple forward passes. This gives us a runtime and computational disadvantage over existing baselines. Even though these limitations exist, we consider them implementation limitations, left for further research work, and emphasize the higher level contribution of our paper which is the proposal of a new paradigm that investigates uncertainty as variance in semantic space.

Conclusion
Existing uncertainty sampling strategies in active learning are no better than random sampling for sequence generation tasks with multiple correct answers. We propose a novel uncertainty sampling paradigm that moves away from a probabilistic to an embedding-based uncertainty estimation and overcomes the paraphrastic nature of language. We evaluate our solution in an active learning setting for the VQA generation task and outperform existing sampling strategies. Our model samples fewer paraphrases and more novel concepts and is 5× more cost-efficient on Visual Genome and 3× on VQA 2.0.

Appendix
We provide more details on how to implement our VQA model. We then motivate our design decisions in developing a new active learning framework for tasks with multiple correct ouputs, by investigating when and how existing uncertainty measurements fail. Next, we explore the active learning framework baselines and provide a more comprehensive quantitative evaluation, as well as a qualitative analysis of our proposed sampling strategy.

Implementation details: VQA model
We implement a bottom-up, top-down attention [2] VQA generation model and use it for all of our active learning experiments. We use a question encoder similar to the LSTM encoder to the Visual-Semantic Encoder. We feed our bottom-up attention mechanism the output hidden state of the LSTM encoder for the question's embedding, along with a 100 region per corresponding image. We add a dropout layer (with p = 0.5) to our attention mechanism when we attend over the joint question-image representation and use the original architecture otherwise. We decode the model's answer using an LSTM decoder. We optimize the model with the same hyper-parameters as the Visual-Semantic model and train it for a total of 15 epochs.

Uncertainty estimation with multiple answers: CLEVR experiment
To motivate our design decisions in developing a new active learning framework for VQA with multiple correct answers, we investigate when and how existing uncertainty measurements fail. To systematically perform this evaluation without confounding factors like noise in real-world datasets, we use the synthetic CLEVR dataset [27]. Even though CLEVR only has questions with one correct answer, we modify the answers by introducing paraphrases. Our insights from experimenting on a modified-CLEVR motivate both the importance of moving to different uncertainty estimation for language generation models, as well as the specific uncertainty measure described in the next section.
Modifying CLEVR to include paraphrases CLEVR is a diagnostic dataset that tests a range of visual reasoning abilities, including Visual Question Answering. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. The dataset is composed of a train set of 700k (I, Q, A) triples with multiple answer categories including binary (yes / no), attributes, counts, objects and spatial relationships.
In a real world dataset, some answers will have multiple correct answers while other answers might only have one. To mimic such a setup, we modify CLEVR by taking a fraction of answer categories and replacing them with syn-  Figure 6: Even though the model is uncertain about the question on the left, existing uncertainty measurements assign it a lower uncertainty than the example on the right. When an input has multiple correct answers, existing measures overestimate uncertainty as they are unable to relate which outputs are paraphrases.
onyms. For example, we corrupt the answer 'yes' to seven different tokens − → {'yes', 'yeah', . . ., 'yup'}. Specifically, for every answer that is 'yes', we randomly modify it to one of the seven paraphrases. This is a conservative modification; in language, there are usually more than seven ways of expressing the same meaning.
Overestimation of uncertainty We train a state-of-the art VQA model [75], on both the modified-CLEVR dataset for 10 epochs with a learning rate of 4e − 3. Next, for all the data points in modified-CLEVR's validation set, we measure the model's uncertainty from the model's outputs and compare how uncertainty measurements differ for questions with multiple correct answers and those with only one.
We report uncertainty scores using entropy, which is the most popular uncertainty measurement used in active learning settings. We run similar experiments with other measures, like least confidence and margin, but omit them from our analysis here as they follow the same trend. Since exactly measuring entropy is intractable, we approximate entropy by decoding the answer with a beam size of 5 [64].
Qualitatively, Figure 6 demonstrates how the model's uncertainty scores differ for questions with one or multiple correct answers. On the left, we show an example of a question that the model is relatively unsure about. It assigns most of its weight to the correct answer ("large") but also assigns a sizable weight to an incorrect answer ("small"). However, the uncertainty associated with this question is lower than the example we show on the right. When multiple correct answers exist, the model fails to choose a clear answer and tends to assign similar weights to multiple an- Figure 7: As existing uncertainty measurements tend to overestimate uncertainty when an input has multiple correct answers, we end up sampling data points that the model already knows how to answer reducing active learning efficiency.
swers with the same meaning. Even though we can interpret such a result as the model learning that for such a question, multiple correct candidates exist, uncertainty measures are unaware of which answers are paraphrases and overestimate uncertainty. This scenario showcases a common failure case of active learning with existing uncertainty measurements, which would choose to collect more labels for the example with multiple correct answers, even though the model might benefit more from sampling the question on the left.
Quantitatively, we show how prevalent this problem is for all questions in the modified-CLEVR's validation set in Figure 7. Entropy assigns high uncertainty for all questions with multiple correct answers (shown in orange). And majority of these questions are assigned a higher uncertainty score than questions with a single correct answer (shown in green). Finally, if we correct the uncertainty scores by summing up the weights assigned to all the synonyms together, we find that the model is actually quite certain about a lot of these questions (shown in blue).
Our experiments with CLEVR allows us to conclude that we need an uncertainty measurement that can perform a similar correction, moving overestimated uncertainty scores (as shown in orange) to their correct values (as shown in blue). We need to design a mechanism which suggests which answers are semantically similar so that such a correction can be performed.

Active learning framework
We visualize the active learning framework described in the main paper using an algorithm in Algorithm 1. We also add additional notes and details about the baselines used below:

11:
Annotate SAMPLES using an oracle 12: Add new data to training set Dtrain ← Dtrain+ SAMPLES 13: Retrain f t using the updated D to get f t+1 14: end for Random Sampling (Passive Learning): We choose U following a uniformly random distribution from P .
where unif () returns k data points from P following a uniform distribution.
Least Confidence Sampling [10]: This approach queries the instances for which our model has the least condence in its most likely generated sequence. We choose U as the set with the top K points from P that the current VQA model has the least confidence in generating an answer for. We define our acquisition function as follows: where P (a * |Q, I) is the probability of the most confident model response.
Margin Sampling [58]: This approach queries the instances with the least margin between the output probabilities for the two most likely generated answers of our model. In this strategy, we use a beam size of 2 and choose U as the set with the top K points from P with the smallest margin. We define our acquisition function as follows: s(Q, I) = 1 + P (a 2 |Q, I) − P (a 1 |Q, I), where P (a i |Q, I) is the probability of the i th most confident model response, i ∈ {1, 2}.
Maximum Entropy Sampling [64]: This approach queries the instances that maximize the entropy of our models output. We choose U as the K data points with the highest prediction entropy. We define our acquisition function as follows: In order to measure entropy of our predictions in a VQA setting, we use a beam size of b to get b different predictions with probabilities P i (a|Q, I) for i = 1, 2, .., b.

More active learning performance:
The experiments we ran in the main paper can also be evaluated using additional language metrics like BLEU, ROUGE and BERT F1. We include those metrics here. They follow the same trend as the other metrics. We also report model accuracy.
Results on Visual Genome are shown in Table 3 and results on VQA 2.0 are shown in Table 4.

Qualitative analysis between Margin vs Ours
As discussed in section 5.2, we qualitatively study the behavior of both Margin, as as our solution on different types of questions. We find that our solution out-performs Margin in all types of questions by a considerable margin. We also study the percentage of sampled points with respect to performance for each type of question. We find that our solution not only performs better when we sample more points but also in cases where we sample less.
In order to understand this behavior, we study samples from the 3 rd iteration of active learning, where we sample significantly less "where" questions and more "what" questions and yet perform better than Margin on both types. We investigate the distribution of answer length for samples in both types of questions. Figure 8 and 9 show the sampled distributions. We find that Margin samples 25.3% more answers of length ≥ 4 on "where" questions (3271 vs 2647).
We also look at the average length of answers in Visual Genome [39] and find that "What" questions have an average length of 1.9(±1.2) while "where" questions have an average length of 3.1(±1.5). Longer answers typically have a higher potential of having paraphrases. We find this assumption to be true qualitatively: an example of the redundancy of Margin is shown in Table 5. We look at sampled points containing the word "building". We find that Margin heavily re-samples concepts that are already avail- able or have paraphrases in the training set. We also report the corresponding samples for Ours in Table 6 and find that we sample less paraphrases and are less redundant.   Table 4: We report our models efficacy with multiple metrics. We use language modeling metrics to measure its capability to generate answers similar to the ground truth as we progress through the AL process. Scores are multiplied by 100 to show more significant digits.

Margin 3 rd iteration samples
Paraphrases already in the Training Set Points between the buildings between the red brick buildings over the street and between buildings between the buildings in between two buildings between the pear and lemon buildings overhead between the buildings between two buildings between buildings hanging between buildings in between buildings above the buildings on buildings on the buildings on the buildings behind the vehicles above buildings on buildings above the train and buildings on the buildings on top of buildings on the buildings in the background on the buildings above the city buildings above the buildings hanging over buildings in front of the buildings in front of buildings parked in front of the buildings in front of the buildings and tower in front of buildings in front of the buildings near the buildings near buildings next to the large brick buildings by the buildings in the street , near buildings next to the buildings by the tall buildings near the buildings beside the buildings by the trees and buildings near buildings close to buildings next to the buildings behind buildings behind the buildings behind the buildings behind the other buildings behind all the buildings Table 5: We look at all the sampled phrases containing the word "building" with Margin. We realize that we end up heavily re-sampling concepts that are already available in our training set. This shows the redundancy of this strategy and explains why sampling more points leads to worse performance.  Table 6: We look at all the sampled phrases containing the word "building" with Margin. We find that compared to Margin, we cover the same range of concepts while being less redundant.