It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C’s). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et al., 2017 and the BiVCCA method of Wang et al., 2016) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset (Liu et al., 2015). Consider the following two-party communication game: a speaker thinks of a visual concept C, such as "men with black hair", and then generates a description y of this concept, which she sends to a listener; the listener interprets the description y, by creating an internal representation z, which captures its "meaning". We can think of z as representing a set of "mental images" which depict the concept C. To test whether the listener has correctly "understood" the concept, we ask him to draw a set of real images S = {x s : s = 1 : S}, which depict the concept C. He then sends these back to the speaker, who checks to see if the images correctly match the concept C. We call this process visually grounded imagination.In this paper, we represent concept descriptions in terms of a fixed length vector of discrete attributes A. This allows us to specify an exponentially large set of concepts using a compact, combinatorial representation. In particular, by specifying different subsets of attributes, we can generate concepts at different levels of granularity or abstraction. We can arrange these concepts into a compositional abstraction hierarchy, as shown in Figure 1 . This is a directed acyclic graph (DAG) in which nodes represent concepts, and an edge from a node to its parent is added whenever we drop one of the attributes from the child's concept definition. Note that we dont make any assumptions about the order in which the attributes are dropped (that is, dropping the attribute "smiling" is just as valid as dropping "female" in Figure 1 ). Thus, the tree shown in the figure is just a subset extracted from the full DAG of concepts, shown for illustration purposes.We can describe a concept by creating the attribute vector y O , in which we only specify the value of the attributes in the subset O ⊆ A; the remaining attributes are unspecified, and are assumed to take all possible legal values. For example, consider the following concepts, in order of increasing abstraction: C msb = (male, smiling, blackhair), C * sb = ( * , smiling, blackhair), and C * * b = ( * , * , blackhair), where the attributes are gender, smiling or not, and hair color, and * represents "don't care". A good model should be able to generate images from different levels of the abstraction hierarchy, as shown in Figure 1 . (This is in contrast to most prior work on conditional generative models of images, which assume that all attributes are fully specified, which corresponds to sampling only from leaf nodes in the hierarchy.) Figure 1 : A compositional abstraction hierarchy for faces, derived from 3 attributes: hair color, smiling or not, and gender. We show a set of sample images generated by our model, when trained on CelebA, for different nodes in this hierarchy.In Section 2, we show how we can extend the variational autoencoder (VAE) framework of BID15 to create models which can perform this task. The first extension is to modify the model to the "multi-modal" setting where we have both an image, x, and an attribute vector, y. More precisely, we assume a joint generative model of the form p(x, y, z) = p(z)p(x|z)p(y|z ), where p(z) is the prior over the latent variable z, p(x|z) is our image decoder, and p(y|z) is our description decoder. We additionally assume that the description decoder factorizes over the specified attributes in the description, so p(y O |z) = k∈O p(y k |z).We further extend the VAE by devising a novel objective function, which we call the TELBO, for training the model from paired data, D = {(x n , y n )}. However, at test time, we will allow unpaired data (either just a description or just an image). Hence we fit three inference networks: q(z|x, y), q(z|x) and q(z|y ). This way we can embed an image or a description into the same shared latent space (using q(z|x) and q(z|y), respectively); this lets us "translate" images into descriptions or vice versa, by computing p(y|x) = dz p(y|z)q(z|x) and p(x|y) = dz p(x|z)q(z|y).To handle abstract concepts (i.e., partially observed attribute vectors), we use a method based on the product of experts (POE) BID8 . In particular, our inference network for attributes has the form q(z|y O ) ∝ p(z) k∈O q(z|y k ). If no attributes are specified, the posterior is equal to the prior. As we condition on more attributes, the posterior becomes narrower, which corresponds to specifying a more precise concept. This enables us to generate a more diverse set of images to represent abstract concepts, and a less diverse set of images to represent concrete concepts, as we show below.Section 3 discusses how to evaluate the performance of our method in an objective way. Specifically, we first "ground" the description by generating a set of images, S(y O ) = {x s ∼ p(x|y O ) : s = 1 : S}. We then check that all the sampled images in S(y O ) are consistent with the specified attributes y O (we call this correctness). We also check that the set of images "spans" the extension of the concept, by exhibiting suitable diversity (c.f. BID36 ). Concretely, we check that the attributes that were not specified (e.g., gender in C * sb above) vary across the different images; we call this coverage. Finally, we want the set of images to have high correctness and coverage even if the concept y O has a combination of attribute values that have not been seen in training. For example, if we train on C msb = (male, smiling, blackhair), and C f nb = (female, notsmiling, blackhair), we should be able to test on C mnb = (male, notsmiling, blackhair), and C f sb = (female, smiling, blackhair). We will call this property compositionality . Being able to generate plausible images in response to truly compositionally novel queries is the essence of imagination. Together, we call these criteria the 3 C's of visual imagination.Section 5 reports experimental results on two different datasets. The first dataset is a modified version of MNIST, which we call MNIST-with-attributes (or MNIST-A), in which we "render" modified versions of a single MNIST digit on a 64x64 canvas, varying its location, orientation and size. The second dataset is CelebA BID16 , which consists of over 200k face images, annotated with 40 binary attributes. We show that our method outperforms previous methods on these datasets.The contributions of this paper are threefold. First, we present a novel extension to VAEs in the multimodal setting, introducing a principled new training objective (the TELBO), and deriving an interpretation of a previously proposed objective (JMVAE) BID31 as a valid alternative in Appendix A.1. Second, we present a novel way to handle missing data in inference networks based on a product of experts. Third, we present novel criteria (the 3 C's) for evaluating conditional generative models of images, that extends prior work by considering the notion of visual abstraction and imagination. We have shown how to create generative models which can "imagine" compositionally novel concrete and abstract visual concepts. In the future we would like to explore richer forms of description, beyond attribute vectors, such as natural language text, as well as compositional descriptions of scenes, which will require dealing with a variable number of objects.
A VAE-variant which can create diverse images corresponding to novel concrete or abstract "concepts" described using attribute vectors.
{"src": "scitldr-AIC_train", "paper_id": "HkCsm6lRb"}