Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work we propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition. This provides a way to efficiently learn a more accurate generative model of real-world images, and serves as an initial step towards learning corresponding object representations. We evaluate our approach on several multi-object image datasets, and find that the generator learns to identify and disentangle information corresponding to different objects at a representational level. A human study reveals that the resulting generative model is better at generating images that are more faithful to the reference distribution. Generative modelling approaches to representation learning seek to recover the process with which the observed data was generated. It is postulated that knowledge about the generative process exposes important factors of variation in the environment (captured in terms of latent variables) that may subsequently be obtained using an appropriate posterior inference procedure. Therefore, the structure of the generative model is critical in learning corresponding representations.Deep generative models of images rely on the expressiveness of neural networks to learn the generative process directly from data BID11 BID24 BID38 . Their structure is determined by the inductive bias of the neural network, which steers it to organize its computation in a way that allows salient features to be recovered and ultimately captured in a representation BID6 BID7 BID24 . Recently, it has been shown that independent factors of variation, such as pose and lighting of human faces may be recovered in this way BID5 .A promising but under-explored inductive bias in deep generative models of images is compositionality at the representational level of objects, which accounts for the compositional nature of the visual world and our perception thereof BID3 BID37 . It allows a generative model to describe a scene as a composition of objects (entities), thereby disentangling visual information in the scene that can be processed largely independent of one another. It provides a means to efficiently learn a more accurate generative model of real-world images, and by explicitly Figure 1: A scene (right) is generated as a composition of objects and background. considering objects at a representational level, it serves as an important first step in recovering corresponding object representations.In this work we investigate object compositionality for Generative Adversarial Networks (GANs; BID11 ), and present a general mechanism that allows one to incorporate corresponding structure in the generator. Starting from strong independence assumptions about the objects in images, we propose two extensions that provide a means to incorporate dependencies among objects and background. In order to efficiently represent and process multiple objects with neural networks, we must account for the binding problem that arises when superimposing multiple distributed representations BID18 . Following prior work, we consider different representational slots for each object BID13 BID34 , and a relational mechanism that preserves this separation accordingly .We evaluate our approach 1 on several multi-object image datasets, including three variations of Multi-MNIST, a multi-object variation of CIFAR10, and CLEVR. In particular the latter two mark a significant improvement in terms of complexity, compared to datasets that have been considered in prior work on unconditional multi-object image generation and multi-object representation learning.In our experiments we find that our generative model learns about the individual objects and the background of a scene, without prior access to this information. By disentangling this information at a representational level, it generates novel scenes efficiently through composing individual objects and background, as can be seen in Figure 1 . As a quantitative experiment we compare to a strong baseline of popular GANs (Wasserstein and Non-saturating) with recent state-of-the-art techniques (Spectral Normalization, Gradient Penalty) optimized over multiple runs. A human study reveals that the proposed generative model outperforms this baseline in generating better images that are more faithful to the reference distribution. The experimental results confirm that the proposed structure is beneficial in generating images of multiple objects, and is utilized according to our own intuitions. In order to benefit maximally from this structure it is desirable to be able to accurately estimate the (minimum) number of objects in the environment in advance. This task is ill-posed as it relies on a precise definition of "object" that is generally not available. In our experiments on CLEVR we encounter a similar situation in which the number of components does not suffice the potentially large number of objects in the environment.Here we find that it does not render the proposed structure useless, but instead each component considers "primitives" that correspond to multiple objects.One concern is in being able to accurately determine foreground, and background when combining the outputs of the object generators using alpha compositing. On CLEVR we observe cases in which objects appear to be flying, which is the result of being unable to route the information content of a "foreground" object to the corresponding "foreground" generator as induced by the fixed order in which images are composed. Although in principle the relational mechanism may account for this distinction, a more explicit mechanism may be preferred BID31 .We found that the pre-trained Inception embedding is not conclusive in reasoning about the validity of multi-object datasets. Similarly , the discriminator may have difficulties in accurately judging images from real / fake without additional structure. Ideally we would have a discriminator evaluate the correctness of each object individually, as well as the image as a whole. The use of a patch discriminator BID20 , together with the alpha channel of each object generator to provide a segmentation, may serve a starting point in pursuing this direction. We have argued for the importance of compositionality at the representational level of objects in deep generative models of images, and demonstrated how corresponding structure may be incorporated in the generator of a GAN. On a benchmark of multi-object datasets we have shown that the proposed generative model learns about individual objects and background in the process of synthesizing samples. A human study revealed that this leads to a better generative model of images. We are hopeful that in disentangling information corresponding to different objects at a representational level these may ultimately be recovered. Hence, we believe that this work is an important contribution towards learning object representations of complex real-world images without any supervision.A EXPERIMENT RESULTS The generator and discriminator neural network architectures in all our experiments are based on DCGAN BID35 .
We propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition
{"src": "scitldr-AIC_train", "paper_id": "BJgEjiRqYX"}