Generative adversarial networks (GANs) learn to map samples from a noise distribution to a chosen data distribution. Recent work has demonstrated that GANs are consequently sensitive to, and limited by, the shape of the noise distribution. For example, a single generator struggles to map continuous noise (e.g. a uniform distribution) to discontinuous output (e.g. separate Gaussians) or complex output (e.g. intersecting parabolas). We address this problem by learning to generate from multiple models such that the generator's output is actually the combination of several distinct networks. We contribute a novel formulation of multi-generator models where we learn a prior over the generators conditioned on the noise, parameterized by a neural network. Thus, this network not only learns the optimal rate to sample from each generator but also optimally shapes the noise received by each generator. The resulting Noise Prior GAN (NPGAN) achieves expressivity and flexibility that surpasses both single generator models and previous multi-generator models. Learning generative models of high-dimensional data is of perpetual interest, as its wide suite of applications include synthesizing conversations, creating artwork, or designing biological agents (Bollepalli et al., 2017; Tan et al., 2017; Blaschke et al., 2018) . Deep models, especially generative adversarial networks (GANs), have significantly improved the state of the art at modeling these complex distributions, thus encouraging further research (Goodfellow et al., 2014) . Whether implicitly or explicitly, works that use GANs make a crucial modeling decision known as the manifold assumption (Zhu et al., 2016; Schlegl et al., 2017; Reed et al., 2016) . This is the assumption that high-dimensional data lies on a single low-dimensional manifold which smoothly varies and where local Euclidean distances in the low-dimensional space correspond to complex transformations in the high-dimensional space. While generally true in many applications, this assumption does not always hold (Khayatkhoei et al., 2018) . For example, recent work has emphasized situations where the data lies not on one single manifold, but on multiple, disconnected manifolds (Khayatkhoei et al., 2018; Gurumurthy et al., 2017; Hoang et al., 2018) . In this case, GANs must attempt to learn a continuous cover of the multiple manifolds, which inevitably leads to the generation of off-manifold points which lie in between (Kelley, 2017) . The generator tries to minimize the number of these off-manifold points, and thus they are generally just a small fraction of the total generated distribution. As such, they barely affect the typical GAN evaluation measures (like Inception and FID scores for images), which measure the quality of the generated distribution as a whole. Thus, this problem is usually ignored, as other aspects are prioritized. However, in some applications, the presence of these bad outliers is more catastrophic than slight imperfections in modeling the most dense regions of the space. For example, consider the goal of an artificial agent acting indistinguishably from a human: the famous Turing Test. Incorrectly modeling sentence density by using a given sentence structure 60% of the time instead of 40% of the time is relatively harmless. However, generating a single gibberish sentence will give away the identity of the artificial agent. Moreover, there are serious concerns about the implications this has for proofs of GAN convergence (Mescheder et al., 2018) . These works address the problem of disconnected manifolds by Figure 1 : The Noise-Prior GAN (NPGAN) architecture. Unlike previous work, the NP network learns a prior over the generators conditioned on the noise distribution z. This allows it to both control the sampling frequency of the generators and shape the input appropriate to each one, in an end-to-end differentiable framework. simultaneously training multiple generators and using established regularizations to coax them into dividing up the space and learning separate manifolds. Methods for getting multiple generators to generate disconnected manifolds can be divided into two categories: (i) imposing information theoretic losses to encourage output from different generators to be distinguishable (Khayatkhoei et al., 2018; Hoang et al., 2018) (ii) changing the initial noise distribution to be disconnected (Gurumurthy et al., 2017) . Our approach falls into the second category. Previous efforts to change the noise distribution to handle disconnectedness has exclusively taken the form of sampling from a mixture of Gaussians rather than the typical single Gaussian (with sampling fixed and uniform over the mixture). Our approach differs significantly from those previously. We use multiple generators as before, but instead of dividing up the noise space into factorized Gaussians and sending one to each generator, we let an additional neural network determine how best to divide up the noise space and dispatch it to each generator. This network learns a prior over the generators, conditioned on the noise space. Thus, we call our additional third network a noise-prior (NP) network. Previous methods have modeled the data with noise z and generators We instead propose a framework to incorporate a richer p(G i |z) into the generator. This framework is entirely differentiable, allowing us to optimize the NP network along with the generators during training. We note that with this strategy, we significantly increase the expressivity of each generator over the previous disconnected manifold models. By dividing up the space into four slices s i and sending s 1 , s 3 to the first generator and s 2 , s 4 to the second generator, we can generate four disconnected manifolds with just two generators. Previous work would have to devote precisely four generators to this task, with degradation in performance if fewer or more generators are chosen for the hyperparameter. Here, the prior network learns to divide the noise space appropriately for whatever number of generators is chosen, and is thus more expressive as well as more robust than previous models. Moreover, much existing work has exclusively framed the problem as, and tailored solutions for, the disconnected manifold problem. Our approach is more generalized, addressing any misspecification between noise distribution and the target distribution. This means that our approach does not become redundant or unnecessary in the case of single complex manifolds, for example. Our contributions can be summarized as: 1. We introduce the first multi-generator ensemble to learn a prior over the noise space, using a novel soft, differentiable loss formulation. 2. We present a multi-generator method that can learn to sample generators in proportion to the relative density of multiple manifolds. 3. We show how our model not only improves performance on disconnected manifolds, but also on complex-but-connected manifolds, which are more likely to arise in real situations. We introduced a novel formulation of multiple-generator models with a prior over the generators, conditioned on the noise input. This results in improved expressivity and flexibility by shaping each generator's input specifically to best perform that generator's task. In this section, we elaborate on the CIFAR experiment from the main text. We use a more complicated architecture here with spectral normalization, self-attention, and ResNet connections, per the best achieving models to-date. We experimented using two, three, four, and five generators in the NPGAN architecture. Figure A .1 shows images generated by the NPGAN with each number of generators. With just two generators, each one creates a wide diversity of images. On the other hand, when increasing the number of generators, each one more homogeneous. For example, in the two generator model, one of them creates dogs, cars, and frogs, while in the five-generator model each generator has specialized to just birds in the sky or just cars. Qualitatively, the noise prior is obviously learning a sensible split of the data across generators and each generator is outputting quality images. However, when comparing the two-generator, threegenerator, four-generator, and five-generator versions of NPGAN to the baseline one-generator of the same model, we do not observe any improvement in FID score. This is unsurprising for the reasons mentioned in the main text. The FID scores treat all points equally across a generated dataset, and thus will be most strongly influenced by where the most points are. A relatively small number of outliers barely register by this metric. Even current state-of-the-art image generation on CIFAR10 is no where close to perfectly modeling the data. When GANs are able to perfectly model the dataset except for trailing outliers between modes, we expect the NPGAN's improvements to be visible in FID scores on this dataset. Until then, the detection of a few bad outliers needs to be done with other evaluation techniques on this dataset. With this caveat, we note that we could achieve an FID score of 26.4 with our NPGAN, compared to 25.8 with our code and one generator, which demonstrates that the NPGAN can scale to stateof-the-art architecture without suffering in quality. The NPGAN is robust to a connected dataset while simultaneously being able to automatically solve the problems of a disconnected dataset. Furthermore, this motivated the creation of our new outlier manifold distance metric, designed to be more sensitive to the creation of outliers than the FID score. Using this metric, we see NPGAN outperform all other models. Relation to Machine Teaching In (Zhu, 2013) , an analogous question is posed: if a teacher network knows the function its student network is supposed to learn, what are the optimal training points to teach it as efficiently as possible? For students following a Bayesian learning approach, this is thought of as finding the best data points D to make the desired model θ * , or minimizing with respect to D: −log(p(θ * |D)) . In our framework, the teacher network NP does not know the function its students should learn ahead-of-time, because this target is changing continually as the discriminator improves simultaneously. Nevertheless, the NP network is still learning to form the optimal curriculum for each individual student such that the collection of students best models the target function given the current parameters of the discriminator. Relation to knowledge distillation Our NP network also has links to the field of knowledge distillation (Kim & Rush, 2016; Chen et al., 2017; Furlanello et al., 2018; Wang et al., 2018) , where a teacher network is trying to compress or distill the knowledge it has about a particular distribution into one or several (Hinton et al., 2015) smaller models. In the case of multiple smaller models, the teacher can be thought of as a generalist whose job it is to find the right specialist for a specific problem.
A multi-generator GAN framework with an additional network to learn a prior over the input noise.
{"src": "scitldr-AIC_train", "paper_id": "HJlISCEKvB"}