Generative Adversarial Networks (GANs), when trained on large datasets with diverse modes, are known to produce conflated images which do not distinctly belong to any of the modes. We hypothesize that this problem occurs due to the interaction between two facts: (1) For datasets with large variety, it is likely that the modes lie on separate manifolds. (2) The generator (G) is formulated as a continuous function, and the input noise is derived from a connected set, due to which G's output is a connected set. If G covers all modes, then there must be some portion of G's output which connects them. This corresponds to undesirable, conflated images. We develop theoretical arguments to support these intuitions. We propose a novel method to break the second assumption via learnable discontinuities in the latent noise space. Equivalently, it can be viewed as training several generators, thus creating discontinuities in the G function. We also augment the GAN formulation with a classifier C that predicts which noise partition/generator produced the output images, encouraging diversity between each partition/generator. We experiment on MNIST, celebA, STL-10, and a difficult dataset with clearly distinct modes, and show that the noise partitions correspond to different modes of the data distribution, and produce images of superior quality. Generative Adversarial Networks BID8 are powerful generative models that have enjoyed significant attention from the research community in the past few years. Despite several successes, the original formulation for GANs is widely acknowledged to be notoriously difficult to train due to instability issues. In particular, GANs face the mode collapse problem, where the generator resorts to generating a handful of samples which are assigned high probability by the discriminator. Several methods have been introduced to fix the mode collapse problem. BID3 , , BID9 , BID15 , BID21 Despite improvements, state-of-art GANs still fail to generate meaningful samples on diverse and complex datasets such as ImageNet BID5 . GANs trained on such datasets produce conflated images which do not distinctly belong to any of the modes present in the dataset.We hypothesize that this problem occurs due to the continuous nature of the generator function, along with the connectedness of the latent noise space, due to which the output set of the generator is also connected. This poses a problem when dealing with complex real life datasets with varied modes. Strong empirical and theoretical evidence suggests that real life images lie on lowdimensional manifolds BID17 . It is highly probable that distinct modes (say bedroom images and human face images) lie on disjoint manifolds. If we assume that the generator does not suffer from the mode dropping problem, it must cover all these manifolds in its output. However, the output set being connected, must contain parts which do not belong to any of the manifolds, but simply join them.We refer to such parts of the output as tunnels, since they connect otherwise disjoint manifolds. Tunnels do not resemble any of the images in the dataset, and are not similar to any of the modes. They correspond to the conflated images produced by the generator, and are undesirable. By this reasoning, we suggest that GANs with continuous generators and connected latent noise sets must suffer either from a certain degree of mode dropping or from producing conflated, garbled outputs when trained on complex and varied datasets like ImageNet.We develop methods that allow GANs to cover disjoint manifolds without the use of tunnels, while not compromising on mode coverage. Our approach is to create learnable discontinuities in the latent noise space. This is done by learning N different linear mappings (partitions) in the input layer of the generator. A noise vector (sampled from the standard normal distribution), gets mapped to N different vectors by the input layer, and the rest of the processing remains the same as in standard generators. The output set of each mapping is a connected set, but the union of the N output sets could potentially be disconnected. Thus, we break the connectedness assumption leading to the existence of tunnels. To facilitate learning distinct modes by each partition, we introduce a classifier that predicts which partition created a given input. We modify the loss functions to adjust for this change.We experiment on standard datasets: MNIST (LeCun et al., 2010) , celebA BID14 , STL-10 (a subset of ImageNet) BID4 , and a tough artificial dataset with very distinct modes -an equal mixture of LSUN BID22 bedrooms and celebA, to verify the efficacy of our method. We compare our results with one of the best performing GAN variant BID9 , and show an improvement in quality.The major contributions of the paper are summarized below:1. We identify a key problem with training GANs on large & diverse datasets, and provide intuition to explain its cause 2. We develop theoretical analyses to support and introduce rigor in the intuitions provided 3. Motivated by these analyses, we introduce a novel GAN setup to alleviate the problem 4. We experiment on a variety of standard datasets and report improvements over state-of-art formulations 2 RELATED WORK BID8 formulated GAN as a minimax game between two neural networks: generator G θ and discriminator D φ . G θ takes a random noise vector z as input and generates sample G θ (z), while D φ identifies whether input sample is real or generated by the generator G θ . Both G θ and D φ play a two-player minimax game with value function V (G, D): DISPLAYFORM0 where P r (x) is the real data distribution, and P(z) is arbitrary noise distribution (typically uniform or normal distribution). In practice, training GANs using above formulation is highly unstable and requires careful balance of generator and discriminator updates. BID19 proposed a class of CNNs called DCGANs (Deep Convolutional GANs) with certain architectural specifications, and demonstrated better image quality than non-convolutional vanilla GAN architecture. BID6 used Laplacian pyramid framework for the generator, where a separate generative convnet model is trained using GAN approach at each level of pyramid, to generate images in coarse-to-fine fashion.Despite better architectures, GANs suffered from problems like unstable training, vanishing gradients of generator, mode collapse. BID20 proposed several heuristics such as feature matching, minibatch discrimination, historical averaging, label smoothing, primarily to stabilize GAN training. BID3 observed that GAN training can push probability mass in wrong direction, hence are prone to missing modes of data. They proposed regularization techniques to stabilize GAN training and alleviate mode missing problem by fair distribution of probability mass across modes of the real data distribution. BID1 provided theoretical analysis of training dynamics of GANs, and problems including instability and saturation. They revealed fundamental problems with original GAN formulation and provided directions towards solving them.Several papers proposed alternative objective function of generator and discriminator. , BID9 proposed new loss function which approximately minimizes Wasserstein distance between real and generated data distribution instead of Jensen Shannon Divergence. They claim their formulation does not require careful balance between generator and discriminator updates, thus lead to stable training without saturating the gradients. They observed no evidence of mode collapse in their experiments. BID15 used squared-loss instead of log-loss in original formulation, which provides generator with better non-vanishing gradients. BID23 view discriminator as an energy function making it possible to use additional loss functions other than logistic output binary classifier, which was found to stabilize GAN training. BID21 propose to train discriminator based on linear separability between hidden representation of real and generated samples and train generator based on decision hyperplanes between hidden representations computed using Linear Discriminant Analysis.For labelled datasets, BID16 , BID18 employed label conditioning in both generator and discriminator to generate discriminable and diverse samples across classes. While this helps produce better samples for complex datasets, it requires the presence of labelled data. In this paper we propose methods to improve performance of GANs on complex datasets without making use of labels. We highlighted a major problem in training GANs on complex image datasets and introduced theoretical analysis for the problem of generation of unrealistic, conflated images in such cases. We proposed the addition of discontinuity in latent noise space of the generator for covering disjoint and diverse modes of the data distribution, and augmented the loss functions to encourage diversity. We showed improvements over existing models without much hyperparameter tuning.In future, we hope to perform an extensive exploration of the search space to obtain a set of hyperparameters along with better methods to introduce discontinuities in the generator that perform well on a variety of datasets, while significantly improving image quality.
We introduce theory to explain the failure of GANs on complex datasets and propose a solution to fix it.
{"src": "scitldr-AIC_train", "paper_id": "HyDMX0l0Z"}