Generative Adversarial Networks (GAN) (, 2014) are an effective method for training generative models of complex data such as natural images. However, they are notoriously hard to train and can suffer from the problem of missing modes where the model is not able to produce examples in certain regions of the space. We propose an iterative procedure, called AdaGAN, where at every step we add a new component into a mixture model by running a GAN algorithm on a reweighted sample. This is inspired by boosting algorithms, where many potentially weak individual predictors are greedily aggregated to form a strong composite predictor. We prove that such an incremental procedure leads to convergence to the true distribution in a finite number of steps if each step is optimal, and convergence at an exponential rate otherwise. We also illustrate experimentally that this procedure addresses the problem of missing modes.
The paper is organized as follows. In Section we present our main theoretical results regarding optimization of mixture models under general @math -divergences. In particular we show that it is possible to build an optimal mixture in an incremental fashion, where each additional component is obtained by applying a GAN-style procedure with a reweighted distribution. In Section we show that if the GAN optimization at each step is perfect, the process converges to the true data distribution at exponential rate (or even in a , for which we provide a necessary and sufficient condition). Then we show in Section that imperfect GAN solutions still lead to the exponential rate of convergence under certain weak learnability'' conditions. These results naturally lead us to a new boosting-style iterative procedure for constructing generative models, which is combined with GAN in , resulting in a new algorithm called . Finally, we report initial empirical results in Section , where we compare AdaGAN with several benchmarks, including original GAN, uniform mixture of multiple independently trained GANs, and iterative procedure of : @cite_19 .
@cite_19
2560169576
Ensembles are a popular way to improve results of discriminative CNNs. The combination of several networks trained starting from different initializations improves results significantly. In this paper we investigate the usage of ensembles of GANs. The specific nature of GANs opens up several new ways to construct ensembles. The first one is based on the fact that in the minimax game which is played to optimize the GAN objective the generator network keeps on changing even after the network can be considered optimal. As such ensembles of GANs can be constructed based on the same network initialization but just taking models which have different amount of iterations. These so-called self ensembles are much faster to train than traditional ensembles. The second method, called cascade GANs, redirects part of the training data which is badly modeled by the first GAN to another GAN. In experiments on the CIFAR10 dataset we show that ensembles of GANs obtain model probability distributions which better model the data distribution. In addition, we show that these improved results can be obtained at little additional computational cost.