Generative Adversarial Networks (GANs) have been shown to produce realistically looking synthetic images with remarkable success, yet their performance seems less impressive when the training set is highly diverse. In order to provide a better fit to the target data distribution when the dataset includes many different classes, we propose a variant of the basic GAN model, a Multi-Modal Gaussian-Mixture GAN (GM-GAN), where the probability distribution over the latent space is a mixture of Gaussians. We also propose a supervised variant which is capable of conditional sample synthesis. In order to evaluate the model's performance, we propose a new scoring method which separately takes into account two (typically conflicting) measures - diversity vs. quality of the generated data. Through a series of experiments, using both synthetic and real-world datasets, we quantitatively show that GM-GANs outperform baselines, both when evaluated using the commonly used Inception Score, and when evaluated using our own alternative scoring method. In addition, we qualitatively demonstrate how the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. We show how this phenomenon can be exploited for the task of unsupervised clustering, and provide quantitative evaluation showing the superiority of our method for the unsupervised clustering of image datasets. Finally, we demonstrate a feature which further sets our model apart from other GAN models: the option to control the quality-diversity trade-off by altering, post-training, the probability distribution of the latent space. This allows one to sample higher quality and lower diversity samples, or vice versa, according to one's needs. Generative models have long been an important and active field of research in machine-learning. Generative Adversarial Networks BID6 include a family of methods for learning generative models where the computational approach is based on game theory. The goal of a GAN is to learn a Generator (G) capable of generating samples from the data distribution (p X), by converting latent vectors from a lower-dimension latent space (Z) to samples in a higher-dimension data space (X). Usually, latent vectors are sampled from Z using the uniform or the normal distribution. In order to train G, a Discriminator (D) is trained to distinguish real training samples from fake samples generated by G. Thus D returns a value D(x) ∈ which can be interpreted as the probability that the input sample (x) is a real sample from the data distribution. In this configuration, G is trained to obstruct D by generating samples which better resemble the real training samples, while D is continuously trained to tell apart real from fake samples. Crucially, G has no direct access to real samples from the training set, as it learns solely through its interaction with D. Both D and G are implemented by deep differentiable networks, typically consisting of multiple convolutional and fully-connected layers. They may be alternately trained using Stochastic Gradient Descent. In the short period of time since the introduction of the GAN model, many different enhancement methods and training variants have been suggested to improve their performance (see brief review below). Despite these efforts, often a large proportion of the generated samples is, arguably, not satisfactorily realistic. In some cases the generated sample does not resemble any of the real samples from the training set, and human observers find it difficult to classify synthetically generated samples to one of the classes which compose the training set (see illustration in FIG0).Figure 1: Images generated by different GANs trained on MNIST (top row), CelebA (middle row) and STL-10 (bottom row). Red square mark images of, arguably, low quality (best seen in color).This problem worsens with the increased complexity of the training set, and specifically when the training set is characterized by large inter-class and intra-class diversity. In this work we focus on this problem, aiming to improve the performance of GANs when the training dataset has large inter-class and intra-class diversity. Related Work. In an attempt to improve the performance of the original GAN model, many variants and extensions have been proposed in the past few years. These include architectural changes to G and D as in BID26, modifications to the loss function as in BID20; BID7, or the introduction of supervision into the training setting as in BID22; BID24. Another branch of related work, which is perhaps more closely related to our work, involves the learning of a meaningfully structured latent space. Thus Info-GAN decomposes the input noise into an incompressible source and a "latent code", Adversarial Auto-Encoders BID19 employ GANs to perform variational inference, and BID16 combine a Variational Auto-Encoder with a Generative Adversarial Network (see Appendix A for a more comprehensive description).Our Approach. Although modifications to the structure of the latent space have been investigated before as described above, the significance of the probability distribution used for sampling latent vectors was rarely investigated. A common practice today is to use a standard normal (e.g. N (0, I)) or uniform (e.g. U) probability distribution when sampling latent vectors from the latent space. We wish to challenge this common practice, and investigate the beneficial effects of modifying the distribution used to sample latent vectors in accordance with properties of the target dataset. Specifically, many datasets, especially those of natural images, are quite diverse, with high interclass and intra-class variability. At the same time, the representations of these datasets usually span high dimensional spaces, which naturally makes them very sparse. Intuitively, this implies that the underlying data distribution, which we try to learn using a GAN, is also sparse, i.e. it mostly consists of low-density areas with relatively few areas of high-density. We propose to incorporate this prior-knowledge into the model, by sampling latent vectors using a multi-modal probability distribution which better matches these characteristics of the data space distribution. It is important to emphasize that this architectural modification is orthogonal to, and can be used in conjunction with, other architectural improvements such as those reviewed above (see for instance FIG6 in Appendix D.) Supervision can be incorporated into this model by adding correspondence (not necessarily injective) between labels and mixture components. The rest of this paper is organized as follows: In Section 2 we describe the family of GM-GAN models. In Section 3 we offer an alternative method which focuses on measuring the trade-off between sample quality and diversity of generative models. In Section 4 we empirically evaluate our proposed model using various diverse datasets, showing that GM-GANs outperform the corresponding baseline methods with uni-modal distribution in the latent space. In Section 5 we describe a method for clustering datasets using GM-GANs, and provide qualitative and quantitative evaluation using various datasets of real images. Unsupervised GM-GAN. The target function which we usually optimize for, when training a GAN composed of Generator G and Discriminator D, can be written as follows: min DISPLAYFORM0 Above p X denotes the distribution of real training samples, and p Z denotes some d-dimensional prior distribution which is used as a source of stochasticity for the Generator. The corresponding loss functions of G and D can be written as follows: DISPLAYFORM1 DISPLAYFORM2 Usually, a multivariate uniform distribution (e.g. DISPLAYFORM3, or a multivariate normal distribution (e.g. N (0, I d×d)), are used as substitute for p Z when training GANs. In our proposed model, we optimize the same target function as in, but instead of using a unimodal random distribution for the prior p Z, we propose to use a multi-modal distribution which can better suit the inherent multi-modality of the real training data distribution, p X.Specifically, we propose to use a mixture of Gaussians as a multi-modal prior distribution where DISPLAYFORM4 Here K denotes the number of Gaussians in the mixture, DISPLAYFORM5 denotes a categorical random variable, and p k (z) denotes the multivariate Normal distribution N (µ k, Σ k), defined by the mean vector µ k, and the covariance matrix Σ k. In the absence of prior knowledge we assume a uniform mixture of Gaussians, that is, DISPLAYFORM6 Gaussian in the mixture can be fixed, or learned along with the parameters of the GAN in an "end-to-end" fashion to allow for a more flexible model. We investigated two corresponding variants of the new model -one (Static) where the the parameters of the Gaussians mixture are fixed throughout the model's training process, and one (Dynamic) where these parameters are allowed to change during the training process in order to potentially converge to a better solution. The details of these two variants are given in Appendix B.Supervised GM-GAN. In the supervised setting, we change the GM-GAN's discriminator so that instead of returning a single scalar, it returns a vector o ∈ R N where N is the number of classes in the dataset. Each element o i in this vector lies in. The Generator's purpose in this setting is, given a latent vector z sampled from the k'th Gaussian in the mixture, to generate a sample which will be classified by the discriminator as a real sample from class f (k), where f: DISPLAYFORM7 is a discrete function mapping identity of Gaussians to class labels. When K = N, f is bijective and the model is trained to map each Gaussian to a unique class in the data space. When K > N f is surjective, and multiple Gaussians can be mapped to the same class in order to model high diversity classes. When K < N f is injective, and multiple classes can be grouped together by mapping to the same Gaussian. We modify both loss functions of G and D to accommodate the class labels. The modified loss functions become the following: DISPLAYFORM8 where y(x) denotes the class label of sample x, and y(z) denotes the index of the Gaussian from which the latent vector z has been sampled. The training procedure for GM-GANs is fully described in Algorithm 1.Algorithm 1 Training the GM-GAN model. K -the number of Gaussians in the mixture. d -the dimension of the latent space (Z). c -the range from which the Gaussians' means are sampled. σ -scaling factor for the covariance matrices. iters -the number of training iterations. b D -the batch size for training the discriminator. b G -the batch size for training the Generator. γ -the learning rate. f -a mapping from Gaussian indices to class indices (in a supervised setting only). DISPLAYFORM0 init the mean vector of Gaussian k 3: DISPLAYFORM1 init the covariance matrix of Gaussian k 4: for i = 1...iters do 5: DISPLAYFORM2 Sample x j ∼ p X get a real sample from the training-set.7: DISPLAYFORM3 generate a fake sample using the Generator 10:if supervised then compute the loss of D 11: DISPLAYFORM4 else 14: DISPLAYFORM5 15: DISPLAYFORM6 16: DISPLAYFORM7 17: DISPLAYFORM8 update the weights of D by a single GD step.18: DISPLAYFORM9 generate a fake sample using the Generator if supervised then compute the loss of G 23: DISPLAYFORM0 else 25: DISPLAYFORM1 27: DISPLAYFORM2 update the weights of G by a single GD step. We describe next a new scoring method for GANs, which is arguably better suited for the task than the commonly used Inception Score proposed in BID28 and described in Appendix C. The Inception score has been used extensively over the last few years, but it has a number of drawbacks: (i) It is limited to the evaluation of GANS which are trained to generate natural images.(ii) It only measures the samples' inter-class diversity, ignoring the intra-class diversity of samples. (iii) It combines together a measure of quality and a measure of diversity into a single score. (iv) Different scores can be achieved by the same GAN, when sampling latent vectors with different parameters of the source probability distribution (e.g. σ, see Figure 6 in Appendix C).The Quality-Diversity trade-off: The quality of sample x ∈ X may be measured by its probability p X (x), which implies that samples drawn from dense areas in the source domain (i.e. close to the modes of the distribution) are mapped to high quality samples in the target domain, and vice versa. Therefore, we can increase the expected quality of generated samples in the target domain by sampling with high probability from dense areas of the source domain, and with low probability from sparse areas of the source domain. While increasing the expected quality of generated samples, this procedure also reduces the sample diversity 1. This fundamental trade-off between quality and diversity must be quantified if we want to compare the performance of different GAN models. Next we propose a new scoring method for either supervised or unsupervised GAN models, which is useful for multi-class image datasets, evaluating the trade-off between samples' quality and diversity. This scoring method also relies on a pre-trained classifier C, but unlike the Inception Score, this classifier is trained on the same training set on which the GAN is trained on. Classifier C is used to measure both the quality and the diversity of generated samples, as explained below. Quality Score To measure the quality of generated sample x, we propose to use the intermediate representation of x in the pre-trained classifier C, and to measure the Euclidean distance from this representation to its nearest-neighbor in the training set. More specifically, if C l (x) denotes the activation levels in the pre-trained classifier's layer l given sample x, then the quality score q(x), for sample x and a set of samples X, is defined as follows: DISPLAYFORM0 Above N N (x) denotes the nearest-neighbor of x in the training set, defined as N N (x) = arg min DISPLAYFORM1, and a denotes a positive constant. Diversity Score To measure the diversity of generated samples, we take into account both the inter-class, and the intra-class diversity. We measure intra-class diversity by the average (negative) MS-SSIM metric BID32 between all pairs of generated images in a given set of generated images X: DISPLAYFORM2 For inter-class diversity, we use the pre-trained classifier to classify the set of generated images, such that for each sampled image x, we have a classification prediction in the form of a one-hot vector c(x). We then measure the entropy of the average one-hot classification prediction vector to evaluate the diversity between classes in the samples set: DISPLAYFORM3 Finally, the diversity score is defined as the geometric mean of FORMULA24 and FORMULA25: DISPLAYFORM4 4 EMPIRICAL EVALUATION In this section we empirically evaluate the benefits of our proposed approach, comparing the performance of GM-GAN with the corresponding baselines. Thus we compare the performance of the unsupervised GM-GAN model to that of the originally proposed GAN BID6, and the performance of our proposed supervised GM-GAN model to that of AC-GAN BID24. In both cases, the baseline models' latent space probability distribution is standard normal, i.e. z ∼ N (0, I). The network architectures and hyper-parameters used for training the GM-GAN models are similar to those used for training the baseline models. For the most part we used the Static GM-GAN with default values d = 100, c = 0.1, σ = 0.15, B D = 64, b G = 128, γ = 0.0002; K and iters varied in the different experiments. The Dynamic GM-GAN model was only used in the experiments summarized in FIG3.In the following experiments we evaluate the different models on the 6 datasets listed in We first compare the performance of our proposed GM-GAN models to the aforementioned baseline models using a toy dataset, which has been created in order to gain more intuition regarding the properties of the GM-GAN model. The dataset consists of 5,000 training samples, where each training sample x is a point in R 2 drawn from a homogeneous mixture of K Gaussians, i.e., ∀x DISPLAYFORM0 In our experiments we used K = 9 Gaussians, ∀k ∈ [K] Σ k = 0.1 * I and µ = {−1, 0, 1} × {−1, 0, 1}. We labeled each sample with the identity of the Gaussian from which it was sampled. We trained two instances of the GM-GAN model, one supervised using the labels of the samples, and one unsupervised oblivious of these labels. In both cases, we used K = 9 Gaussians in the mixture from which latent vectors were sampled. Figure 2 presents samples generated by the baseline models (GAN, AC-GAN) and samples generated by our proposed GM-GAN models (both unsupervised and supervised variants). It is clear that both variants of the GM-GAN generate samples with a higher likelihood, which matches the original distribution more closely as compared to the baseline methods. The trade-off between quality and diversity is illustrated in Figure 3, showing high quality and low diversity for σ = 0.25, and vice versa for σ = 2.0 (see Section 4.3). An intriguing observation is that the GM-GAN's Generator is capable, without any supervision, of mapping each Gaussian in the latent space to samples in the data-space which are almost perfectly aligned with a single Gaussian. We also observe this phenomenon when training unsupervised GM-GAN on the MNIST and Fashion-MNIST datasets. In Section 5 we exploit this phenomenon to achieve a clustering algorithm. Finally, we note that the GM-GAN models converge considerably faster than the classical GAN model, see FIG4 in Appendix D. We next turn to evaluate our proposed models when trained on more complex datasets. We start by using the customary Inception Score BID28 to evaluate and compare the performance of the difference models, the two GM-GAN models and the baseline models (GAN and AC-GAN). We trained the models on two real datasets with 10 classes each, CIFAR-10 and STL-10 (see TAB0). Each variant of the GM-GAN model was trained multiple times, each time using a different number (K) of Gaussians in the latent space probability distribution. In addition, each model was trained 10 times using different initial parameter values. We then computed for each model its mean Inception Score and the corresponding standard error. The for the two unsupervised and two supervised models are presented in TAB2. In all cases, the two GM-GAN models achieve higher scores when compared to the respective baseline model. The biggest improvement is achieved in the supervised case, where the supervised variant of the GM-GAN model outperforms AC-GAN by a large margin. Model (unsupervised) Score As discussed in Section 3, the Inception Score is not sufficient, on its own, to illustrate the trade-off between the quality and the diversity of samples which a certain GAN is capable of generating. In our experiments, we control the quality-diversity trade-off by varying, after the model's training, the probability distribution which is used to sample latent vectors from the latent space. We do so by multiplying the covariance matrix of each Gaussian by a scaling factor σ. Specifically, when using the baseline models we sample z ∼ N (0, σ * I), and when using the GM-GAN models we DISPLAYFORM0 Thus, when σ < 1, latent vectors are sampled with lower variance around the modes of the latent space probability distribution, and therefore the respective samples generated by the Generator are of higher expected quality, but lower expected diversity. The opposite happens when σ > 1, where the respective samples generated by the Generator are of lower expected quality, but higher expected diversity. Figures 3, 4 demonstrate qualitatively the quality-diversity trade-off offered by GM-GANs when trained on the Toy and MNIST datasets. DISPLAYFORM1 Figure 4: Samples taken from a GM-GAN trained on the MNIST dataset. In each panel, latent vectors samples are drawn using different σ values (σ = 1.0 was used during training). Clearly The quality of samples decreases, and the diversity increases, as σ grows. Next, we evaluated each model by calculating our proposed Quality Score from Eq. FORMULA22, and the Combined Diversity Score from Eq. FORMULA26, for each σ ∈ {0.5, 0.6, ..., 1.9, 2.0}. Each model was trained 10 times using different initial parameter values. We computed for each model its mean Quality and mean Combined Diversity scores and the corresponding standard errors. The Quality and Diversity Scores of the GM-GAN and baseline models, when trained on the CIFAR-10 and STL-10 datasets, are presented in FIG3 (see additional datasets in FIG5 in Appendix D).In some cases (e.g. supervised training on CIFAR-10 and STL-10) the show a clear advantage for our proposed model as compared to the baseline, as both the quality and the diversity scores of GM-GAN surpass those of AC-GAN, for all values of σ. In other cases (e.g. unsupervised training on CIFAR-10 and STL-10), the show that for the lower-end range of σ, the baseline model generates images of higher quality but dramatically lower diversity, as compared to our proposed model. In accordance, when visually examining the samples generated by the two models, we notice that most samples generated by the baseline model belong to a single class, while samples generated by our model are much more diverse and are scattered uniformly among the different classes. In all cases, the charts predictably show an ascending Quality Score, and a descending Combined Diversity Score, as σ is increased. Throughout our experiments, we noticed an intriguing phenomenon where the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. Specifically, each Gaussian in the latent space is usually mapped, by the GM-GAN's Generator, to a single class in the data space. FIG0 in Appendix D demonstrates this phenomenon using different datasets. The fact that the latent space in our proposed model is sparse, while being composed of multiple Gaussians with little overlap, may be the underlying reason for this phenomenon. In this section we exploit this observation to develop a new clustering algorithm, and provide quantitative evaluation of the proposed method. Clustering Method In the proposed method we first train an unsupervised GM-GAN which receives as input the data points without corresponding labels. K, the number of Gaussians forming the latent space, is set to equal the number of clusters in the intended partition. Using the trained GM-GAN model, we sample from each Gaussian k ∈ [K] a set of M latent vectors, from which we generate a set of M synthetic samples DISPLAYFORM0. We then train a K-way multi-class classifier on the unified set of samples from all Gaussians k∈[K] X k, where the label of sample x ∈ X k is set to k, i.e. the index of the Gaussian from which the corresponding latent vector has been sampled. Finally, we obtain the soft-assignment to clusters of each sample x in the original dataset by using the output of this classifier c(x) ∈ K when given x as input. Each element c(x) k (k ∈ [K]) of this output vector marks the association level of the sample x to the cluster k. Hard-assignment to clusters can be trivially obtained from the soft-assignment vector by selecting the most likely clusterk = arg max k∈ [K] c(x) k. This procedure is formally described in Algorithm 2.Algorithm 2 Unsupervised clustering procedure using GM-GANs. Require:X -a set of samples to cluster. K -number of clusters. M -number of samples to draw from each Gaussian. DISPLAYFORM1 Train an unsupervised GM-GAN on X using K Gaussians. DISPLAYFORM2 Sample M latent vectors from the k'th latent Gaussian. Generate M samples using the set of latent vectors Z k. ∀ x ∈ X k y(x) ← k Label every sample by the Gaussian from which it was generated. 6: X ← k X k Unite all samples into the set X. 7: c ← classifier(X, y)Train a classifier on samples X and labels y. DISPLAYFORM0 Cluster X using classifier c. Method ACC NMI MNIST K-Means 0.5349 0.500 AE + K-Means 0.8184 -DEC 0.8430 -DCEC BID8 0.8897 0.8849 InfoGAN 0.9500 -CAE-l 2 + K-Means BID1 0.9511 -CatGAN 0.9573 -DEPICT BID4 0.9650 0.9170 DAC BID2 0.9775 0.9351 GAR BID11 0.9832 -IMSAT BID9 0 Table 3: Clustering performance of our method on different datasets. Scores are based on clustering accuracy (ACC) and normalized mutual information (NMI). Results of a broad range of recent existing solutions are also presented for comparison. The of alternative methods are the ones reported by the authors in the original papers. Methods marked with (*) are based on our own implementation, as we didn't find any published scores to compare to. We evaluated the proposed clustering method on three different datasets: MNIST, Fashion-MNIST, and a subset of the Synthetic Traffic Signs Dataset containing 10 selected classes (see TAB0). To evaluate clustering performance we adopt two commonly used metrics: Normalized Mutual Information (NMI), and Clustering Accuracy (ACC). Clustering accuracy measures the accuracy of the hard-assignment to clusters, with respect to the best permutation of the dataset's ground-truth labels. Normalized Mutual Information measures the mutual information between the ground-truth labels and the predicted labels based on the clustering method. The range of both metrics is. The unsupervised clustering scores of our method are presented in Table 3. We noet in passing that Algorithm 2 can be implemented with other GAN variants which are augments with a GM distribution of the latent space with similar beneficial , see FIG6 in Appendix D. This work is motivated by the observation that the commonly used GAN architecture may be ill suited to model data in such cases where the training set is characterized by large inter-class and intra-class diversity, a common case with real-world datasets these days. To address this problem we propose a variant of the basic GAN model where the probability distribution over the latent space is a mixture of Gaussians, a multi-modal distribution much like the target data distribution which the GAN is trained to model. This model can be used with or without label supervision. In addition, the proposed modifications can be applied to any GAN model, regardless of the specifics of the loss function and architecture (see, for example, FIG6 in Appendix D.).In our empirical study, using both synthetic and real-world datasets, we quantitatively showed that GM-GANs outperform baselines, both when evaluated using the commonly used Inception Score BID28, and when evaluated using our own alternative scoring method. We also demonstrated how the quality-diversity trade-off offered by our models can be controlled by altering, post-training, the probability distribution of the latent space. This allows one to sample higherquality, lower-diversity samples or vice versa, according to one's needs. Finally, we qualitatively demonstrated how the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. We further showed how this phenomenon can be exploited for the task of unsupervised clustering, and backed our method with quantitative evaluation. GANs have been extensively used in the domain of computer-vision, where their applications include super resolution from a single image BID18 ), text-to-image translation BID27, image-to-image translation BID10 BID12, image in-painting BID36 and video completion BID21. Aside from their use in the computer-vision domain, GANs have been used for other tasks such as semi-supervised learning BID14, music generation BID5 ), text generation BID37 and speech enhancement BID25.Subsequently much effort was directed at improving GANs through architectural changes to G and D, as in the DCGANs described in BID26. Improved performance was reported in BID20; BID7, among others, by modifying the loss function used to train the GAN model. Additional improvement was achieved by introducing supervision into the training setting, as in conditional GANs BID22 BID24. These conditional variants were shown to enhance the quality of the generated sample, while also improving the stability of the notorious training process of these models. In an effort to impose a meaningfully structure on the latent space, Info-GAN decomposes the input noise into an incompressible source and a "latent code", attempting to discover latent sources of variation by maximizing the mutual information between the latent code and the Generator's output. This latent code can be used to discover object classes in a purely unsupervised fashion, although it is not strictly necessary that the latent code be categorical. Adversarial AutoEncoders BID19 employ GANs to perform variational inference by matching the aggregated posterior of the auto-encoder's hidden latent vector with an arbitrary prior distribution. As a , the decoder of the adversarial auto-encoder learns a deep generative model that maps the imposed prior to the data distribution. BID16 combined a Variational Auto-Encoder with a Generative Adversarial Network in order to use the learned feature representations in the GAN's discriminator as basis for the VAE reconstruction objective. As a , this hybrid model is capable of learning a latent space in which high-level abstract visual features (e.g. wearing glasses) can be modified using simple arithmetic of latent vectors. The parameters of the mixture of Gauusian distribution used to sample the latent vector can be fixed or learned. One may be able to choose these parameters by using prior knowledge, or pick them randomly. Perhaps a more robust solution is to learn the parameters of the Gaussian Mixture along with the parameters of the GAN in an "end-to-end" fashion. This should, intuitively, allow for a more flexible, and perhaps better performing model. We therefore investigated two variants of the new model -one (static) where the the parameters of the Gaussians mixture are fixed throughout the model's training process, and one (dynamic) where these parameters are allowed to change during the training process in order to potentially converge to a better a solution. These variants are described in detail next:Static GM-GAN. In the basic GM-GAN model, which we call Static Multi-Modal GAN (Static GM-GAN), we assume that the parameters of the mixture of Gaussians distribution are fixed before training the model, and cannot change during the training process. More specifically, each of the mean vectors µ k is uniformly sampled from the multivariate uniform distribution U [−c, c] d, and each of the covariance matrices Σ k has the form of σ * I d×d, where c ∈ R and σ ∈ R are hyper-parameters left to be determined by the user. Dynamic GM-GAN. We extend our basic model in order to allow for the dynamic tuning of parameters for each of the Gaussians in the mixture. We start by initializing the mean vectors and covariance matrices as in the static case, but we include them in the set of learnable parameters that are optimized during the GAN's training process. This modification allows the Gaussians' means to wander to new locations, and lets each Gaussian have a unique covariance matrix. This potentially allows the model to converge to a better local optimum, and achieve better performance. The architecture of the Dynamic GM-GAN is modified so that G receives as input a categorical random variable k, which determines from which Gaussian the latent vector should be sampled. This vector is fed into a stochastic node used for sampling latent vectors given the Gaussian's index, i.e. z|k ∼ N (µ k, Σ k). In order to optimize the parameters of each Gaussian in the training phase, back-propagation would have to be performed through this stochastic node, which is not possible. To overcome this obstacle, we use the re-parameterization trick as suggested by BID13: instead of sampling z ∼ N (µ k, Σ k) we sample ∼ N (0, I) and define z = A k + µ k, where A ∈ R d×d and µ k ∈ R d are parameters of the model, and d is the dimension of the latent space. We thus get µ(z) = µ k and Σ(z) = A k A (a) (b) Figure 6: Inception Scores of Static GM-GAN models trained on (a) CIFAR-10 and (b) STL-10, when latent vectors are sampled using different values of σ. In both cases, the same model achieves very different Inception Scores when different values of σ are used. Both models were trained using σ = 1. Note that the best score is obtained for σ < 1, far from the training value σ = 1. ) trained on the fashion-MNIST dataset. We evaluated the original model with uniform and Gaussian latent space distribution. In addition, we incorporated multi-modal Gaussian distribution of the latent space into the model with 3, 5 and 10 mixture components. The of the GM-info-GAN variants are clearly better than vanilla, with best obtained for 5 Gaussian components. (a) (b) FIG0: Samples taken from two unsupervised GM-GAN models trained on the MNIST (top panels), Fashion-MNIST (middle panels) and CIFAR-10 (bottom panels) datasets. In (a) the Gaussian mixture contains K = 10 Gaussians; in each panel, each row contains images sampled from a different Gaussian. In (b) the Gaussian mixture contains K = 20 Gaussians; in each panel, each half row contains images sampled from a different Gaussian.
Multi modal Guassian distribution of latent space in GAN models improves performance and allows to trade-off quality vs. diversity