Self-labeled Conditional GANs

This paper introduces a novel and fully unsupervised framework for conditional GAN training in which labels are automatically obtained from data. We incorporate a clustering network into the standard conditional GAN framework that plays against the discriminator. With the generator, it aims to find a shared structured mapping for associating pseudo-labels with the real and fake images. Our generator outperforms unconditional GANs in terms of FID with significant margins on large scale datasets like ImageNet and LSUN. It also outperforms class conditional GANs trained on human labels on CIFAR10 and CIFAR100 where fine-grained annotations or a large number of samples per class are not available. Additionally, our clustering network exceeds the state-of-the-art on CIFAR100 clustering.


Introduction
Generative adversarial networks(GANs) [15] are known as one of the most useful subset of generative models. Stan-dard GANs are independent of any kind of human supervision, involving adversarial training of a generator that synthesizes synthetic images given a low dimensional noise vectors, and a discriminator that aims to distinguish between synthetic and real images. They are also called unconditional GANs(uGANs), as there is no common source of information between the discriminator and the generator. Despite the high capability of GANs to generate realistic images, they are notorious for their instability and prone to mode collapse [1].
Conditional GANs(cGANs) [25] have effectively addressed the aforementioned challenges. In the cGAN framework, both the generator and discriminator are fed with an auxiliary input representing some factor that contributes to observations, e.g. object labels or data from different modalities. The discriminator aims to distinguish real image/conditioning term pairs from the fake pairs. Conditional GANs guide the training by considering different modes of the data distributions. When conditioned on labels, cGANs explicitly approximate the joint distribution of observations and labels. For the sake of simplicity, we denote cGANs as those conditioned on class labels throughout 1 the paper.
Although cGANs benefit from a rich prior, their drawback lies in dependency on human labels. To cope with this limitation, we propose Self-labeled Conditional GANs(slcGANs) that learn to assign labels to the images automatically by incorporating an additional clustering network. The benefits are twofold. First, training is independent of human labels. Second, the slcGAN framework provides a rich supervisory signal for our clustering network.
A well known motivation for unsupervised learning from a practical point of view is to alleviate the cost and error of the labeling process. Being independent of human labels not only saves the cost of labeling but also allows us to find a presumably better set of pseudo-labels through which conditional GANs can achieve higher performance on the task of image generation. This is because human labels are based on semantic contents that still include large variety of intra-class variations. This issue is more problematic in datasets like CIFAR10 or LSUN [34], where a fine set of labels is not available, and a large variety of images with different object and background appearances are grouped as a single class. Approximating such complex conditional distributions is still challenging. As it is shown in Figure 1, our clustering network splits the dataset into fine-grained clusters based on abstract factors of variation such as shape, appearance, pose, interaction with other objects and/or background, scale, quantity, etc. Each cluster proposes a simpler subdistribution that includes fewer variations and therefore is easier to approximate for the generator.
Our proposed slcGAN framework includes a clustering network in addition to a standard cGAN. It assigns a Kdimensional probability vector to a real image as input, where K is a given parameter indicating the number of clusters. An image and its corresponding class probability vector is fed to the discriminator as a real pair. The generator takes a randomly selected label, i.e. conditioning term, in addition to a noise vector sampled from a multivariate Gaussian, and generates a fake image. A generated image and its corresponding conditioning term is fed to the discriminator as a fake pair. The clustering network and the generator aim to fool the discriminator such that it fails to distinguish between the real and fake pairs. For this purpose, the clustering network and the generator should follow the same strategy on assigning pseudo-labels to the real images and associating conditioning term to the fake images. The pseudo-label assignment strategy should facilitate the generator task of producing realistic images, as low quality images enable the discriminator to distinguish between the fake and real pairs based on the images alone regardless of the mechanism with which they are tied to the pseudo-labels.
A degenerate solution for the clustering network and the generator of the three player game described above is to hide any dependency between the images and the pseudolabels. To prevent this, we explicitly enforce the generator to establish dependency between the conditioning term and the fake images. Moreover, we provide an auxiliary supervisory signal for the clustering network via multiple view clustering that enforces consistency between the pseudolabels assigned to an image and its augmented versions.
Our experiments show that our proposed slcGAN framework trained with a relatively large number of clusters outperforms unconditional GANs on ImageNet and LSUN datasets with significant margins. Additionally, it outperforms cGANs trained on human labels in terms of FID on CIFAR10 and CIFAR100. Moreover, our clustering network exceeds the state-of-the-art on CIFAR100 in terms of clustering accuracy. Our contributions are: I) We introduce slcGANs, a novel and fully unsupervised framework for joint clustering and conditional GAN training. II) We show that slcGANs outperform uGANs on several large scale datasets, and cGANs conditioned on human labels in terms of FID for image generation on CIFAR10 and CI-FAR100. III) We show that our clustering network exceeds the state-of-the-art for image clustering on the CIFAR100 dataset.

Related work
GANs as a source of supervisory signal. GANs have been exploited before as a source of supervisory signal for learning representations. Radford et al. [29] transfer features of the discriminator of unconditional GANs. Chen et al. [6] investigated the same direction by training the discriminator of unconditional GANs jointly with the auxiliary task of rotation prediction [14]. Adversarially learned inference (ALI) [12] or bidirectional GAN(BiGAN) [2] proposed primarily to augment the standard GAN with an encoder network that maps real data to the latent space. In the case of an optimal discriminator, [2] showed that a deterministic BiGAN acts like an autoencoder, i.e. combination of the encoder and generator, that minimizes 0 reconstruction costs, where the shape of the reconstruction error surface is dictated by a parametric discriminator. Our objective is fundamentally different. We aim to decompose the distribution into smaller subdistributions, each of which is easier for the generator to approximate. The BigBiGAN framework [11] is a recently revised version of BiGAN using the BigGAN [10] generator and a modified discriminator.
Clustering. End-to-end learning for image clustering has been recently explored. A line of methods [33,4,3] follow an iterative approach that alternates between obtaining pseudo-labels via clustering, e.g. k-means, and feature learning by predicting the pseudo-labels obtained from the previous stage. The idea of these methods is tied to the architecture of CNNs as a prior to cluster images. Another line of research [20,19] exploits maximizing the mutual information between an image and its augmentations. More related to our approach, Mukherjee et al. [28] exploit GANs for clustering. They perform clustering in the latent space of unconditional GANs by joint training of an encoder that projects back a generated image to the corresponding latent space. Apart from the complexity of training an optimal encoder in their approach and the suboptimality of unconditional GANs, their encoder never observes real images during the training. Our clustering method obtains the supervisory signal from the generator indirectly to train a separate clustering network. Our clustering network is updated through real images, however, the training signal originates from the generated images via cooperation with the generator.
Conditional GANs without annotations. Lucic et al. [24] exploit the feature space of a pre-trained selfsupervised task along with a fraction of ground truth labels in a semi-supervised setting to alleviate the need for a fully annotated training set. Our approach requires neither a pretrained network nor a partially labeled training set. Very recently and concurrently with us, Liu et al. [23] pursued a similar problem setup as ours. They perform clustering via k-means on the feature space of the discriminator and alternate between clustering and the conditional GAN training. This approach suffers from two main drawbacks. First, their performance relies on the quality of the intermediate representation of the discriminator, which is not necessarily an optimal feature extractor. Second, their method requires an existing clustering algorithm, e.g. k-means. By incorporating a new clustering network to the conditional GAN framework, our method not only is independent of an existing feature extractor but also learns useful features independently of a clustering algorithm. Our experiments show that our method obtains clusters with significantly higher purity, which is essential for the image generation task.

Method
Conditioning on class labels has been effective on alleviating the mode collapse issue of GANs. They enlarge the support of the generator by guiding it to cover all the classes available in the dataset. Our proposed method aims for the same goal in an unsupervised setting in which we obtain the pseudo-labels by incorporating a clustering network. Given a set of unlabeled images, our objective is to train simultaneously a clustering network that assigns a cluster probability vector to each datapoint used as a conditioning term for training conditional GANs. We review the conditional GAN framework in the following and discuss our method in more detail.

Conditional GANs
GANs involve adversarial training of two neural networks, a generator(G) and a discriminator(D). Given a noise vector sampled from a simple low dimensional distribution, the generator aims to approximate the underlying unknown data distribution P r to fool the discriminator, while the discriminator aims to distinguish between real and fake samples.
Conditional GANs extend the GAN framework by exploiting an auxiliary source of information available for G and D as an extra input, where the real joint samples of images and the conditioning terms are empirically available in the training dataset. Class labels are the most widely used conditioning information in Equation 2. We further extend the class conditional GAN framework to not rely on explicit labels in the dataset.

Self-labeled Conditional GANs
Our proposed method involves adversarial training of three networks. In addition to G and D of cGANs, we incorporate a clustering network (C) that assigns pseudo-labels to the images cooperating with G against D. A primary objective function can be obtained simply by substituting labels in the cGAN framework with p(y|x) approximated by C, where K is the number of clusters, 1 K (c) is a Kdimensional one-hot encoding of c sampled from prior distribution, p c , and p(y|x) = softmax(C(x)) is a Kdimensional class probability vector of x determined by C. p z is a multivariate Gaussian and p c is a uniform distribution.
To fool the discriminator, the above objective function requires C and G to match the joint pairs of (x, p(y|x)) sampled from the real distribution to the synthetic pairs of (G(z, c), 1 K (c)). This encourages G to generate realistic images, C to distribute uniformly one-hot encoding probability vectors over the images, and both C and G to follow a shared mapping for associating pseudo-labels with the real and fake images respectively. Moreover, this mapping should assist G with generating more realistic images. The discriminator(D) takes an image and the corresponding pseudo-label as input. A real pair is composed of a real image and the assigned probability vector by C(bottom row). A fake pair is composed of a randomly generated image by the generator(G) and the conditioning term(c) by which it has been created(top row). C and G are trained against D such that it fails to distinguish between real and fake pairs. Mutual information maximization is imposed via a cross entropy loss(top row). C is not updated through this loss, shown in gray. Transformation consistency is imposed by a separate cross entropy loss on real samples(bottom row). Following the BigBiGAN framework, the adversarial loss involves separate terms for image(u) and joint inputs(s).
Preventing random label assignment. Any mismatch on pseudo-label association between C and G to the real and fake images provides D with a signal to distinguish between the fake and real pairs. A degenerate solution for C and G to hide this signal from D is to decouple images from the pseudo-labels. That is, random label assignment to the real images by C and independent fake images of the conditioning term by G.
To prevent this degenerate solution, we explicitly enforce the fake images to depend on the conditioning term. We follow the same approach as the InfoGAN framework [7] by maximizing a variational lower bound of I(c; G(z, c)) parameterized by a neural network, where I denotes mutual information. The InfoGAN framework adapts the discriminator for this purpose by adding a classification head for the fake images in addition to the one for real/fake discrimination.
We however can not follow the same approach as our discriminator is conditional. It takes c as the second input, which makes predicting c a trivial task. We thus use C alternatively by adding the following loss function: where p(y = c|G(z, c)) is estimated by C as it is performed for the real images in Equation 3. Note that C can also be updated by L mi . However, we obtained better performance by training this loss only for G. One reason could be that direct communication between G and C yields degenerate solutions by encoding shortcuts in the fake images by G. These shortcuts allow G and C to minimize L mi without establishing a comprehensive dependency between the fake images and the conditioning term. Furthermore, optimizing L mi only for G provides the generator with another supervisory signal for generating more realistic images consistent with C. Intuitively, it encourages G to generate realistic images such that C behaves on them in a similar manner to real images in terms of pseudo-label prediction. Note that C is only updated via real images throughout the training.
Multiple view clustering. We further provide an auxiliary supervisory signal for C via multiple view clustering. That is, C should assign the same pseudo-label to an image and its augmented versions, Where x t is a randomly augmented version of x via cropping, color jittering, and flipping.
Our approach of multiple view clustering is inline with learning via invariances across augmentations [5], an approach that has shown significant improvements in the field of self-supervised representation learning. These methods mostly rely on contrastive learning [31] to prevent the degenerate solution of constant prediction, which is not required in our method. The supervisory signal originating from the adversarial training enforces C to mimic a similar distribution as p c , which is simply a uniform distribution.

4
Our final objective function is the following.

29: end for
Architecture and adversarial losses. Our generator and discriminator architecture is based on those of Big-GAN [10]. We use a linear transformation to map pseudoclass probability vectors to higher dimensional dense vectors that are later provided to G with class-conditional BatchNorm [13,8], and to D with projection [26]. We use spectral normalization [27] in both C and G and train the adversarial objective functions using the hinge loss [22,32]. Following the BigBiGAN framework, we include a unary term in our adversarial loss that is a function of only images, and split the loss values across two terms. The unary term encourages G towards explicitly generating realistic images, which is in turn implicitly performed via the joint term, i.e. Equation 3. Algorithm 1 demonstrates the training steps of slcGANs in detail.

Experiments
We evaluate our generator in terms of image generation performance and the accuracy of our clustering network. Ground truth labels provide a plausible upper bound for performance in both cases. We conduct experiments on MNIST, CIFAR10, CIFAR100, LSUN, and ImageNet datasets.
Implementation details. We set the channel multiplier to 128 and the number of residual blocks per stage to 1 for both G, D. We use standard ResNet18 [17] as the backbone of our clustering network, Adam [21] optimizer with a batch size of 256 for all networks, a learning rate of 1e − 4, and update the discriminator twice per generator update.

Image synthesis
We evaluate image generation performance quantitatively by reporting the Inception Score(IS) [30] and the Fréchet Inception Distance(FID) [18]. The unconditional GAN, class conditional GAN, and slcGAN are trained with the same architecture. We evaluate our method for small, medium, and large values of K, and perform qualitative evaluations later in section 4.3.
CIFAR. We train our model with (10, 50, 100) on CI-FAR10 and (20, 100, 500) on CIFAR100 as small, medium, and large values of K respectively. As it is shown in Table 1, the slcGAN framework outperforms the unsupervised baselines as well as the cGAN by significant margins. We conjecture that this is due to the number of modes imposed by a large number of clusters, yielding more variant synthetic images, and the flexibility of associating pseudo-labels to the fake images by the generator compared to preassigned labels.

CIFAR10
CIFAR100    LSUN and (100, 1000, 2000) as small, medium, and large values of K respectively. We outperform the unsupervised baselines, closing the gap with the ground truth labels.

Clustering
We evaluate the clustering accuracy when the number of clusters matches the ground truth. For CIFAR100, we consider two cases of 20 superclasses and the original finegrained 100 classes. Although slcGAN does not outperform the baselines on MNIST and CIFAR10, and 20 superclasses on CIFAR100, it outperforms IIC [20] on CIFAR100 when the ground truth includes fine-grained labeling, 15.7% versus 9.5%. 1 We believe this is related to the better generator performance on approximating smaller subdistributions represented by each fine-grained cluster. Our method benefits from a large number of clusters as our generator provides the clustering network with more accurate gradients via the discriminator. The results in Table 3 shows the superiority of our method on fine-grained clustering.
Overclustering. Our generator favors a fine-grained clustering where each cluster represents specific attributes of data. A plausible assessment for this purpose is purity in terms of ground truth labels as images with the same ground truth label are more likely to share the same finegrained attributes. Purity involves assigning each cluster MNIST CIFAR10 CIFAR100 k = 20 k = 100 DCGAN [29] 82.8 31.  Table 3: Clustering evaluations. We find the best permutation that maps our pseudo-labels to the ground truth and measure the accuracy. The numbers of other methods are taken from [20]. We significantly outperform IIC on fine-grained clustering on CIFAR100.
to the class which is most frequent in the cluster and then measuring the accuracy of this assignment by counting the number of correctly assigned datapoints. It is computed as . . , π K } is the set of clusters, C = {c 1 , . . . , c J } is the ground truth classes, and N is the dataset size. Given a dataset with a coarse set of classes, a higher purity indicates dividing each class into fine pseudo-classes.
Purity is affected by the K value as well as cluster distribution prior, i.e. p c in Equation 3, which both are unknowns. To obtain a better insight into how semantics is involved in the clustering, we train a linear classifier on the 512 dimensional features of the layer before the last for image classification using the ground truth labels. The higher performance indicates more semantics involved in the intermediate layers of C for clustering images. As it is shown in Table 4, purity correlates with linear separability, and our method achieves an acceptable performance, Liu et al. [23] achieves a purity of 11.73 on CIFAR10 with K = 100.  Clusters distribution. As it is mentioned in section 3.2, our objective function in Equation 3 encourages C to distribute uniformly one-hot encoding probability vectors over the images. Note that we do not benefit from true ground truth distribution here, since the number of clusters in our training is relatively higher than ground truth. Moreover, it turns out that imposed constraint via adversarial training is not strong. Figure 4 compares our method with k-means in terms of the histogram of the clusters length, the number of samples assigned to the clusters. Our method is trained on LSUN with K = 1000, and k-means is applied on the features of the layer before the last for the same number of clusters. Our method covers a broader range of clusters length than k-means. This observation indicates that our method benefits from the weekly imposed constraint via adversarial training to adjust the distribution of the clusters to facilitate the image generation task of the generator.   Table 4, indicates that the multiple view clustering loss is effective, and more importantly that the main signal originates from the adversarial training.

Qualitative evaluation
We show several synthesised and real clusters of Ima-geNet in Figure 5 and LSUN in Figure 6. Our model is trained on both datasets with 1000 clusters. More qualitative results are available in the supplementary material.
Reconstruction comparison with BigBiGAN. Despite that the BigBiGAN framework can act like an autoencoder, i.e. a combination of the encoder and generator, our method does not reconstruct the copy of a given image. However, our model allows us to reconstruct multiple instances of the same cluster that a given image belongs to. That is, we obtain the pseudo-label of a given image via the clustering network and feed it to the generator with multiple randomly sampled noise vectors. In the case of fine-grained labels, we are able to reconstruct multiple instances with the same style of a given image. Figure 3 shows some examples on the MNIST dataset. Given an image (a), we show BigBi-GAN reconstruction (b), samples generated by our model trained with 10 pseudo-classes (c), and 50 pseudo-classes (d). For each digit we show two samples with different styles. The results show that a fine-grained model recovers the same style of the input, while the coarse-grained model recovers images with the same category. BigBiGAN reconstructs the exact input image.

Conclusion
We have shown that class conditional GANs can be trained independently of the human labels, achieving higher performance than unconditional GANs. Our main idea involves incorporating a clustering network into the class conditional GAN framework that automatically assigns pseudo-7 Generated cluster Real cluster Generated cluster Real cluster Figure 5: Visualization of the generated and real images of several clusters for ImageNet. Each row shows a cluster and the generated images conditioned on the same pseudo-label.
Generated cluster Real cluster Generated cluster Real cluster Figure 6: Visualization of the generated and real images of several clusters for LSUN 20 object categories. Each row shows a cluster and the generated images conditioned on the same pseudo-label.
labels to the images. The benefits of our proposed method are twofold. Our framework not only improves image generation performance on unlabeled datasets but also achieves a useful clustering network that exceeds the state-of-theart for fine-grained clustering. We believe our work develops further the active line of research of generative mod-els towards higher quality and annotation free frameworks. Moreover, we show GANs provide rich supervisory signals for clustering and representation learning.

Generated cluster
Real cluster Figure 7: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. 11 Generated cluster Real cluster Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel. Figure 1: Visualization of the generated and real images of several clusters for ImageNet. Each row shows 8 samples with the highest confidence score by the clustering network of a cluster and the generated images conditioned on the same pseudolabel.