We present a novel architecture of GAN for a disentangled representation learning. The new model architecture is inspired by Information Bottleneck (IB) theory thereby named IB-GAN. IB-GAN objective is similar to that of InfoGAN but has a crucial difference; a capacity regularization for mutual information is adopted, thanks to which the generator of IB-GAN can harness a latent representation in disentangled and interpretable manner. To facilitate the optimization of IB-GAN in practice, a new variational upper-bound is derived. With experiments on CelebA, 3DChairs, and dSprites datasets, we demonstrate that the visual quality of samples generated by IB-GAN is often better than those by β-VAEs. Moreover, IB-GAN achieves much higher disentanglement metrics score than β-VAEs or InfoGAN on the dSprites dataset. Learning good representations for data is one of the essential topics in machine learning community. Although any strict definition for it may not exist, the consensus about the useful properties of good representations has been discussed throughout many studies BID9 Lake et al., 2017; BID10 . A disentanglement, one of those useful properties of representation, is often described as a statistical independence or factorization; each independent factor is expected to be semantically well aligned with the human intuition on the data generative factor (e.g. a chair-type from azimuth on Chairs dataset BID6 , or age from azimuth on CelebA dataset (Liu et al., 2015) ). The learned representation distilling each important factors of data into a single independent direction is hard to be done but highly valuable for many other downstream tasks (Ridgeway, 2016; Higgins et al., 2017b; .Many models have been proposed for disentangled representation learning (Hinton et al., 2011; Kingma et al., 2014; Reed et al., 2014; Narayanaswamy et al., 2017; BID13 . Despite their impressive results, they either require knowledge of ground-truth generative factors or weak-supervision (e.g. domain knowledge or partial labels). In contrast , among many unsupervised approaches BID14 Kingma & Welling, 2013; Rezende et al., 2014; Springenberg, 2015; BID15 ), yet the two most successful approaches for the independent factor learning are β-VAE BID20 and InfoGAN BID12 . BID20 demonstrate that encouraging the KL-divergence term of Variational autoencoder (VAE) objective (Kingma & Welling, 2013; Rezende et al., 2014) by multiplying a constant β > 1 induces a high-quality disentanglement of latent factors. As follow-up research , BID10 provide a theoretical justification of the disentangling effect of β-VAE in the context of Information Bottleneck theory BID25 BID24 BID12 propose another fully unsupervised approach based on Generative Adversarial Network (GAN) BID18 . He achieves the goal by enforcing the generator to learn disentangled representations through increasing the mutual information (MI) between the generated samples and the latent representations. Although InfoGAN can learn to disentangle representations for relatively simple datasets (e.g. MNIST, 3D Chairs), it struggles to do so on more complicated datasets such as CelebA. Moreover, the disentangling performance of the learned representations from InfoGAN is known as not good as the performance of the β-VAE and its variant models BID20 Kim & Mnih, 2018; BID11 .Stimulated by the success of β-VAE models BID10 BID20 Kim & Mnih, 2018; BID11 BID17 with the Information Bottleneck theory BID5 BID0 ) in disentangled representations learning task, we hypothesize that the weakness of InfoGAN in the representation learning may originate from that it can only maximize the mutual information but lacks any constraining mechanisms. In other words, InfoGAN misses the term upper-bounding the mutual information from the perspective of IB theory.We present a novel unsupervised model named IB-GAN (Information Bottleneck GAN) for learning disentangled representations based on IB theory. We propose a new architecture of GANs from IB theory so that the training objective involves an information capacity constraint that InfoGAN lacks but β-VAE has. We also derive a new variational approximation algorithm to optimize IB-GAN objective in practice. Thanks to the information regularizer , the generator can use the latent representations in a manner that is both more interpretable and disentangled than InfoGANThe contributions of this work are summarized as follows:1. IB-GAN is a new GAN-based model for fully unsupervised learning of disentangled representations. To the best of our knowledge, there is no other unsupervised GAN-based model for this sake except the InfoGAN's variants BID20 Kim & Mnih, 2018) .2. Our work is the first attempt to utilize the IB theory into the GAN-based deep generative model. IB-GAN can be seen as an extension to the InfoGAN, supplementing an information constraining regularizer that InfoGAN misses.3. IB-GAN surpasses state-of-the-art disentanglement scores of BID20 BID16 on dSprites dataset (Matthey et al., 2017) . The quality of generated samples by IB-GAN on 3D Chairs BID6 and CelebA (Liu et al., 2015) is also much realistic compared to that of the existing β-VAE variants of the same task. Connection to rate-distortion theory. Information Bottleneck theory is a generalization of the rate-distortion theory BID25 authors, 2019) , in which the rate R is the code length per data sample to be transmitted through a noisy channel, and the distortion D represents the approximation error of reconstructing the input from the source code authors, 2019; Shannon et al., 1951) . The goal of RD-theory is minimizing D without exceeding a certain level of rate R, can be formulated as min R,D D + βR, where β ∈ [0, ∞] decides a theoretical achievable optimal frontier in the auto-encoding limit .Likewise , z and r in IB-GAN can be treated as an input and the encoding of the input, respectively. The distortion D is minimized by optimizing the variational reconstructor q φ (z|x(r)) to predict the input z from its encoding r, that is equivalent to maximizing I L (z, G(z)). The minimization of rate R is related minimizing the KL(e ψ (r|z)||m(r)) which measures the in-efficiency (or excess rate) of the representation encoder e ψ (r|z) in terms of how much it deviates from the prior m(r).Disentanglement-promoting behavior. The disentanglement-promoting behavior of β-VAE is encouraged by the variational upper-bound of MI term (i.e. KL(q(z|x)||p(z))). Since p(z) is often a factored Gaussian distribution, the KL-divergence term is decomposed into the form containing a total correlation term (Hoffman & Johnson, 2016; Kim & Mnih, 2018; BID11 BID17 BID10 , which essentially enforces the encoder to output statistically factored representations (Kim & Mnih, 2018; BID11 . Nevertheless, in IB-GAN, a noise input z is fed into the representation encoder e ψ (r|z) instead of the image x. Therefore, the disentangling mechanism of IB-GAN must be different from those of β-VAEs.From the formulation of the Eq.(11), we could obtain another important insight: the GAN loss in IB-GAN can be seen as the secondary capacity regularizer over the noisy channel since the discriminator of GAN is the JS-divergence (or the reverse KL-divergence) between the generator and the empirical data distribution p(x) in its optimal BID18 BID22 . Hence, λ controls the information compression level of z in the its encoding x = G(r(z)) 6 . In other words, the GAN loss in IB-GAN is a second rate constraint in addition to the first rate constraint KL(e ψ (r|z)||m(r)) in the context of the rate-distortion theorem.Therefore, we describe the disentanglement-promoting behavior of IB-GAN regarding the ratedistortion theorem. Here, the goal is to deliver the input source z through the noisy channel using the coding r and x. We want to use compact encoding schemes for r and x. (1 ) The efficient encoding scheme for r is defined by minimizing KL(e ψ (r|z)||m(r)) with the factored Gaussian prior m(r), which promotes statistical independence of the r. (2) The efficient encoding scheme for x is defined by minimizing the divergence between G(z) and the data distribution p(x) via the discriminator; this promote the encoding x to be the realistic image. (3) Maximizing I L (z, G(z)) in IB-GAN indirectly maximize I(r, G(r)) since I(z, G(z)) ≤ I(r, G(r)). In other words, maximizing the lower-bound of MI will increases the statistical dependency between the coding r and G(r), while these encoding need to be efficient in terms of their rate. Therefore, a single independent changes in r must be coordinated with the variations of a independent image factor.How to choose hyperparameters. Although setting any positive values for λ and β is possible , we set β ∈ [0, 1] and fix λ = 1. We observe that, in the most of the cases, IU (r, R(z)) collapses to 0 when β > 0.75 in the experiments with dSprites. Although λ is another interesting hyperparameter that can control the rate of x (i.e. the divergence of the G(z) from p(x)), we aims to support the usefulness of IB-GAN in the disentangled representation learning tasks, and thus we focus on the effect of β ∈ [0, 1.2] on the I U (r, R(z)) while fixing λ = 1. More discussion on the hyperparameter setting will be discussed in Appendix. (Kim & Mnih, 2018; BID16 . Our model's scores are obtained from 32 random seeds, with a peak score of (0.91, 0.78). The baseline scores except InfoGAN are referred to BID17 . We use DCGAN (Radford et al., 2016) with batch normalization (Ioffe & Szegedy, 2015) as our base model for the generator and the discriminator. We let the reconstructor share the same frontend feature with the discriminator for efficient use of parameters as in the InfoGAN BID12 . Also, the MLP-based representation encoder is used before the generator. We train the model using RMSProp BID23 optimizer with momentum of 0.9. The minibatch size is 64 in all experiments. Lastly, we constrain true and synthetic images to be normalized as [−1, 1]. Almost identical architectural configurations for the generator, discriminator, reconstructor , and representation encoder are used in all experiments except that the numbers of parameters are changed depending on the datasets. We defer more details on the models and experimental settings to Appendix. The proposed IB-GAN is a novel unsupervised GAN-based model for learning disentangled representation. We made a crucial modification on the InfoGAN's objective inspired by the IB theory and β-VAE; specifically, we developed an information capacity constraining term between the generator and the latent representation. We also derived a new variational approximation technique for optimizing IB-GAN. Our experimental results showed that IB-GAN achieved the state-of-the-art performance on disentangled representation learning. The qualitatively generated samples of IB-GAN often had better quality than those of β-VAE on CelebA and 3D Chairs. IB-GAN attained higher quantitative scores than β-VAE and InfoGAN with disentanglement metrics on dSprites dataset.There are many possible directions for future work. First, our model can be naturally extended to adapt a discrete latent representation, as discussed in section 3.3. Second, many extensions of β-VAE have been actively proposed such as BID10 Kim & Mnih, 2018; BID11 BID17 , most of which are complementary for the IB-GAN objective. Further exploration toward this direction could be another interesting next topic. Reconstruction of input noise z. The resulting architecture of IB-GAN is partly analogous to that of β-VAE since both are derived from the IB theory. However, β-VAE often generates blurry output images due to the large β > 1 ( Kim & Mnih, 2018; BID11 BID17 since setting β > 1 typically increases the distortion . Recently, demonstrates the possibility of achieving small distortion with the minimum rate by adopting a complex auto-regressive decoder in β-VAE and by setting β < 1. However, their experiment is performed on relatively small dataset (e.g. MNIST, Omniglot).In contrast, IB-GAN may not suffer from this shortcoming since the generator in IB-GAN learns to generate image by minimizing the rate. Moreover , it does not rely on any probabilistic modeling assumption of the decoder unlike VAEs and can inherit all merits of InfoGANs (e.g. producing images of good quality by an implicit decoder, and an adaptation of categorical distribution). One downside of our model would be the introduction of additional capacity control parameter λ. Although, we fixed λ = 1 in all of our experiment, which could also affect the convergence or the generalization ability of the generator. Further investigation on this subject could be an interesting future work.Behaviors of IB-GAN according to β. If β is too large such that the KL-divergence term is almost zero, then there would be no difference between the samples from the representation encoder e ψ (r|z) and the distortion prior m(r). Then, both representation r and generated data x contain no information about z at all, resulting in that the signal from the reconstructor is meaningless to the generator. In this case, the IB-GAN reduces to a vanilla GAN with an input r ∼ p(r).Maximization of variational lower-bound. Maximizing the variational lower-bound of generative MI has been employed in IM algorithm BID2 and InfoGAN BID12 . Recently, offer the lower-bound of MI, named GILBO, as a data independent measure for the complexity of the learned representations for trained generative models. They discover the optimal lower-bound of the generative MI correlates well with the common image quality metrics of generative models (e.g. INCEPTION Salimans et al. (2016) or FID Heusel et al. (2017) ). In this work, we discover a new way of upper-bounding the generative MI based on the causal relationship of deep learning architecture, and show the effectiveness of the upper-bound by measures the disentanglement of learned representation.Implementation of IB-GAN. Since the representation encoder e ψ ( r|z) is stochastic, reparametrization trick (Kingma & Welling, 2013 ) is needed to backpropagate gradient signals for training the encoder model. The representation r can be embedded along with an extra discrete code c ∼ p(c) before getting into the generator (i.e. G(r, c)), and accordingly the reconstructor network becomes q(r, c|x) to predict the discrete code c as well. In this way, it is straightforward to introduce a discrete representation into IB-GAN, which is not an easy task in β-VAE based models.Theoretically, we can choose the any number for the dimension of r and z. However, The disentangled representation of IB-GAN is learned via the representation encoder e ψ (r|z). To obtain the representation r back from the real data x, we first sample z using the learned reconstructor q φ (z|x), and input it to the representation encoder e ψ (r|z). Therefore, we typically choose a smaller r dimension than that of z. For more details on the architecture of IB-GAN, please refer Appendix.E.Related Work. Many extensions of β-VAE BID20 have been proposed . BID10 modify β-VAE's objective such that the KL term is minimized to a specific target constant C instead of scaling the term using β. Kim & Mnih (2018) and BID11 demonstrate using the ELBO surgery (Hoffman & Johnson, 2016; Makhzani & Frey, 2017 ) that minimizing the KL-divergence enforces factorization of the marginal encoder, and thus promotes the independence of learned representation. However, a high value of β can decrease the MI term too much, and thus often leads to worse reconstruction fidelity compared to the standard VAE. Hence, they introduce a total correlation BID26 based regularization to overcome the reconstruction and disentanglement trade-off. These approaches could be complementary to IB-GAN, since the objective of IB-GAN also involves with the KL term. This exploration could be an interesting future work.
Inspired by Information Bottleneck theory,  we propose a new architecture of GAN for a disentangled representation learning
{"src": "scitldr-AIC_train", "paper_id": "ryljV2A5KX"}