In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent vari-ables.   Unfortunately, this well-motivated strategy often fail to achieve disentanglement due to a problematic difference between the sampled latent representation and its corresponding mean representation.   We provide a theoretical explanation that low total correlation of sample distribution cannot guarantee low total correlation of the  mean representation. We prove that for the mean representation of arbitrarily high total correlation, there exist distributions of latent variables of abounded total correlation.   However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose a remedy,  RTC-VAE, which rectifies the total correlation penalty.    Experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g.,β-TCVAE and FactorVAE. VAEs (Variational AutoEncoders) Kingma & Welling (2013) ; Bengio et al. (2007) follow the common assumption that the high-dimensional real world observations x can be re-generated by a lowerdimension latent variable z which is semantically meaningful. Recent works Kim & Mnih (2018) ; Chen et al. (2018) ; Kumar et al. (2017) suggest that decomposing the ELBO (Evidence Lower Bound) could lead to distinguishing the factor of disentanglement. In particular, recent works Kim & Mnih (2018) ; Chen et al. (2018) focused on a term called total correlation (TC). The popular belief Chen et al. (2018) is that by adding weights to this term in objective function, a VAE model can learn a disentangled representation. This approach appears to be promising since the total correlation of a sampled representation should describe the level of factorising since total correlation is defined to be the KL-divergence between the joint distribution z ∼ q(z) and the product of marginal distributions j q(z j ). In this case, a low value suggests a less entangled joint distribution. However, Locatello et al. (2018) pointed out that the total correlation of sampled distribution T C sample being low does not necessarily give rise to a low total correlation of the corresponding mean representation T C mean . Conventionally, the mean representation is used as the encoded latent variables, an unnoticed high T C mean is usually the culprit behind the undesirable entanglement. Moreover, Locatello et al. (2018) found that as regularization strength increases, the total correlation of sampled representation T C sample and mean representation T C mean are actually negatively correlated. Locatello et al. (2018) put doubts on most methods of disentanglement including penalizing the total correlation term Kim & Mnih (2018) ; Chen et al. (2018) , and they concluded that "the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases". Acknowledging the difficulty in learning disentangled representation, we provide a detailed explanation of the seemingly contradictory behaviors of the total correlations of sampled and mean representation in previous works on TC penalizing strategy. Moreover, we find that this problem described above can be remedied simply with an additional penalty term on the variance of a sampled representation. Our contributions: • In Theorem 1, we prove that for all mean representations, there exists a large class of sample distributions with bounded total correlation. Particularly, a mean representation with arbitrarily large total correlation can have a corresponding sample distribution with low total correlation. This implies that a low total correlation of sample distribution cannot guarantee a low total correlation of the mean representation. (Section. 2) • Acknowledging the issue above, we further delve into total correlation, and provide a simple remedy by adding an additional penalty term on the variance of sample distribution. The penalty term forces a sampled representation to behave similar to the corresponding mean representation. Such penalty term is necessary for the strategy of penalizing T C mean in the view of Theorem 1. (Section. 4) • We study several different methods of estimating total correlation. They are compared and benchmarked against the ground truth value on the multivariate Gaussian distribution Locatello et al. (2018) . We point out that the method of (minibatch) estimators suffers from the curse of dimensionality and other drawbacks, making their estimation accuracy decay significantly with the increase of the dimension of the latent space, and some strong correlated distributions can be falsely estimated to have low total correlation. (Section. 5) In this work, we demonstrated that our RTC-VAE, which rectifies the total correlation penalty can remedy its peculiar properties (disparity between total correlation of the samples and the mean representations). Our experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models including β-TCVAE and FactorVAE. We also provide several theoretical proofs which could help diagnose several specific symptoms of entangle-ment. Hopefully, our contributions could add to the explainability of the unsupervised learning of disentangled representations.
diagnosed all the problem of STOA VAEs theoretically and qualitatively
{"src": "scitldr-AIC_train", "paper_id": "SkeuipVKDH"}