In the problem of unsupervised learning of disentangled representations, one of the promising methods is to penalize the total correlation of sampled latent vari-ables. Unfortunately, this well-motivated strategy often fail to achieve disentanglement due to a problematic difference between the sampled latent representation and its corresponding mean representation. We provide a theoretical explanation that low total correlation of sample distribution cannot guarantee low total correlation of the mean representation. We prove that for the mean representation of arbitrarily high total correlation, there exist distributions of latent variables of abounded total correlation. However, we still believe that total correlation could be a key to the disentanglement of unsupervised representative learning, and we propose a remedy, RTC-VAE, which rectifies the total correlation penalty. Experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models, e.g.,β-TCVAE and FactorVAE. VAEs (Variational AutoEncoders); follow the common assumption that the high-dimensional real world observations x can be re-generated by a lowerdimension latent variable z which is semantically meaningful. Recent works;; suggest that decomposing the ELBO (Evidence Lower Bound) could lead to distinguishing the factor of disentanglement. In particular, recent works; focused on a term called total correlation (TC). The popular belief is that by adding weights to this term in objective function, a VAE model can learn a disentangled representation. This approach appears to be promising since the total correlation of a sampled representation should describe the level of factorising since total correlation is defined to be the KL-divergence between the joint distribution z ∼ q(z) and the product of marginal distributions j q(z j). In this case, a low value suggests a less entangled joint distribution. pointed out that the total correlation of sampled distribution T C sample being low does not necessarily give rise to a low total correlation of the corresponding mean representation T C mean. Conventionally, the mean representation is used as the encoded latent variables, an unnoticed high T C mean is usually the culprit behind the undesirable entanglement. found that as regularization strength increases, the total correlation of sampled representation T C sample and mean representation T C mean are actually negatively correlated. put doubts on most methods of disentanglement including penalizing the total correlation term; , and they concluded that "the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases". Acknowledging the difficulty in learning disentangled representation, we provide a detailed explanation of the seemingly contradictory behaviors of the total correlations of sampled and mean representation in previous works on TC penalizing strategy. Moreover, we find that this problem described above can be remedied simply with an additional penalty term on the variance of a sampled representation. Our contributions: • In Theorem 1, we prove that for all mean representations, there exists a large class of sample distributions with bounded total correlation. Particularly, a mean representation with arbitrarily large total correlation can have a corresponding sample distribution with low total correlation. This implies that a low total correlation of sample distribution cannot guarantee a low total correlation of the mean representation. (Section. 2) • Acknowledging the issue above, we further delve into total correlation, and provide a simple remedy by adding an additional penalty term on the variance of sample distribution. The penalty term forces a sampled representation to behave similar to the corresponding mean representation. Such penalty term is necessary for the strategy of penalizing T C mean in the view of Theorem 1. (Section. 4) • We study several different methods of estimating total correlation. They are compared and benchmarked against the ground truth value on the multivariate Gaussian distribution. We point out that the method of (minibatch) estimators suffers from the curse of dimensionality and other drawbacks, making their estimation accuracy decay significantly with the increase of the dimension of the latent space, and some strong correlated distributions can be falsely estimated to have low total correlation. (Section. 5) In information theory, total correlation is one of the generalizations of mutual information, which measures the difference between the joint distribution of multiple random variables and the product of their marginal distributions. A high value means the joint distribution is far from an independent distribution, and hence it suggests high entanglement among these random variables. Definition 1. Total correlation of random variable x,. Naturally, people seek the solution of disentanglement in the form of low total correlation of the latent variables, e.g.;. However, there can be large difference between the total correlations of sample representation and mean representation. Forcing the former to be small does not guarantee the latter being small. In fact, given a mean representation of arbitrarily large total correlation, we can construct a family of distribution of sample representation that have a bounded total correlation, where the bound does not rely on the total correlation of the mean. Theorem 1. Let µ ∼ N (0, Σ) and σ j be the standard deviation of µ j, j = 1, · · ·, D, and, where Σ (µ) is diagonal and satisfies that for some R > 0, The details of the proof are presented in Appendix A.3. Here's another way to interpret Theorem 1: with C and parameters R, c 0, · · ·, c 4, l fixed, one can make T C(µ) arbitrarily large, since T C(µ) depends only on the correlation matrix of µ (see Proposition 1). Theorem 1 provides an explanation to the contradiction observed by that TC(z) is low does not mean TC(µ) is low (actually much higher than TC(z)). Since there exist such a large class of distributions of z that all have bounded T C(z). When the objective function only penalizes T C(µ), neural networks are so flexible to easily find a distribution with low T C(z), and total correlation estimators like MSS can encourage shutting down latent dimensions (see Section 5.3), which together cause the disparity of T C(µ) and T C(z). This fact was not noticed until , and our investigation gives an explanation of the peculiar property of total correlation. Hence, Theorem 1 leads to the necessity of a regularizer of the difference between the distributions of µ and z when penalizing T C sample. In Section. 4, we propose a simple regularizer that serves this goal. It is an interesting question whether there exists a distribution of z N (µ, Σ (µ)) with arbitrarily small T C(z) given µ. If not, what's the lower bound of T C(z)? These questions remain open to us for now, and we leave them to future work. In the study of disentanglement, proposed a modification of the VAE framework and introduced an adjustable hyperparameter β that balances latent channel capacity and independence constraints with reconstruction accuracy. One drawback of β-VAE is the trade-off between the reconstruction quality and disentanglement. Motivated to alleviate this trade-off of β- proposed FactorVAE which decomposes the evidence lower bound and penalize a term measuring the total correlation between latent variables. Around the same time, proposed a similar ELBO decomposition method called β-TCVAE. The major difference between FactorVAE and β-TCVAE lies in their different strategies of estimating total correlation. used formulated estimators while utilized the density-ratio trick which requires an auxiliary discriminator network and an inner optimization loop. We will discuss these two strategies more in details in Section. 5. The works above belong to representative learning without inductive biases. There are also works about representative learning with inductive biases, see and references therein. As for the disentanglement metric, this will be discussed in Section. 6. To simplify notation, let p(n) = p(x n), q(z|x n) = q(z|n). Recall the average evidence lower bound (ELBO), and independently by introduced objective function that penalizes total correlation, which can be formulated as This approach unfortunately has a drawback. It turns out that instead of being able to obtain disentangled representation, we often find a sample representation appears to be disentangled while the mean representation is still entangled. In fact, when we are maximizing L β−TC, we could totally end up learning a distribution of z that makes TC(z) goes low, while the total correlation of its mean µ is still high. To resolve this, we define RTC-VAE, where Our penalty originates from the first term of the law of total covariance Factorized representation 1 indicates a diagonal covariance matrix Cov q(z) [z]. Motivated by this, penalizes the off-diagonal terms in the second term, while ignores E p(n) Cov q(z|n) [z] since it is diagonal. Their penalty term leads to a vanishing µ, which is the mean representation. The remedy to this is to add another penalty term on the distance between µ's and 1. employs this remedy, however, DIP-VAE does not outperform other VAE's when measured by various disentanglement metrics, e.g., FactorVAE score, see Fig. 3 & 14 in. This is actually not surprising since the two penalty terms in DIP-VAE contribute in opposite directions, with one leading to vanishing µ's and another fighting against it. This formulation can easily get the model stuck in saddle points. Our objective, on the other hand, does not penalize directly on µ. Instead, it penalizes on σ, the standard deviation of the distribution q(z|n). This may seem little counter-intuitive at first sight, since penalizing a diagonal component of covariance Cov[z] = Cov q(z) [z] seems not helpful to factorising. However, in the view of Theorem 1, our objective will force the distribution of z to be similar to the distribution of µ. Hence, it pushes us away from the situation of large TC(µ) and low TC(z). Consequently, by minimizing TC(z) we get a model that has low TC(µ), a disentangled mean representation. The naive Monte Carlo method comes with an intrinsic issue of underestimating total correlation. To avoid or resolve this, proposed a discriminator network with the help of density-ratio trick (see equation and Appendix D. of ). , two kinds of estimator of total correlation are proposed, Minibatch Weighted Sampling (MWS) and Minibatch Stratified Sampling (MSS) (see Appendix C.1 and C.2 in). For instance, MSS can be described as followed. For a minibatch of sample, For the convenience of readers, MWS is listed in Appendix A.1. Density-ratio trick; can be used to estimate KLdivergence, where D is discriminator that classifies z being sampled from q(z) or j q(z j). For detail implementation, please refer to section 3 in. For multivariate normal distribution, the total correlation can be explicitly calculated which can be used as a ground truth for our comparison. To be specific, Proposition 1. Let x ∼ N (0, Σ), then 2 There is a small part of the implementation of MSS in Chen et al.'s code that is not quite clear to us, specifically, the computation of log importance weight matrix in equation 6. In our experiment, we implement MSS with our understanding and denote it as MSS1, and we denote Chen et al.'s implementation MSS0. See Appendix A.2 It's difficult to track the exact reference of Proposition 1 since it is a fundamental property in information theory. used this proposition to approximate the total correlation of the mean representation in latent space. In appendix, we provide a simple proof for the convenience of the readers. We compared the performance of each method, MWS, MSS 0 and MSS 1 on the estimation of total correlation. For µ ∼ N (0, I), and z|µ ∼ N (µ, Σ) where Σ = diag(σ 2) and σ = 0.1. We choose σ small so that the distribution of z can be approximated by normal distribution. Results are presented in Figure 1. From Figure. 1, we can summarize the following observations: 1. MWS tends to underestimate total correlation in general; 2. For latent space of dimension ≤ 4, MSS 0 and MSS 1 are quite accurate; 3. For latent space of high dimension, both MSS 0 and MSS 1 tend to overestimate total correlation when the actual value of total correlation is small; 4. Overall MSS 1 estimates closer to ground truth than MMS 0 does. In the following analysis of the above observations, we use a less formal way of analyzing, which can be formalized to be rigorous, in order to convey our idea directly. To interpret the third observation, let µ ∼ N (0, Id) and z|µ ∼ N (0, Σ), where Id is identity matrix and Σ = diag(σ 2). Then T C(µ) = 0 and T C(z) small if σ small. Consider q(z is a sample drawn in a minibatch and z (i) = z(n (i) ). We claim this: when the ground truth total correlation of z is low (the off-diagnal values of correlation matrix is small), only the elements on the diagonal surface of the cube, namely those elements of index (i, i, k), take some bounded values O, and all the other elements are very small o (since σ = 0.1). To see the claim, let's first consider 1-D case, where µ ∼ N, z|µ ∼ N (0, σ 2). When σ is small, z can be approximately treated as N. z (i) and µ (j) are independent for i = j, hence z (i) − µ (j) ∼ N, and See a proof in Appendix A.5. Then for D-dimension, the probability P (|z . Now, only if t takes value as small as σ, q(z is not small. For example, σ = 0.1 and D = 10 and the chance of such case to happen is O(10 −10). Compared to batchsize, usually O(10 3), the amount of such cases can be ignored. Hence, Assigning weights to elements q(z) such as MSS does not make essential change to the analysis above. In addition, β-TCVAE (trained with MSS in our experiments and with MWS in) seems to have an increasing total correlation of mean representation as regularization strength increases (higher β's), as observed by. Here, we provide an explanation to the cause of this problem: Now, consider any strongly correlated z's (e.g., (z 1, z 2) ∼ N (0, Σ), where Σ = 0.01 −0.1 −0.1 1, see Figure 2 ). Then the Gaussian (ground truth) total correlation is arbitrarily large (TC(z 1, z 2) = ∞). This kind of distribution can score a relatively low TC value (for instance lower than z) with estimators such as MSS and MWS by the analysis above. Hence, as β increases, VAE trained with these estimators will be encouraged to obtain some dimensions of very low variance, and these dimensions are easily trapped in a strong correlation with other dimensions (like z 1 and z 2). Figure 2: One dimension of mean has low variance (shutting down), and the distribution is strongly correlated (It appears to be almost flat due to small scale of the shutdown dimension). A sampled distribution (e.g. z|µ ∼ N (µ, 1)) has a very low TC. Shutting down dimension is not preferable because latent dimensions should not be fewer than ground truth. Moreover, considering datasets such as dSprites, though shape is labelled as a single dimension, models can learn to represent complex geometry with multiple dimensions, hence more active dimensions are learned than the number of ground-truth dimensions. Based on the reasons above, we opt for the method of discriminators (density-ratio trick) in our implementation. Experiment shows that density-ratio trick provides a more stable estimation of total correlation when training VAE. The datasets we use include dSprites, Shapes3D and 3D faces. At the time of writing, the scale of our experiments is limited, but there are already some evidence to deliver our arguments. We have scheduled further experiments and tests on larger scale in future works. For every model, we trained with 10 different initialization. Hyperparameter β, also the regularization strength, takes 2, 4, · · ·, 10. For hyperparameter η, we fix η = 10 for a simple reason. Since the variance term in equation 4 becomes small (close to 0) shortly after training begins, this term will not contribute much compared with L beta−T C term. So, η = 10 is enough to strengthen the penalty at the beginning of training. In our experiments, higher value of η cannot bring further improvement since z is close to µ already. And lower values may not guarantee z being close to µ. From experiments, we observe that RTC-VAE has much lower T C mean with different regularization strength than FactorVAE does (Figure 4). And on different datasets, this is also the case (see Appendix). The T C mean behaves almost identically as T C sample in RTC-VAE (see Figure 3 (b) ). The problem of contradictory behaviors of T C mean and T C sample is evidently remedied by RTC-VAE. In addition, the ELBO of RTC-VAE seems to converge faster than FactorVAE as a byproduct (see Figure 5 and Figure 6). Examining the distributions of latent dimensions (mean representation), FactorVAE tends to have some strongly correlated latent dimensions (see Figure 8), and RTC-VAE shows well factorized latent distributions (see Figure 7). We wish there were a widely accepted metric of disentanglement to compare our model RTCVAE with other models. Unfortunately, it is still an open question, how we can measure disentanglement. Various attempts have been made so far, but challenged most of them, indicating that the score under any metric varies due to different initializations and datasets. Here, we analyse several important metrics and attempt to point out some blind spots that have not being considered by these metrics. proposed using a classifier to measure each dimension of latent space and each ground truth factor, e.g. (x, y) coordinates, scale, rotation, etc. glement metric that considers modularity, compactness and explicitness. made analysis on compactness, and compactness mean that each ground truth factor associates with only one or a few latent dimensions. They pointed out that in some situation a perfectly disentangled representation may not be compact (see Section. 3 in). Here we argue that modularity also should be reconsidered. A modular representation means that each dimension of latent space conveys information of at most one ground truth factor. This is exactly the goal attempted by;; , etc. However, multiple latent dimensions can work together to represent multiple ground truth factors meanwhile these latent dimensions are disentangled. For instance, x and y coordinates can be represented by r and θ in polar coordinate system (or any coordinate system under rotation, i.e., (x, y) T = A(x, y) T where A is any orthogonal matrix). These coordinate systems are perfectly disentangled but r (or x) conveys information of both x and y. In this work, we demonstrated that our RTC-VAE, which rectifies the total correlation penalty can remedy its peculiar properties (disparity between total correlation of the samples and the mean representations). Our experiments show that our model has a more reasonable distribution of the mean representation compared with baseline models including β-TCVAE and FactorVAE. We also provide several theoretical proofs which could help diagnose several specific symptoms of entangle-ment. Hopefully, our contributions could add to the explainability of the unsupervised learning of disentangled representations. , A.2 MMS 0 AND MMS 1 There is no mathematical difference between MSS 0 and MSS 1 (same formulation as equation 5), only a difference in implementation. We replace this chunk of code (https://github.com/ rtqichen/beta-tcvae/blob/master/vae_quant.py#L199-L201) to f o r i i n r a n g e (b a t c h s In the following proof, we use the convention of mathematical analysis that the meaning of C can change through lines to eliminate some redundant work of tracking. Theorem (Theorem 1 restated). Let µ ∼ N (0, Σ) and σ j be the standard deviation of µ j, j = 1, · · ·, D, and max j σ j = c 0. For a fixed µ, let z ∼ N (µ, Σ (µ)), where Σ (µ) is diagonal and satisfies that for some R > 0, for some constants c 1, c 2, c 3, c 4. Then TC(z) ≤ C for some C = C(R, c 0, · · ·, c 4, l) > 0. Proof. Let Since KL-divergence is non-negative, if TC(z) + is bounded, then TC(z) must be bounded. In the following, we work on S +, i.e., we assume p(z) ≥ j p(z j). For |z| < R, For |z| > 2R, where r 0 = max(c 0, c 2) and r 1 = min(c 1, c 4), and since l ≥ 1 and, it is easy to see that p(z) < C. And for |z| > R, Hence, Proof. First, recall that the KL-divergence between two distributions P and Q is defined as Also, the density function for a multivariate Gaussian distribution N (µ, Σ) is p(x) = 1 (2π) n/2 det(Σ) 1/2 exp(− 1 2 (x − µ) T Σ −1 (x − µ)).
diagnosed all the problem of STOA VAEs theoretically and qualitatively