In practice, \(KL\)  term is intractable to compute. The multiplication of marginal distributions \(p(h^i)p(s^i)\)  is not analytically computable, so we take a sampling approach to simulate it. After getting the a batch of representations \(\lbrace z^i_j=(s^i_j;h^i_j)\rbrace _{j=1}^N\)  in \(i\) -th layer, we randomly permute across the batch for \(\lbrace s^i_j\rbrace _{j=1}^N\)  and \(\lbrace h^i_j\rbrace _{j=1}^N\)  to generate sample batch under distribution \(p(z^i)p(s^i)\) . But direct estimating density ratio \(\frac{p(z^i)}{p(h^i)p(s^i)}\)  is often impossible. Thus, with random samples, we conduct a density ratio method [1]}, [2]}: use an additional classifier \(D(x)\)  that distinguishes between samples from the two distributions, at the bottom of Fig REF :
\(\mathcal {L}_{info}^{uns}(i)&=KL(p(z^i)\Vert p(h^i)p(s^i))\nonumber \\&=TC(z^i)\nonumber \\&=\mathbb {E}_{q(z)}[\log \frac{p(z^i)}{p(h^i)p(s^i)}]\nonumber \\&\approx \mathbb {E}_{q(z)}[\log \frac{D(z^i)}{1-D(z^i)}]\) 
