Information bottleneck [1]} (IB) tends to enforce an upper bound on the mutual information \(I(X,Z)\)  between the latent representation \(Z\)  learned by the encoder and the original input \(X\) . As pointed in [1]}, for a supervised learning task, IB encourages \(Z\)  to be predictive of the label \(Y\) , and simultaneously, push the \(Z\)  to “forget” the original input \(X\)  as much as possible. This is equivalent to upperbound a \(Kullback-Leibler (KL-)\)  divergence between the joint probability \(P(X,Z)\)  and the product of the marginals \(P(X) \times P(Z)\)  to a specific bottleneck value \(I_c\) . Although the information bottleneck principle is appealing, it suffers from the fact that mutual information computation is computationally challenging [3]}, which is especially hard to be instantiated in the context of CNNs. Inspired by a similar approach in variational autoencoders (VAE) [4]}, recent methods [5]}, [6]} implemented the IB in practical deep models by leveraging a variational bound and the reparameterization trick. This paper follows such strategy to instantiate the IB in the context of adversarial learning-based domain adaptation.
<FIGURE>