Information Theory has been proposed by Shannon in 1948 [1]}, which came from communication research. Mutual information is the fundamental metric for measuring the relationship about information between random variables. In representation learning, it has been applied widely [2]}, [3]}, [4]}, [5]}, with graph network [6]}, [7]}, and gets some explanatory meaning on machine learning [8]}. We can conclude the application as two ideas: The first one is Information Maximization Principle(InfoMax) [9]}, [10]}, which enforces representation to preserve more information about the input data through the transformers (CNN, GNN); some works [3]}, [4]}, [13]} regularize their original model with InfoMax term to get more informative and interpretable model. The other one is the Information Bottleneck(IB) theory [8]}, [15]}, [16]}. It analyzes the process of information transmission and the loss through the networks.
IB theory considers the network process as a Markov chain and uses the Data Processing Inequality (DPI) [17]} to explain the variation of information in deep networks. In 2015, Variational Information Bottleneck (VIB) method [18]} offers a variational form of supervised IB theory.
Also, IB theory has been revealed a unique ability [19]} to explain how and why VAEs models design this architecture.
With this knowledge of disentanglement and information, we initiate our model, blocked and hierarchical variational autoencoder (BHiVAE), completely from information theory perspective to get better interpretability and controllability. In BHiVAE, because of the neural network's different ability to extract features with different net depth, we locate data factors into different layers. Furthermore, the weak expressiveness of single-neuron pushes us to use neuron blocks to represent features. We also discuss the supervised and unsupervised version model. In the supervised model, we utilize the label to separate the representation from feature information. In the unsupervised model, we give out a unique prior distribution to better meet our model and use additional discriminators to split information. Of course we give enough experiments in MNIST [20]}, CelebA [21]} and dSprite [22]} datasets to show the great performance in disentanglement. In summary, our work mainly makes the following contributions:
