RENs: Relevance Encoding Networks
The manifold assumption for high-dimensional data assumes that the data is
generated by varying a set of parameters obtained from a low-dimensional latent
space. Deep generative models (DGMs) are widely used to learn data
representations in an unsupervised way. DGMs parameterize the underlying
low-dimensional manifold in the data space using bottleneck architectures such
as variational autoencoders (VAEs). The bottleneck dimension for VAEs is
treated as a hyperparameter that depends on the dataset and is fixed at design
time after extensive tuning. As the intrinsic dimensionality of most real-world
datasets is unknown, often, there is a mismatch between the intrinsic
dimensionality and the latent dimensionality chosen as a hyperparameter. This
mismatch can negatively contribute to the model performance for representation
learning and sample generation tasks. This paper proposes relevance encoding
networks (RENs): a novel probabilistic VAE-based framework that uses the
automatic relevance determination (ARD) prior in the latent space to learn the
data-specific bottleneck dimensionality. The relevance of each latent dimension
is directly learned from the data along with the other model parameters using
stochastic gradient descent and a reparameterization trick adapted to
non-Gaussian priors. We leverage the concept of DeepSets to capture permutation
invariant statistical properties in both data and latent spaces for relevance
determination. The proposed framework is general and flexible and can be used
for the state-of-the-art VAE models that leverage regularizers to impose
specific characteristics in the latent space (e.g., disentanglement). With
extensive experimentation on synthetic and public image datasets, we show that
the proposed model learns the relevant latent bottleneck dimensionality without
compromising the representation and generation quality of the samples.