Identifiability of deep generative models under mixture priors without auxiliary information
We prove identifiability of a broad class of deep latent variable models that
(a) have universal approximation capabilities and (b) are the decoders of
variational autoencoders that are commonly used in practice. Unlike existing
work, our analysis does not require weak supervision, auxiliary information, or
conditioning in the latent space. Recently, there has been a surge of works
studying identifiability of such models. In these works, the main assumption is
that along with the data, an auxiliary variable $u$ (also known as side
information) is observed as well. At the same time, several works have
empirically observed that this doesn't seem to be necessary in practice. In
this work, we explain this behavior by showing that for a broad class of
generative (i.e. unsupervised) models with universal approximation
capabilities, the side information $u$ is not necessary: We prove
identifiability of the entire generative model where we do not observe $u$ and
only observe the data $x$. The models we consider are tightly connected with
autoencoder architectures used in practice that leverage mixture priors in the
latent space and ReLU/leaky-ReLU activations in the encoder. Our main result is
an identifiability hierarchy that significantly generalizes previous work and
exposes how different assumptions lead to different "strengths" of
identifiability. For example, our weakest result establishes (unsupervised)
identifiability up to an affine transformation, which already improves existing
work. It's well known that these models have universal approximation
capabilities and moreover, they have been extensively used in practice to learn
representations of data.