Indeterminacy in Latent Variable Models: Characterization and Strong Identifiability
Most modern latent variable and probabilistic generative models, such as the
variational autoencoder (VAE), have certain indeterminacies that are
unresolvable even with an infinite amount of data. Recent applications of such
models have indicated the need for strongly identifiable models, in which an
observation corresponds to a unique latent code. Progress has been made towards
reducing model indeterminacies while maintaining flexibility, most notably by
the iVAE (arXiv:1907.04809 [stat.ML]), which excludes many -- but not all --
indeterminacies. We construct a full theoretical framework for analyzing the
indeterminacies of latent variable models, and characterize them precisely in
terms of properties of the generator functions and the latent variable prior
distributions. To illustrate, we apply the framework to better understand the
structure of recent identifiability results. We then investigate how we might
specify strongly identifiable latent variable models, and construct two such
classes of models. One is a straightforward modification of iVAE; the other
uses ideas from optimal transport and leads to novel models and connections to
recent work.