Generalization Gap in Amortized Inference
The ability of likelihood-based probabilistic models to generalize to unseen
data is central to many machine learning applications such as lossless
compression. In this work, we study the generalizations of a popular class of
probabilistic models - the Variational Auto-Encoder (VAE). We point out the two
generalization gaps that can affect the generalization ability of VAEs and show
that the over-fitting phenomenon is usually dominated by the amortized
inference network. Based on this observation we propose a new training
objective, inspired by the classic wake-sleep algorithm, to improve the
generalizations properties of amortized inference. We also demonstrate how it
can improve generalization performance in the context of image modeling and
lossless compression.