Mitigating Out-of-Distribution Data Density Overestimation in Energy-Based Models
Deep energy-based models (EBMs), which use deep neural networks (DNNs) as
energy functions, are receiving increasing attention due to their ability to
learn complex distributions. To train deep EBMs, the maximum likelihood
estimation (MLE) with short-run Langevin Monte Carlo (LMC) is often used. While
the MLE with short-run LMC is computationally efficient compared to an MLE with
full Markov Chain Monte Carlo (MCMC), it often assigns high density to
out-of-distribution (OOD) data. To address this issue, here we systematically
investigate why the MLE with short-run LMC can converge to EBMs with wrong
density estimates, and reveal that the heuristic modifications to LMC
introduced by previous works were the main problem. We then propose a Uniform
Support Partitioning (USP) scheme that optimizes a set of points to evenly
partition the support of the EBM and then uses the resulting points to
approximate the EBM-MLE loss gradient. We empirically demonstrate that USP
avoids the pitfalls of short-run LMC, leading to significantly improved OOD
data detection performance on Fashion-MNIST.