In this work, we are primarily interested in learning density models that assign the majority of probability mass in realistic regions of the image space. This goal is, surprisingly, distinct from the goal of likelihood maximization. In particular, one can achieve high likelihood with mixture models where only an infinitesimal portion of mass is assigned to a mixture component that approximates the true density, while the majority of probability mass is assigned to a degenerate distribution (see [1]}, “Great Log Likelihood but Poor Samples"). The steady-state of MCMC sampling with such a mixture distribution would concentrate on the degenerate distribution even though the structures of the true density exist in high-energy regions. Therefore, log likelihood cannot detect if a model has assigned probability mass in realistic region of the image space. Recent observations have shown that this situation is not just hypothetical [2]}. Non-convergent EBMs are practical examples, since these models consist of a mixture of partially formed high-energy energy basins enabling effective shortrun synthesis in realistic regions of the image space and much lower-energy basins in unrealistic image regions that dominate the probability mass. The same misaligned landscape structure extends beyond EBM to RBMs [3]}, score models, normalizing flows, and diffusion models (see Appendix REF ). Log likelihood can only indicate the presence of energy landscape features that are similar to the ground truth landscape, but it cannot detect whether these features will leak into lower-energy basins that represent the true mass distribution. Like shortrun sample quality, high log likelihood is often misleading false evidence that a model density concentrates on realistic images.
