Energy-Based Models. Early forms of EBMs include the exponential family distribution, the FRAME model [1]} and Restricted Boltzmann Machines [2]}. Recent work has introduced the EBM with a ConvNet potential [3]}, [4]}. This dramatically increased the learning capacity of the model which led to many follow-up works on image synthesis [5]}, [6]}, [7]}, adversarial robustness [8]}, and joint learning of discriminative and generative models [9]}. Several works investigate training and EBM in tandem with an auxilary model. [10]} train an EBM and generator and tandem without MCMC by using samples from the generator as direct approximations of the EBM density and training the generator using a variational objective. A similar approach is explored by [11]}. Cooperative learning [12]} trains the EBM and generator by using the generator to initialize samples needed to train the EBM and uses reconstruction loss between generator and EBM samples to learn the generator. [13]} learn an EBM using Noise Contrastive Estimation with an auxiliary flow model. [14]} use a pretrained VAE to facilitate EBM learning. Our work builds on cooperative learning by identifying and resolving symmetry breaking problems in early training, leading to state-of-the-art EBM synthesis for unconditional ImageNet. Despite the formulation of the EBM as an unnormalized density, it has been shown that most EBMs have strong misaligned steady-state distributions [15]}. Our work introduces new methods to learn a model with correct steady-state alignment.
