EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling
This work presents strategies to learn an Energy-Based Model (EBM) according
to the desired length of its MCMC sampling trajectories. MCMC trajectories of
different lengths correspond to models with different purposes. Our experiments
cover three different trajectory magnitudes and learning outcomes: 1) shortrun
sampling for image generation; 2) midrun sampling for classifier-agnostic
adversarial defense; and 3) longrun sampling for principled modeling of image
probability densities. To achieve these outcomes, we introduce three novel
methods of MCMC initialization for negative samples used in Maximum Likelihood
(ML) learning. With standard network architectures and an unaltered ML
objective, our MCMC initialization methods alone enable significant performance
gains across the three applications that we investigate. Our results include
state-of-the-art FID scores for unnormalized image densities on the CIFAR-10
and ImageNet datasets; state-of-the-art adversarial defense on CIFAR-10 among
purification methods and the first EBM defense on ImageNet; and scalable
techniques for learning valid probability densities. Code for this project can
be found at https://github.com/point0bar1/ebm-life-cycle.