The plug-and-play generative model [1]} was first proposed to use a latent-space EBM for controllable image synthesis. If a fixed generative model is used, then theoretically, any image-space EBM can be viewed as a latent-space EBM, as shown by [2]} and [3]}. We define the latent-space EBM following similar formulation as [1]}, [2]}, [3]}, with some new energy functions.
We then extend the formulation to incorporate the moment constraint [7]}, which was adopted for language modeling [8]}, but unlike [8]}, we define the moment constraint in the latent space to accommodate generative vision models.
