To address the above challenges, this paper advocates a unified formulation, distributional control of generative models, which enables controllability by incorporating the knowledge of off-the-shelf models (e.g., CLIP [1]}, classifiers, or inverse graphics models [2]}).
Based on this unified view, we propose to learn
distributions in the latent space of a pre-trained generative model while keeping the pre-trained weights fixed.
Given its conceptual similarity to prompt learning [3]}, [4]}, [5]}, [6]}, we term our framework as Generative Visual Prompt (PromptGen).
PromptGen requires no training data, and the only supervision comes from off-the-shelf models that help define the control.
Specifically, PromptGen allows the user to define controls using an energy-based model (EBM) approximated by an invertible neural network (INN).
Unlike methods that require optimization at inference in EBM sampling [7]}, [8]}, [9]}, [10]}, PromptGen samples images in a feed-forward manner, which is highly efficient. Moreover, PromptGen stands alone at inference, meaning that we can discard the off-the-shelf models, which define the control, after training.
