Diffusion models as plug-and-play priors
We consider the problem of inferring high-dimensional data $\mathbf{x}$ in a
model that consists of a prior $p(\mathbf{x})$ and an auxiliary constraint
$c(\mathbf{x},\mathbf{y})$. In this paper, the prior is an independently
trained denoising diffusion generative model. The auxiliary constraint is
expected to have a differentiable form, but can come from diverse sources. The
possibility of such inference turns diffusion models into plug-and-play
modules, thereby allowing a range of potential applications in adapting models
to new domains and tasks, such as conditional generation or image segmentation.
The structure of diffusion models allows us to perform approximate inference by
iterating differentiation through the fixed denoising network enriched with
different amounts of noise at each step. Considering many noised versions of
$\mathbf{x}$ in evaluation of its fitness is a novel search mechanism that may
lead to new algorithms for solving combinatorial optimization problems.