Typically, the step size \(\epsilon _t\)  for sampling has to be small enough for ensuring stability in diffusion. Otherwise, the model will fail to converge to the target distribution [1]}, similar to the situation when applying a numerical solver for ODE (ordinary differential equation) and SDE.
As a result, existing SGMs suffer from
a lengthy sampling process as many iterations are needed.
This is inefficient and unscalable as compared to GANs, especially for the high-resolution image generation.
We conjugate that the inconvergence issue with acceleration is mainly caused by the noise term \(\sqrt{\epsilon _t}\mathbf {z}_t\) , which is under-controlled when \(\epsilon _t\)  is too large. At every step \(t\) , the SGM needs to push the sample \(\mathbf {x}_t\)  closer to the target distribution. The noise term, however, has the risk to counter this effect by
pulling the sample further from the target distribution.
Intuitionally, suppose we add a noise closer to the target distribution, this drifting risk would be alleviated since it is easier for the SGM to take the control of the sampling process. In this work, we show that the convergence of the sampling process can be sped up dramatically when we constraint the noise term closer to the target distribution both in the spatial and frequency domain.
