Directly training the standard DDPM [1]}
on the single image or its large crops results in "overfitting", namely the model only generates the same image crops.
We postulate that this phenomenon occurs because of the receptive field of the core backbone network in DDPM, which is the entire input image.
To this end we modify the backbone UNet [2]} network of DDPM, in order to reduce the size of its receptive field.
We remove
the attention layers as they have global receptive field. We also remove the downsampling and upsampling layers which cause the receptive field to grow too rapidly.
Removing the attention layers has an unwanted side-effect - harming the performance of the diffusion model.
[3]} proposed a fully convolutional network that matches the attention mechanism on many vision tasks. Inspired by this idea, we replace the ResNet [4]} blocks in the network with ConvNext [3]} blocks. This architectural choice is meant to replace the functionality of the attention layers, while keeping a non-global receptive field. It also has the advantage of reducing computation time. The overall receptive field of our network is then determined by the number of ConvNext blocks in the network. Changing the number of ConvNext blocks allows us to control the diversity of the output samples.
Please see further analysis and hyperparameter choice in sec:effectofcsandrf.
The rest of our backbone network is similar to DDPM, as well as the embedding network (\(\phi \) ) which is used to incorporate the diffusion timestep \(t\)  into the model (and will be later used to embed the video frame difference, see sec:predictor). See fig:backbone for details.
