The role of the Predictor model is to generate new frames, each conditioned on its previous frame. At each training iteration we sample a condition frame from the video \(x^n_0\)  and a noisy version of the \((n+k)\) 'th frame (\(x^{n+k}_t\) ), which is to be denoised. The two frames are concatenated along the channels axis before being passed to the model (as in [1]}).
The model is also given an embedding of the temporal difference (i.e frame index difference) between the two frames (\(\phi (k)\) ). This embedding is concatenated to the timestep embedding (\(\phi (t)\) ) of the DDPM. At early training \(k=1\) , and in following iterations it is gradually increased to be sampled at random from \(k=[-3,3]\) . We find that such a curriculum learning approach improves outputs quality (even when at inference \(k=1\) ).
