Diffusion Models for Video Prediction and Infilling
To predict and anticipate future outcomes or reason about missing information
in a sequence is a key ability for agents to be able to make intelligent
decisions. This requires strong temporally coherent generative capabilities.
Diffusion models have shown huge success in several generative tasks lately,
but have not been extensively explored in the video domain. We present
Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to
videos using 3D convolutions, and introduces a new conditioning technique
during training. By varying the mask we condition on, the model is able to
perform video prediction, infilling and upsampling. Since we do not use
concatenation to condition on a mask, as done in most conditionally trained
diffusion models, we are able to decrease the memory footprint. We evaluated
the model on two benchmark datasets for video prediction and one for video
generation on which we achieved competitive results. On Kinetics-600 we
achieved state-of-the-art for video prediction.