In general, video generation is a broad field of research
including many areas such as video GANs [1]}, [2]}, [3]}, [4]}, video-to-video translation [5]}, [6]} or autoregressive prediction models [7]}, [8]}, [9]}, [10]}, to name a few. Diffusion models for video generation are fairly recent and most rely on DDPM [11]} framework for image generation, extended to handle videos.
RVD [12]} tackles video prediction by conditioning the generative process on recurrent neural networks.
RaMViD [13]} and MCVD [14]} train an autoregressive model conditioned on previous frames for video prediction and infilling using masking mechanisms.
VDM [15]} introduces unconditional video generation by modifying the Conv2D layers in the basic DDPM UNet to Conv3D, as well as autoregressive generation.
Imagen-Video [16]} extends VDM to text-to-video and also include spatio-temporal superresolution conditioned on upsampled versions of smaller scales.
FDM [17]} modifies DDPM to include temporal attention mechanism and can be conditioned on any number of previous frames.
The above-mentioned methods can synthesize beautiful videos. However, none of them can modify or manipulate an existing input video provided by the user.
Also, all these models are trained on large datasets, some with ridiculous amounts of compute power (VDM using 128-256 TPUs) whereas our training time is much more affordable (a few hours per video on a single GPU).
