Similar to the image domain, extensions of SinGAN [1]} to generation from a single video were proposed [2]}, [3]}, generating diverse new videos of similar appearance and dynamics to the input video. These too, were outperformed by patch nearest-neighbour methods [4]} in both output quality and speed.
However, these video-based nearest-neighbour methods suffer from drawbacks similar to the image case. While the generated samples are of high quality and look realistic, the main reason for this is that the samples are essentially copies of parts of the original video stitched together. They fail to exhibit motion generalization capabilities.
None of the above-mentioned methods can handle input videos longer than a few dozens frames. Single-video GAN based methods are limited in compute time (e.g., HP-VAE-GAN [2]} takes 8 days to train on a short video of 13 frames), whereas VGPNN [4]} is limited in memory (since each space-time patch in the output video searches for its nearest-neighbor space-time patch in the entire input video, at each iteration).
In contrast, our method can handle any length of input video. While it can generalize well from just a few frames, it can also easily train on a long input video at a fixed and very small memory print, and at reasonable compute time (a few hours per video).
