In the project page we show the results of Video-Diffusion-Models (VDM) [1]} trained on a single video. Since the official implementation was not published at the time of writing, we use a third-party implementation https://github.com/lucidrains/video-diffusion-pytorch). Since VDM expects a dataset of videos, we slice a long video of 420 frames into 42 short videos of 10 frames each and let VDM train on those videos. We could only train the model on a resolution of \(64\times 64\)  pixels before exceeding the memory of our GPU. We trained the model for about 150 epochs using two different learning rates (total run time was about a day on a V100 GPU).
The results seems to capture the motions of the original videos, but it is difficult to evaluate since the resolution is too low compared to the original videos. The results also contain artifacts that may result from the low amount of data usually needed to train such models (without including our proposed modifications). It is qualitatively evident from the results that our framework, SinFusion, generates outputs of much higher quality when trained on a single video.
For video results please visit our project page.
