Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. While recent generative models of video have had some success, current progress is hampered by the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples. To this extent we propose Fréchet Video Distance (FVD), a new metric for generative models of video based on FID. We contribute a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos. Recent advances in deep generative models have lead to remarkable success in synthesizing highquality images (Karras et al., 2018; Brock et al., 2018) . A natural next challenge is to consider video generation. This is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. Generative models of video will enable many applications, including missing-frame prediction (Jiang et al., 2018) , improved instance segmentation (Haller & Leordeanu, 2017) , or complex (relational) reasoning tasks by conducting inference (Lerer et al., 2016) .While great progress has been made in recent years, video generation models are still in their infancy, and generally unable to synthesize more than a few seconds of video (Babaeizadeh et al., 2017) . Learning a good dynamics model remains a major challenge in generating real world videos. However, in order to qualitatively measure progress in synthesizing videos, we require metrics that consider visual quality, temporal coherence, and diversity of generated samples.We contribute Fréchet Video Distance (FVD), a new metric for generative models of video. FVD builds on the principles underlying Fréchet Inception Distance (FID; Heusel et al. (2017) ), which has been successfully applied to images. We introduce a feature representation that captures the temporal coherence of the content of a video, in addition to the quality of each frame. Unlike popular * Both authors contributed equally to this work while interning at Google Brain.Figure 1: Generated videos by various models ranked according to FVD (lower is better). metrics such as Peak Signal to Noise Ratio (PSNR) or the Structural Similarity (SSIM; Wang et al. (2004) ) index, FVD considers a distribution over videos, thereby avoiding the drawbacks of framelevel metrics (Huynh-Thu & Ghanbari, 2012) . We contribute extensive experiments to evaluate FVD, including a large-scale human study which confirms that FVD coincides well with qualitative human judgment of generated videos. We introduced the Fréchet Video Distance (FVD), a new evaluation metric for generative models of video, and an important step towards better evaluation of models for video generation. Our experiments confirm that FVD is accurate in evaluating videos that were modified to include static noise, and temporal noise. More importantly, a large scale human study among generated videos from several recent generative models reveals that FVD consistently outperforms SSIM and PSNR in agreeing with human judgment.
We propose FVD: a new metric for generative models of video based on FID. A large-scale human study confirms that FVD correlates well with qualitative human judgment of generated videos.
{"src": "scitldr-AIC_train", "paper_id": "rylgEULtdN"}