Existing unsupervised video-to-video translation methods fail to produce translated videos which are frame-wise realistic, semantic information preserving and video-level consistent. In this work, we propose a novel unsupervised video-to-video translation model. Our model decomposes the style and the content, uses specialized encoder-decoder structure and propagates the inter-frame information through bidirectional recurrent neural network (RNN) units. The style-content decomposition mechanism enables us to achieve long-term style-consistent video translation results as well as provides us with a good interface for modality flexible translation. In addition, by changing the input frames and style codes incorporated in our translation, we propose a video interpolation loss, which captures temporal information within the sequence to train our building blocks in a self-supervised manner. Our model can produce photo-realistic, spatio-temporal consistent translated videos in a multimodal way. Subjective and objective experimental results validate the superiority of our model over the existing methods. Recent image-to-image translation (I2I) works have achieved astonishing results by employing Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) . Most of the GAN-based I2I methods mainly focus on the case where paired data exists (Isola et al. (2017b) , , Wang et al. (2018b) ). However, with the cycle-consistency loss introduced in CycleGAN , promising performance has been achieved also for the unsupervised image-to-image translation (Huang et al. (2018) , Almahairi et al. (2018) , Liu et al. (2017) , Mo et al. (2018) , Romero et al. (2018) , Gong et al. (2019) ). While there is an explosion of papers on I2I, its video counterpart is much less explored. Compared with the I2I task, video-to-video translation (V2V) is more challenging. Besides the frame-wise realistic and semantic preserving requirements, which is also required in the I2I task, V2V methods additionally need to consider the temporal consistency issue for generating sequence-wise realistic videos. Consequently, directly applying I2I methods on each frame of the video will not work out as those methods fail to utilize temporal information and can not assure any temporal consistency within the sequence. In their seminal work, Wang et al. (2018a) combined the optical flow and video-specific constraints and proposed a general solution for V2V in a supervised way. Their sequential generator can generate long-term high-resolution video sequences. However, their vid2vid model (Wang et al. (2018a) ) relies heavily on labeled data. Based on the I2I CycleGAN approach, previous methods on unsupervised V2V proposed to design spatio-temporal translator or loss to achieve more temporally consistent results while preserving semantic information. In order to generate temporally consistent video sequences, Bashkirova et al. (2018) proposed a 3DCycleGAN method which adopts 3D convolution in the generator and discriminator of the CycleGAN framework to capture temporal information. However, since the small 3D convolution operator (with temporal dimension 3) only captures dependency between adjacent frames, 3DCycleGAN can not exploit temporal information for generating long-term consistent video sequences. Furthermore, the vanilla 3D discriminator is also limited in capturing complex temporal relationships between video frames. As a result, when the gap between input and target domain is large, 3DCycleGAN tends to sacrifice the image-level reality and generates blurry and gray outcomes. Recently, Bansal et al. (2018) designed a Recycle loss for jointly modeling the spatio-temporal relationship between video frames. They trained a temporal predictor to predict the next frame based on two past frames, and plugged the temporal predictor in the cycle-loss to impose the spatio-temporal constraint on the traditional image translator. As the temporal predictors can be trained from video sequences in source and target domain in a self-supervised manner, the recycle-loss is more stable than the 3D discriminator loss proposed by Bashkirova et al. (2018) . The RecycleGAN method of Bansal et al. (2018) achieved state-of-the-art unsupervised V2V results. Despite its success in translation scenarios with less variety, such as faceto-face or flower-to-flower, we have experimentally found that applying RecycleGAN to translate videos between domains with a large gap is still challenging. We think the following two points are major reasons which affect the application of RecycleGAN method in complex scenarios. On one hand, the translator in Bansal et al. (2018) processes input frames independently, which has limited capacity in exploiting temporal information; and on the other hand, its temporal predictor only imposes the temporal constraint between a few adjacent frames, the generated video content still might shift abnormally: a sunny scene could change to a snowy scene in the following frames. In a concurrent work, incorporate optical flow to add motion cycle consistency and motion translation constraints. However, their Motion-guided CycleGAN still suffers from the same two limitations as the RecycleGAN method. In this paper, we propose UVIT, a novel method for unsupervised video-to-video translation. We assume that a temporally consistent video sequence should simultaneously be: 1) long-term style consistent and 2) short-term content consistent. Style consistency requires the whole video sequence to have the same style, it ensures the video frames to be overall realistic; while the content consistency refers to the appearance continuity of contents in adjacent video frames and ensures the video frames to be dynamically vivid. Compared with previous methods which mainly focused on imposing short-term consistency between frames, we have considered in addition the long-term consistency issue which is crucial to generate visually realistic video sequences. Figure 1: Overview of our proposed UVIT model: given an input video sequence, we first decompose it to the content by Content Encoder and the style by Style Encoder. Then the content is processed by special RNN units-TrajGRUs (Shi et al., 2017) to get the content used for translation and interpolation recurrently. Finally, the translation content and the interpolation content are decoded to the translated video and the interpolated video together with the style latent variable. We depict here the video translation loss (orange), the cycle consistency loss (violet), the video interpolation loss (green) and the style encoder loss (blue). To simultaneously impose style and content consistency, we adopt an encoder-decoder architecture as the video translator. Given an input frame sequence, a content encoder and a style encoder firstly extract its content and style information, respectively. Then, a bidirectional recurrent network propagates the inter-frame content information. Updating this information with the single frame content information, we get the spatio-temporal content information. At last, making use of the conditional instance normalization (Dumoulin et al. (2016) , Perez et al. (2018) ), the decoder takes the style information as the condition and utilizes the spatio-temporal content information to generate the translation result. An illustration of the proposed architecture can be found in figure 1 . By applying the same style code to decode the content feature for a specific translated video, we can produce a long-term consistent video sequence, while the recurrent network helps us combine multi-frame content information to achieve content consistent outputs. The conditional decoder also provides us with a good interface to achieve modality flexible video translation. Besides using the style dependent content decoder and bidirectional RNNs to ensure long-term and short-term consistency, another advantage of the proposed method lies in our training strategy. Due to our flexible Encoder-RNN-Decoder architecture, the proposed translator can benefit from the highly structured video data and being trained in a self-supervised manner. Concretely, by removing content information from frame t and using posterior style information, we use our Encoder-RNNDecoder translator to solve the video interpolation task, which can be trained by video sequences in each domain in a supervised manner. In the RecycleGAN method, Bansal et al. (2018) proposed to train video predictors and plugged them into the GAN losses to impose spatio-temporal constraints. They utilize the structured video data in an indirect way: using video predictor trained in a supervised way to provide spatio-temporal loss for training video translator. In contrast, we use the temporal information within the video data itself, all the components, i.e. Encoders, RNNs and Decoders, can be directly trained with the proposed video interpolation loss. The processing pipelines of using our Encoder-RNN-Decoder architecture for the video interpolation and translation tasks can be found in figure 2, more details of our video interpolation loss can be found in section 2. The main contributions of our paper are summarized as follows: 1. a novel Encoder-RNN-Decoder framework which decomposes content and style for temporally consistent and modality flexible unsupervised video-to-video translation; 2. a novel video interpolation loss that captures the temporal information within the sequence to train translator components in a self-supervised manner; 3. extensive experiments showing the superiority of our model at both video and image level. In this paper, we have proposed UVIT, a novel method for unsupervised video-to-video translation. A novel Encoder-RNN-Decoder architecture has been proposed to decompose style and content in the video for temporally consistent and modality flexible video-to-video translation. In addition, we have designed a video interpolation loss which utilizes highly structured video data to train our translators in a self-supervised manner. Extensive experiments have been conducted to show the effectiveness of the proposed UVIT model. Without using any paired training data, the proposed UVIT model is capable of producing excellent multimodal video translation results, which are image-level realistic, semantic information preserving and video-level consistent.
A temporally consistent and modality flexible unsupervised video-to-video translation framework trained in a self-supervised manner.
{"src": "scitldr-AIC_train", "paper_id": "HkevCyrFDS"}