The critical problem for extending image stylization methods to stylize videos is to ensure temporal coherence in stylized frames.
By introducing a short-term temporal consistency loss based on the optical flow in the frames, it was shown that the optimization-based image stylization model [1]} can produce stable stylized videos [2]}. However, these methods inherit the low speed of the optimization-based image stylization method. It is preferred to start from learning-based methods for image stylization [3]}, [4]}, [5]}, [6]}, [7]}, [8]}. A successful attempt of this was the model developed by Huang et al. [9]} that uses the image stylization model of Johnson et al.[3]} but introduces a short-term temporal loss for training the generative network. Another successful attempt was carried out by Ruder et al.'s [11]} who developed a model that also uses the image stylization model of Johnson et al.[3]} but introduces both short-term temporal loss and long-term temporal loss. A recurrent neural network was proposed to stabilize the generated video frames [13]}. This approach differs from that of the model of Huang et al. [9]} in the generation process, in which the output of the model at the previous time step is needed as input to the model to generate the output at the current time step. With this setting, the original video frames are required to be input to the model sequentially during training and testing. A patch-based image stylization method [15]} is extended to stylize videos [16]}. Image parts are tracked in different frames based on an optical flow method and inconsistent patches are synthesized. Another method estimates a low-resolution optic flow in every training step and minimizes a compound temporal loss to stabilize the videos [17]}.
