Thanks to the success of learned image compression, more attempts are verified to see where learning can obtain gain in video compression [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}. DVC [4]} is proposed by replacing each module for the traditional codec as a CNN network. All of these networks are trained in an end-to-end manner and optimized by PSNR or MS-SSIM loss functions. It can be seen as a "deep" version of traditional video coding scheme: an optical flow network (pre-trained spynet [12]}) is utilized to predict motion between a frame and its previous compressed frame; a motion compensation network is applied over warped frames as a learning-based loop filter; finally, motion and residual information are compressed by two auto-encoder style compression networks.
Bidirectional motion prediction method [6]} and multi-frames based unidirectional one [9]} obtain advantages over DVC in general. Hierarchical learned video compression (HLVC)  [8]} extends these methods by combining unidirectional and bi-directional compression with a weighted recurrent quality enhancement network. These methods are all based on the traditional video coding scheme which has some inherent problems: error propagation owing to the use of lossy frames (from lossy reconstruction) as references, inefficient residual compression caused by inaccurate motion prediction, too complicated networks to convergence due to the extra calculations and complicated modules in pre-trained optical flow models.
