In this paper, we mainly discuss an open challenge that whether we could achieve competitive performance with dominant learned video compression methods in a simple framework. Firstly, we compare the dominant framework of learned based methods and that of traditional video coding schemes. It is shown that the current learned one suffers from its complex structure and error propagation problems. Secondly, we discuss some efforts, e.g., error propagation aware training, and implicit flow network, to solve these inherent issues. Although the methods have brought some improvement, little attention has been paid to the framework itself. Finally, a learned image compression style framework is proposed in this paper, called motion-free video compression (MFVC). It utilizes a variable-rate auto-encoder and a spatiotemporal entropy model to outperform SOTA performance under the metric of MS-SSIM. Last but not the least, the MFVC indicates that we may attempt more advanced sequence technologies, e.g., ConvLSTMs [1]}, 3D Convolution, explicit/implicit optical flow, and transformers, on entropy model rather than on raw pixels. More experiments results are present in the Appendix.
