As shown in Fig. REF , in the compression process, the data encoding has two modes, intra-mode and inter-mode. Following the traditional hybrid video coding codecs [1]} and the exiting learning-based methods [2]}, [3]}, we first divide the original video sequence into groups of pictures (GoP). Let \(x = \lbrace x_1, x_2,...,x_t, x_{t+1},..., x_N\rbrace \)  denote the frames of one GoP unit, where \(N\)  means the GoP length. Assumed that \(x_t\)  has been coded by intra-mode, in the next inter-mode coding process, \(x_{t+1}, x_{t+2},..., x_N\)  is encoded frame-by-frame in a sequential order.
