TECHNICAL FIELD

The present document related to video coding and decoding.

BACKGROUND

In spite of the advances in video compression, digital video still accounts for the largest bandwidth use on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the bandwidth demand for digital video usage will continue to grow.

SUMMARY

Various techniques are provided that can be implemented by digital video encoders, transcoders and decoders to use optical flow based refinement of prediction in processing of video.

A first example of a video processing method includes determining a refined prediction sample P′(x,y) at a position (x,y) in a video block by modifying a prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x, y) in a first direction estimated at the position (x,y) and a second gradient component Gy(x, y) in a second direction estimated at the position (x,y) and a first motion displacement Vx(x,y) estimated for the position (x,y) and a second motion displacement Vy(x,y) estimated for the position (x,y), wherein x and y are integer numbers; and performing a conversion between the video block and a bitstream representation of the video block using a reconstructed sample value Rec(x,y) at the position (x,y) that is obtained based on the refined prediction sample P′(x,y) and a residue sample value Res(x,y).

A second example of a video processing method includes determining a refined prediction sample P′(x,y) at a position (x,y) in a video block by modifying a prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x, y) in a first direction estimated at the position (x,y) and a second gradient component Gy(x, y) in a second direction estimated at the position (x,y) and a first motion displacement Vx(x,y) estimated for the position (x,y) and a second motion displacement Vy(x,y) estimated for the position (x,y), where x and y are integer numbers; and encoding a bitstream representation of the video block to include a residue sample value Res(x,y) based on a reconstructed sample value Rec(x,y) at the position (x,y) that is based at least on the refined prediction sample P′(x,y).

A third example of a video processing method includes determining a first motion displacement Vx(x,y) at a position (x,y) and a second motion displacement Vy(x,y) at the position (x,y) in a video block coded using an optical flow based method, wherein x and y are fractional numbers, wherein Vx(x,y) and Vy(x,y) are determined based at least on the position (x,y) and a center position of a basic video block of the video block; and performing a conversion between the video block and a bitstream representation of the current video block using the first motion displacement and the second motion displacement.

A fourth example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) in a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) in the video block, wherein the first gradient component and the second gradient component are based on a final prediction sample value of a prediction sample P(x,y) at the position (x,y), and wherein x and y are integer numbers; and performing a conversion between the video block and a bitstream representation of the video block using a reconstructed sample value Rec(x,y) at the position (x,y) that is obtained based on a residue sample value Res(x,y) added to the final prediction sample value of a prediction sample P(x,y), which is refined using the gradient (Gx(x,y), Gy(x,y)).

A fifth example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) in a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) in the video block, wherein the first gradient component and the second gradient component are based on a final prediction sample value of a prediction sample P(x,y) at the position (x,y), and wherein x and y are integer numbers; and encoding a bitstream representation of the video block to include a residue sample value Res(x,y) based on a reconstructed sample value Rec(x,y) at the position (x,y), wherein the reconstructed sample value Rec(x,y) is based on the residue sample value Res(x,y) added to the final prediction sample value of a prediction sample P(x,y), which is refined using the gradient (Gx(x,y), Gy(x,y)).

A sixth example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) in a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) in the video block, wherein the first gradient component and the second gradient component are based on an intermediate prediction sample value of a prediction sample P(x,y) at the position (x,y), wherein a final prediction sample value of the prediction sample P(x,y) is based on the intermediate prediction sample value, and wherein x and y are integer numbers; and performing a conversion between the video block and a bitstream representation of the video block using a reconstructed sample value Rec(x,y) at the position (x,y) that is obtained based on the final prediction sample value of a prediction sample P(x,y) and a residue sample value Res(x,y).

A seventh example of a video processing method includes determining a first gradient component Gx(x,y) in a first direction estimated at a position (x,y) in a video block and a second gradient component Gy(x,y) in a second direction estimated at the position (x,y) in the video block, wherein the first gradient component and the second gradient component are based on an intermediate prediction sample value of a prediction sample P(x,y) at the position (x,y), wherein a final prediction sample value of the prediction sample P(x,y) is based on the intermediate prediction sample value, and wherein x and y are integer numbers; and encoding a bitstream representation of the video block to include a residue sample value Res(x,y) based on a reconstructed sample value Rec(x,y) at the position (x,y), wherein the reconstructed sample value Rec(x,y) is based on the final prediction sample value of a prediction sample P(x,y) and the residue sample value Res(x,y).

A eighth example of a video processing method includes determining a refined prediction sample P′(x,y) at a position (x,y) in an affine coded video block by modifying a prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x, y) in a first direction estimated at the position (x,y) and a second gradient component Gy(x, y) in a second direction estimated at the position (x,y) and a first motion displacement Vx(x,y) estimated for the position (x,y) and a second motion displacement Vy(x,y) estimated for the position (x,y), wherein the first direction is orthogonal to the second direction, and wherein x and y are integer numbers; determining a reconstructed sample value Rec(x,y) at the position (x,y) based on the refined prediction sample P′(x,y) and a residue sample value Res(x,y); determining a refined reconstructed sample value Rec′(x,y) at the position (x,y) in the affine coded video block, wherein Rec′(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y); and performing a conversion between the affine coded video block and a bitstream representation of the affine coded video block using the refined reconstructed sample value Rec′(x,y).

A ninth example of a video processing method includes determining a refined prediction sample P′(x,y) at a position (x,y) in an affine coded video block by modifying a prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x, y) in a first direction estimated at the position (x,y) and a second gradient component Gy(x, y) in a second direction estimated at the position (x,y) and a first motion displacement Vx(x,y) estimated for the position (x,y) and a second motion displacement Vy(x,y) estimated for the position (x,y), wherein the first direction is orthogonal to the second direction, and wherein x and y are integer numbers; determining a reconstructed sample value Rec(x,y) at the position (x,y) based on the refined prediction sample P′(x,y) and a residue sample value Res(x,y); determining a refined reconstructed sample value Rec′(x,y) at the position (x,y) in the affine coded video block, wherein Rec′(x,y)=Rec(x,y)+Gx(x,y)×Vx(x,y)+Gy(x,y)×Vy(x,y); and encoding a bitstream representation of the affine coded video block to include the residue sample value Res(x,y).

A tenth example of a video processing method includes determining, for a video block in an affine mode, a motion vector with a 1/N pixel precision; determining a motion displacement vector (Vx(x,y), Vy(x,y)) estimated for a position (x,y) in the video block, wherein the motion displacement vector is derived with a 1/M pixel precision, wherein N and M are positive integers, and wherein x and y are integer numbers; and performing a conversion between the video block and a bitstream representation of the video block using the motion vector, and the motion displacement vector.

A eleventh example of a video processing method includes determining two sets of motion vectors for a video block or for a sub-block of the video block, wherein each of the two sets of motion vectors has a different motion vector pixel precision, and wherein the two sets of motion vectors are determined using a temporal motion vector prediction (TMVP) technique or using a sub-block based temporal motion vector prediction (SbTMVP) technique; and performing a conversion between the video block and a bitstream representation of the video block based on the two sets of motion vectors.

A twelfth example of a video processing method includes performing an interweaved prediction technique on a video block coded using an affine coding mode by splitting the video block into multiple partitions using K sub-block patterns, where K is an integer greater than 1; generating prediction samples of the video block by performing a motion compensation using a first of the K sub-block patterns, wherein a prediction sample at a position (x,y) is denoted as P(x,y), wherein x and y are integer numbers; determining, for at least one of remaining of the K sub-block patterns, denoted as Lth pattern, an offset value OL(x,y) at the position (x,y) based on prediction samples derived with a first sub-block pattern, and a difference between motion vectors derived using the first of the K sub-block patterns and the Lth pattern; determining a final prediction sample for the position (x,y) as a function of OL(x,y) and P(x,y); and performing a conversion between a bitstream representation of the video block and the video block using the final prediction sample.

A thirteenth example of a video processing method includes performing a conversion between a bitstream representation of a video block and the video block using a final prediction sample, wherein the final prediction sample is derived from refined intermediate prediction samples by: (a) performing, based on a rule, an interweaved prediction technique followed by an optical flow-based prediction refinement technique, or (b) performing a motion compensation technique.

A fourteenth example of a video processing method includes performing a conversion between a bitstream representation of a video block and the video block using a final prediction sample when bi-prediction is applied, wherein the final prediction sample is derived from refined intermediate prediction samples by: (a) performing an optical flow-based prediction refinement technique, wherein an interweaved prediction technique is disabled, or (b) performing a motion compensation technique.

A fifteenth example of a video processing method includes performing a conversion between a bitstream representation of a video block and the video block using a prediction sample, wherein the prediction sample is derived from refined intermediate prediction samples by performing an optical flow-based prediction refinement technique, wherein the performing the optical-flow based prediction refinement technique depends on only of a first set of motion displacement Vx(x,y) estimated in a first direction for the video block or a second set of motion displacement Vy(x,y) estimated in a second direction for the video block, wherein x and y are integer numbers, and wherein the first direction is orthogonal to the second direction.

A sixteenth example of a video processing method includes obtaining a refined motion vector for a video block by refining a motion vector for the video block, wherein the motion vector is refined prior to performing a motion compensation technique, wherein the refined motion vector has a 1/N pixel precision, and wherein the motion vector has a 1/M pixel precision; obtaining a final prediction sample by performing an optical flow-based prediction refinement technique on the video block, wherein the optical flow-based prediction refinement technique is applied to a difference between the refined motion vector and the motion vector; and performing a conversion between a bitstream representation of the video block and the video block using the final prediction sample.

A seventeenth example of a video processing method includes determining, using a multi-step decoder-side motion vector refinement process a video block, a final motion vector, wherein the final motion vector has a 1/N pixel precision; and performing conversion between the current block and the bitstream representation using the final motion vector.

A eighteenth example of a video processing method includes obtaining refined intermediate prediction samples of a video block by performing an interweaved prediction technique and an optical flow-based prediction refinement technique on intermediate prediction samples of the video block; deriving a final prediction sample from the refined intermediate prediction samples; and performing a conversion between a bitstream representation of the video block and the video block using the final prediction sample.

A nineteenth example of a video processing method includes obtaining refined intermediate prediction samples of a video block by performing an interweaved prediction technique and an phase-variant affine subblock motion compensation (PAMC) technique on intermediate prediction samples of the video block; deriving a final prediction sample from the refined intermediate prediction samples; and performing a conversion between a bitstream representation of the video block and the video block using the final prediction sample.

A twentieth example of a video processing method includes obtaining refined intermediate prediction samples of a video block by performing an optical flow-based prediction refinement technique and an phase-variant affine subblock motion compensation (PAMC) technique on intermediate prediction samples of the video block; deriving a final prediction sample from the refined intermediate prediction samples; and performing a conversion between a bitstream representation of the video block and the video block using the final prediction sample.

A twenty-first example of a video processing method includes determining, during a conversion between a video block and a bitstream representation of the video block, a refined prediction sample P′(x,y) at a position (x,y) in the video block by modifying a prediction sample P(x,y) at the position (x,y) as a function of gradients in a first direction and/or a second direction estimated at the position (x,y) and a first motion displacement and/or a second motion displacement estimated for the position (x,y), and performing the conversion using a reconstructed sample value Rec(x,y) at the position (x,y) from the refined prediction sample P′(x,y).

A twenty-second example of a video processing method includes determining a first displacement vector Vx(x,y) and a second displacement vector Vy(x,y) are at a position (x,y) in a video block corresponding to an optical flow based method of coding the video block based on information from a neighboring block or a basic block, and performing a conversion between the video block and a bitstream representation of the current video block using the first displacement vector and the second displacement vector.

A twenty-third example of a video processing method includes determining, during a conversion between a video block and a bitstream representation of the video block, a refined prediction sample P′(x,y) at a position (x,y) in the video block by modifying a prediction sample P(x,y) at the position (x,y), and wherein a gradient in a first direction and a gradient in a second direction at the position (x,y) are determined based on a final prediction value determined from the refined prediction sample P′(x,y) and a residual sample value at the position (x,y), and performing the conversion using the gradient in the first direction and the gradient in the second direction.

A twenty-fourth example of a video processing method includes determining a reconstructed sample Rec(x,y) at a position (x,y) in a video block that is affined coded, refining Rec(x,y) using first and second displacement vectors and first and second gradients at the position (x,y) to obtained a refined reconstruction sample Rec′(x,y, and performing a conversion between the video block and a bitstream representation of the current video block using the refined reconstruction sample.

A twenty-fifth example of a video processing method includes performing, during a conversion between a video block that is coded using an affine coding mode and a bitstream representation of the video block, an interweaved prediction of the video block by splitting the video block into multiple partitions using K sub-block patterns, where K is an integer greater than 1, performing a motion compensation using a first of the K sub-block patterns to generate prediction samples of the video block, wherein a prediction sample at a position (x,y) is denoted as P(x,y), determining, for at least one of remaining of the K sub-block patterns, denoted as Lth pattern, an offset value OL(x,y) at the position (x,y) based on P(x,y) and a different between motion vectors derived using the first of the K sub-blocks and the Lth pattern, determining a final prediction sample for the position (x,y) as a function of OL(x,y) and P(x,y); and performing the conversion using the final prediction sample.

In yet another example aspect, a video encoder apparatus configured to implement one of the methods described in this patent document is disclosed.

In yet another example aspect, a video decoder apparatus configured to implement one of the methods described in this patent document is disclosed.

In yet another aspect, a computer-readable medium is disclosed. A processor-executable code for implementing one of the methods described in this patent document is stored on the computer-readable medium. Thus, non-transitory computer readable medium having code for implementing a method recited in any of the above-described methods and methods described in this patent document.

These, and other aspects are described in detail in the present document.

