1. Field of the Invention
The present invention relates to an illumination compensation method and apparatus, and more particularly, to a method and apparatus for compensating for illumination for image encoding and decoding.
2. Description of the Related Art
In multi-view coding (MVC) for 3-dimensional (3D) display applications, when prediction between neighboring views is performed, illumination changes between the neighboring views occur due to an incorrectly calibrated camera, different perspective projection directions, and different reflection effects, and as a result, encoding efficiency is lowered. In addition, in the case of a single view, the encoding efficiency is lowered due to illumination changes caused by screen switching and the like.
In order to solve these problems, the H.264 standard uses a weighted prediction technique.
This weighted prediction technique is applied to motion compensation at a slice level, and illumination is compensated for according to an appropriate weighted factor W and an additional offset O. An improved method for this weighted prediction technique is illumination change-adaptive motion estimation/compensation (ICA ME/MC).
According to ICA ME/MC, in the case of a Y component in the YUV color format, ICA ME/MC is performed in units of 16×16 blocks, and a difference value of illumination change (DVIC) in relation to each macroblock is obtained.
In ICA ME/MC, there are two modes. One is an IC-inter 16×16 mode which uses ICA ME/MC, and is used in a P or B slice. The other mode is an IC-direct 16×16 mode which does not use ICA ME/MC and is used only in a B slice. In order to compensate for local illumination change, a 1-bit flag, i.e., mb_ic_flag, is required for each of the inter 16×16 block mode and the direct 16×16 block mode.
Since there are very high correlations between the DVIC of a current block and the DVICs of neighboring blocks, the DVIC of the current block is transmitted by obtaining and encoding the difference between the DVIC of the current block and the DVICs of the neighboring blocks.
The ICA ME/MC in units of macroblocks for an inter 16×16 mode will now be explained.
For ICA ME/MC, a new sum of absolute differences (SAD) should be defined. Assuming that a pixel at coordinates (i,j) of a current frame is f(i,j), and a pixel at coordinates (i,j) of a reference frame is r(i,j), the SAD of blocks of a size of S×T is calculated according to Equation 1 below. Here, S×T may be 16×16, 16×8, 8×16, 8×8, 8×4, 4×8, and 4×4.
                              SAD          ⁡                      (                          x              ,              y                        )                          =                              ∑                          i              =              m                                      m              +              S              -              1                                ⁢                                    ∑                              j                =                n                                            n                +                T                -                1                                      ⁢                                                                          f                  ⁡                                      (                                          i                      ,                      j                                        )                                                  -                                  r                  ⁡                                      (                                                                  i                        +                        x                                            ,                                              j                        +                        y                                                              )                                                                                                                        (                  Equation          ⁢                                          ⁢          1                )            Here, (x,y) is a candidate motion vector, and (m,n) is the position of a current block.
In order to compensate for illumination change, a new SAD is required. The new SAD is obtained according to Equations 2 and 3 below:
                                          M            cur                    =                                    1                              S                ×                T                                      ⁢                                          ∑                                  i                  =                  m                                                  m                  +                  S                  -                  1                                            ⁢                                                ∑                                      j                    =                    n                                                        n                    +                    T                    -                    1                                                  ⁢                                  f                  ⁡                                      (                                          i                      ,                      j                                        )                                                                                      ⁢                                  ⁢                                            M              ref                        ⁡                          (                              p                ,                q                            )                                =                                    1                              S                ×                T                                      ⁢                                          ∑                                  i                  =                  p                                                  p                  +                  S                  -                  1                                            ⁢                                                ∑                                      j                    =                    q                                                        q                    +                    T                    -                    1                                                  ⁢                                  r                  ⁡                                      (                                          i                      ,                      j                                        )                                                                                                          (                  Equation          ⁢                                          ⁢          2                )            Here, Mcur is the mean value of pixels of a current block, and Mref is the mean value of pixels in a reference block. Also, (p,q) is the position of the reference block. The new SAD, i.e., NewSAD(x,y) is obtained according to Equation 3 below:
                              NewSAD          ⁡                      (                          x              ,              y                        )                          =                              ∑                          i              =              m                                      m              +              S              -              1                                ⁢                                    ∑                              j                =                n                                            n                +                T                -                1                                      ⁢                                                                          {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          M                      cur                                                        }                                -                                  {                                                            r                      ⁡                                              (                                                                              i                            +                            x                                                    ,                                                      j                            +                            y                                                                          )                                                              -                                                                  M                        ref                                            ⁡                                              (                                                                              m                            +                            x                                                    ,                                                      n                            +                            y                                                                          )                                                                              }                                                                                                      (                  Equation          ⁢                                          ⁢          3                )            
In ICA ME/MC, a block which minimizes the NewSAD(x,y), for example, a 16×16 block, is searched for based on Equation 3, and a motion vector (MV) corresponding to the block is found.
If the motion vector MV(x′,y′) minimizing the NewSAD(x,y) is determined, an illumination compensation residual signal NewR(i,j) is determined according to Equation 4 below:
                                                                        NewR                ⁡                                  (                                      i                    ,                    j                                    )                                            =                            ⁢                                                {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          M                      cur                                                        }                                -                                  {                                                            r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                              -                                                                                                                                        ⁢                                                M                  ref                                ⁡                                  (                                                            m                      +                                              x                        ′                                                              ,                                          n                      +                                              y                        ′                                                                              )                                            }                                                                          =                            ⁢                                                {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                                              }                                -                                                                                                      ⁢                              {                                                      M                    cur                                    -                                      M                    ref                                                                                                                                        ⁢                              (                                                      m                    +                                          x                      ′                                                        ,                                      n                    +                                          y                      ′                                                                      )                            }                                                                          =                            ⁢                                                {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                                              }                                -                DVIC                                                                        (                  Equation          ⁢                                          ⁢          4                )            
In this case, the mb_ic_flag that is the 1-bit flag is stored in a syntax in order to indicate whether or not ICA ME/MC is used. The DPCM value of the DVIC is also included in the syntax. In the current example, if the mb_ic_flag is 0, it indicates that ICA MC for a current block has not been performed. If the mb_ic_flag is 1, it indicates that ICA MC for the current block has been performed.
Also, if the mb_ic_flag is 1, the ICA ME/MC unit of a decoding apparatus obtains a restored pixel by using Equation 5 below:
                                                                                          f                  ′                                ⁡                                  (                                      i                    ,                    j                                    )                                            =                            ⁢                                                {                                                                                    NewR                        ″                                            ⁡                                              (                                                                              x                            ′                                                    ,                                                      y                            ′                                                    ,                          i                          ,                          j                                                )                                                              +                                          r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                                              }                                +                                                                                                      ⁢                              {                                                      M                    cur                                    -                                                            M                      ref                                        ⁡                                          (                                                                        m                          +                                                      x                            ′                                                                          ,                                                  n                          +                                                      y                            ′                                                                                              )                                                                      }                                                                                        =                            ⁢                                                {                                                                                    NewR                        ″                                            ⁡                                              (                                                                              x                            ′                                                    ,                                                      y                            ′                                                    ,                          i                          ,                          j                                                )                                                              +                                          r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                                              }                                +                                                                                                      ⁢              DVIC                                                          (                  Equation          ⁢                                          ⁢          5                )            Here, NewR″(i,j) is a restored illumination compensation residual signal, and f′(i,j) is a restored pixel in a current frame.
In this conventional ICA ME/MC method, DVIC information should be transmitted, and as a result, the encoding efficiency is lowered.
A method of compensating for illumination capable of omitting DVIC information by improving the conventional illumination method has been suggested.
In order to omit the DVIC information, an illumination-compensated reference block is generated by using Equation 6 below:f(i,j)=ax,y·r′x,y(i,j)+bx,y  (Equation 6)
In Equation 6, ax,y and bx,y have values varying with respect to a motion vector (x,y), and have constant values in relation to each motion vector. Also, r′x,y(i,j) is a motion compensated reference block, and {circumflex over (f)}(i,j) is an illumination-compensated reference block. In the illumination compensation method using Equation 6, calculation of ax,y and bx,y is performed by using restored pixels neighboring a current block according to a linear regression method, an average of difference based prediction method, and a difference of average based prediction method. Accordingly, the DVIC information can be omitted. However, the illumination compensation method omitting the DVIC information calculates ax,y and bx,y from the restored pixels neighboring the current block, and therefore the performance of illumination change prediction by the method is lower than the performance by a method of calculating ax,y and bx,y by using the current block.