1. A video decoding apparatus comprising: a first restoration unit performing a predetermined operation of an encoded image and generating a first prediction residue image of the encoded image; an image information detection unit determining whether the encoded image is an R-G-B image or a Y-Cb-Cr image and whether a color component of the encoded image is a reference color component of the R-G-B image; a second restoration unit generating a second prediction residue image of the encoded image, on a basis of the reference color component if the encoded image is the R-G-B image and the color component of the encoded image is not the reference color component; and a deblocking filter unit reducing a block effect of a decoded image of an encoded image restored on a basis of the first prediction residue image and the second prediction residue image.