1. An image processing apparatus for performing filling processing for input image data, comprising: a holding unit which holds, for a block formed from a predetermined number of pixels included in the image data, a representative color of the block generated by compression, a color layout of pixels included in the block, and an interpolation color which is a color other than the representative color included in the block; and a fill unit which performs filling processing of the compressed image data using the representative color, the color layout, and the interpolation color, when filling the block of the compressed image data with a specific color, said fill unit replacing the representative color corresponding to a position based on a type of the filling processing with the specific color and replacing the interpolation color with the same specific color as that of the representative color.