1. An image processing apparatus comprising: an extraction unit that extracts representative colors from a target image; a color difference calculation unit that calculates a color difference between the representative color and a pixel of interest of the target image; a change unit that changes a brightness component of the color difference depending on chroma of the representative colors; and a replacement unit that selects the representative color depending on the color difference changed in the change unit to replace a color of the pixel of interest by the representative color.