1. An image processing apparatus comprising: a memory that stores instructions; a processor configured to execute the instructions to act as: an extraction unit that extracts representative colors from a target image; a color difference calculation unit that calculates a color difference between each of the representative colors and a pixel of interest of the target image; a change unit that changes a brightness component of the color difference depending on chroma of the representative colors; and a replacement unit that selects one of the representative colors depending on the color difference changed in the change unit to replace a color of the pixel of interest by the selected one of the representative colors.