1. An image processing apparatus for applying an adjusting process to an image including multicolor image signals, comprising: an extracting unit arranged to extract an image area including a pixel to be processed; a generation unit arranged to generate one representative signal value from signal values corresponding to a plurality of colors of pixels included in the image area; a feature amount calculation unit arranged to calculate a feature amount associated with the pixel to be processed based on the representative signal value; a unit arranged to select a substitute candidate pixel common to the plurality of colors from the image area based on the calculated feature amount; and a unit arranged to substitute signal values of the pixel to be processed by new pixel values, which are calculated from the signal values of the pixel to be processed and signal values of the substitute candidate pixel.