1. An image processing apparatus that restores an image quality of an image obtained based on an optical image formed on an image pickup device via an image pickup optical system, from degradation, comprising: a first color feature amount calculation unit configured to calculate a feature amount of a color of a pixel in the image before a predetermined restoration processing, as a first color feature amount; an image restoration processing unit configured to perform the predetermined restoration processing on the image, using an image restoration filter; a second color feature amount calculation unit configured to calculate a feature amount of a color of a pixel in an image after the predetermined restoration processing, which is output from said image restoration processing unit, as a second color feature amount; an attenuation unit configured to attenuate high-frequency components included in the first color feature amount and the second color feature amount according to a photographing condition at a time of photographing of the image; and a correction unit configured to correct a pixel value in the image after the predetermined restoration processing, based on the first color feature amount and the second color feature amount of which the high-frequency components are attenuated by said attenuation unit.