1. An image processing apparatus comprising at least one processor that functions as: a first calculation unit configured to calculate a feature amount of a color of pixels in first image data obtained according to an optical image formed on an image pickup device via an image pickup optical member, as a first color feature amount; a restoration processing unit configured to perform restoration processing on the first image data, using a restoration filter for improving image quality degraded due to the image pickup optical member, to thereby generate second image data; a second calculation unit configured to calculate a feature amount of a color of pixels in the second image data as a second color feature amount; and a correction unit configured to correct pixel values of the second image data according to a result of comparison between an amount of change in the second color feature amount with respect to the first color feature amount, and a threshold value, wherein said correction unit is configured to set the threshold value based on a predetermined condition.