1. An image processing apparatus comprising at least one processor that functions as: a first calculation unit configured to calculate a feature amount of a color of pixels in first image data obtained according to an optical image formed on an image pickup device via an image pickup optical member, as a first color feature amount; a restoration processing unit configured to perform restoration processing on the first image data, using a restoration filter for improving image quality degraded due to the image pickup optical member, to thereby generate second image data; a second calculation unit configured to calculate a feature amount of a color of pixels in the second image data as a second color feature amount; and a correction unit configured to correct pixel values of the second image data according to a result of comparison between an amount of change in the second color feature amount with respect to the first color feature amount, and a threshold value, wherein said correction unit is configured to change the threshold value based on an edge intensity of the first image data such that the threshold value decreases as the edge intensity increases.