1. An image processing method comprising: obtaining a color input image produced by taking an image using an optical system; calculating a color difference of a pixel in the input image; performing an image restoration process for the input image using an image restoration filter selected or made based on aberration information of the optical system; calculating a color difference of a pixel in a restored image obtained by the image restoration process; and correcting a signal value of a specific pixel in the restored image in a case that the calculated color difference of the specific pixel in the restored image is larger than the calculated color difference of the specific pixel in the input image, such that a difference between the calculated color difference of the specific pixel in the restored image and the calculated color difference of the specific pixel in the input image is reduced.