1. An image processing apparatus comprising: image acquiring unit configured to acquire an input image; and image restoration processing unit configured to generate a restored image by calculating the input image and an image restoration filter, the image restoration filter being based on a transfer function of an image pickup system that is used to form an object image as the input image, wherein the image restoration filter is a filter that performs restoration so that when an object is a white point light source, a difference between spectra of two color components in the restored image is made smaller than a difference between spectra of the two color components in the input image.