1. An image processing apparatus comprising: an image obtaining unit that obtains a first image photographed through an optical system and a second image photographed under optical conditions different from optical conditions in which the first image is photographed; and a defect information generating unit that generates defect information indicating a defect in the first image or the second image by using the first image and the second image, wherein: the first image and the second image are photographed under different optical conditions with regard to at least either of an aperture value and a pupil position, the defect information generating unit includes an optical condition conversion unit that converts at least either the first image or the second image so as to conform to a specific optical condition, in order to eliminate a mismatch of the optical conditions for the first image and the second image, if the optical conditions with regard to the aperture value are different, the optical condition conversion unit executes low pass filter processing on a pixel signal generated based upon the first image or the second image so as to convert a defect state corresponding to the first image or the second image to a defect state estimated to manifest at a matching aperture value.