1. An image processing apparatus comprising: a defect determining unit determining whether a target pixel is a defect or not, on the basis of a pixel value of the target pixel and pixel values of a plurality of adjacent pixels which are pixels for the same color as the target pixel and are positioned at the periphery of the target pixel, wherein the defect determining unit includes: a first determining unit performing defect determination by comparing the maximum value and minimum value of the pixel values of the plurality of adjacent pixels with the pixel value of the target pixel; a second determining unit performing defect determination according to an illumination light component, which is a component of illumination light irradiated onto an object, of the pixel values of the plurality of adjacent pixels; and a third determining unit performing defect determination according to a reflectivity component, which is a component based on a unique reflectivity of the object, of the pixel values of the plurality of adjacent pixels.