1. An image processing apparatus comprising: an image acquisition unit configured to acquire a mosaic image that is photographed by an image pickup unit, the image pickup unit including an image pickup element having a predetermined basic array pattern repeatedly arranged in a horizontal direction and a vertical direction, the basic array pattern being composed of horizontally m×vertically n pixels (m, n: one is an integer of 2 or more, and another is an integer of 3 or more) and including color filters of at least three colors arrayed therein; a first storage unit configured to store first correction data containing a coordinate of a defective pixel in the image pickup element and correction information associated with a position of the defective pixel in the basic array pattern; a second storage unit configured to store second correction data containing a correction pattern for a correction process of the defective pixel, the correction pattern indicating positions of plural peripheral pixels relative to the defective pixel, each of the peripheral pixels having a same color as the defective pixel, the correction pattern corresponding to the correction information; and a defect correction processing unit configured to calculate a pixel value of each defective pixel by interpolating pixel values of the peripheral pixels having the same color, based on the first correction data and the second correction data respectively stored in the first storage unit and the second storage unit, the peripheral pixels being determined for the defective pixel, wherein the defect correction processing unit comprises: a coordinate calculation unit configured to read the correction pattern from the second storage unit based on the correction information contained in the first correction data, and to calculate coordinates of the plural peripheral pixels having the same color and being used in the correction process of the defective pixel, based on the coordinate of the defective pixel contained in the first correction data and the read correction pattern; a pixel value extraction unit configured to extract the pixel values corresponding to the calculated coordinates of the plural peripheral pixels, from the mosaic image; and a pixel value calculation unit configured to calculate the pixel value of the defective pixel by interpolating the extracted pixel values corresponding to the coordinates of the plural peripheral pixels.