1. An image processing apparatus, comprising: an image acquiring device which acquires a plurality of images consecutively shot while varying a focus position of a lens; a sharpness calculating device which calculates sharpness on each pixel of the plurality of images acquired by the image acquiring device; a first reference value calculating device which calculates a first reference value indicating an image of the plurality of images to which a pixel whose sharpness calculated by the sharpness calculating device is the highest among the pixels located on an identical position in the plurality of images belongs, on each pixel of the images; a second reference value calculating device which calculates a second reference value on each pixel of the images based on the first reference value calculated by the first reference value calculating device by spatially smoothing the first reference value on each pixel based on the first reference values on adjacent pixels; and an image processing device which image-processes at least one of the plurality of images based on the second reference values.