1. An image processing apparatus, comprising: an image pickup unit configured to pick up a plurality of images of an observation sample at different focal depths; a difference calculation processor configured to calculate differences of luminance values of pixels in the images at different focal depths; and a difference distribution generation processor configured to generate a distribution of the differences of the luminance values with respect to the different focal depths, wherein the difference distribution generation processor is configured to calculate luminance total values by totaling the differences of the luminance values for each pixel column in a first direction of the images, to arrange the luminance total values of each pixel column in a second direction perpendicular to the first direction at the different focal depths to form a plurality of analysis pixel columns, and to generate an analysis image based on the plurality of the analysis pixel columns in a diagram having a focal depth axis indicating a focal depth of the images and a positional information axis indicating a position in the second direction, and wherein the analysis image has a color representing the luminance total values, and an analysis processor configured to calculate a size of the observation sample based on the analysis image.