1. An arithmetically operating device comprising: an image acquiring portion configured to acquire phase difference images of a sample; a retrieval range setting portion configured to set a range of pixels each to be made a retrieval object in the other image in the phase difference images for pixels each to be made a representative set in one image in the phase difference images, and pixels other than the pixels each to be made the representative; a correlation calculating portion configured to calculate correlations between the pixels for the one image, and the pixels in the range set for the pixels; and a parallax calculating portion configured to calculate parallaxes for the pixels for the one image in accordance with the correlations thus calculated; wherein said retrieval range setting portion sets a range narrower than that set for the pixels each to be made the representative for the pixels other than the pixels each to be made the representative in accordance with the parallaxes calculated for the pixels each to be made the representative.