1. An image processing device comprising: a distance information calculation unit that calculates distance information indicating a distance from an imaging lens to a reference position in a captured image corresponding to image data, wherein the distance information is based on a phase difference distribution that is generated by using a phase difference between parallax images detected in a direction different from a direction of an edge area including an edge along a parallax direction; and an image processing unit that performs image processing based on the image data using the distance information calculated by the distance information calculation unit.