1. An image processing device comprising: a distance information calculation unit that calculates distance information indicating a distance from an imaging lens to a reference position in a captured image corresponding to image data, wherein the distance information is based on a phase difference distribution that is generated by using a first phase difference between a first parallax image and a second parallax image along a first edge area of an object image in a first direction and a second phase difference between the second parallax image and a third parallax image along a second edge area of the object image in a second direction, wherein the first direction is different from the second direction; and an image processing unit that performs image processing based on the image data using the distance information calculated by the distance information calculation unit.