1. An image processing device for generating depth data, utilizing a first image and a second image which are captured from different viewpoints, the image processing device comprising: a processor; and a non-transitory computer-readable medium having instructions stored thereon that, when executed by the processor, cause the image processing device to function as: a disparity value calculation unit configured to calculate, for each of plural representative pixels included in pixels in the first image, a disparity value of the representative pixel, based on a positional relationship between the representative pixel and a pixel corresponding to the representative pixel, in the second image; a segmentation unit configured to partition the first image into plural segments, based on a similarity between pixel values; a segment combine unit configured to select an empty segment which includes no representative pixel from among the plural segments, and combine the empty segment and a segment adjacent to the empty segment together; and a depth data generation unit configured to determine, for each segment, a disparity value of the segment, based on the disparity value of the representative pixel included in the segment to generate depth data indicative of depths corresponding to the plural segments, wherein the depth data generation unit is configured to generate the depth data, based on a segment obtained by the segment combine unit combining the empty segment and the segment adjacent to the empty segment together.