1. An image processing device comprising: a comparison portion that compares a first image and a second image; and a determination portion that determines, based on a comparison result of the comparison portion, which of the first image and the second image is the image for a right and which is the image for a left, wherein the comparison portion performs template matching, on the second image, of a block extracted from the first image, compares a position of a matching block on the second image with a position of a block of a predetermined size on the first image, calculates a reliability by calculating a luminance dispersion of the matching block, and calculates, as a displacement evaluation value, an average value of differences in a horizontal direction of matching blocks with reliability higher than a threshold, wherein the comparison portion and the determination portion are each implemented via at least one processor.