1. An intermediate image generation apparatus, comprising: a generation unit to generate an intermediate image from any one image of a left image and a right image by referring to a disparity map of the one image of the left image and the right image; and an interpolation unit to interpolate the intermediate image by referring to the other image of the left image and the right image.