1. An image processing device that generates a three-dimensional image to be displayed on a display unit including a display element unit that has pixels each containing a plurality of sub-pixels arranged thereon and a ray control element unit that controls emission directions of rays emitted from the sub-pixels, the image processing device comprising: a first acquiring unit configured to acquire three-dimensional data for generating the three-dimensional image; and a generating unit configured to calculate, for each of the sub-pixels, a luminance value of a sub-pixel on the basis of an emission direction of a ray emitted from the sub-pixel through the ray control element unit and the three-dimensional data to generate the three-dimensional image.