1. An image processing device that generates a three-dimensional image to be displayed on a display unit including a display element unit that has pixels each containing a plurality of sub-pixels arranged thereon and a ray control element unit that controls emission directions of rays emitted from the sub-pixels, the image processing device comprising: a first acquiring unit configured to acquire three-dimensional data for generating the three-dimensional image, the three-dimensional data being represented by a space partitioning model or a boundary representation model; and a generating unit configured to calculate luminance values for each of the sub-pixels on the basis of the three-dimensional data and emission directions of a ray emitted from the sub-pixels through the ray control element unit, wherein the generating unit includes a first calculating unit configured to calculate ray information for identifying a rendering ray, for each of the sub-pixels on the basis of the emission direction, the rendering ray being a ray passing through a viewpoint associated with the sub-pixel and being used to calculate the luminance value of the sub-pixel, and a second calculating unit configured to calculate the luminance value of the sub-pixel by using the ray information and the three-dimensional data for each of the sub-pixels, and the generating unit identifies, for each of the sub-pixels, a viewpoint corresponding to a direction obtained by correcting a direction represented by the emission direction according to a coordinate system in which the three-dimensional data is defined, as a viewpoint of the sub-pixel.