1. A three-dimensional image display apparatus, comprising: a display unit comprising a two-dimensional image display device comprising a display surface with a plurality of pixels arranged in a matrix, and a plurality of light ray control elements provided on the display surface of the two-dimensional image display device corresponding to the pixels, the light ray control elements being configured to control light rays from the pixels, an intensity of each of the light rays measured at a distance from the lens distributing in a range; and an optimal elemental image generating unit configured to generate an optimal elemental image by determining a value of at least one pixel from a weighted sum of values of at least two pixels of a generated image or an input image, and weights calculated based on light ray distributions indicating directions and scattering degrees of light rays from the light ray control elements, the display unit displaying the optimal elemental image, wherein the optimal elemental image generating unit receives a target image displayed on the display unit from at least one predetermined viewpoint, and wherein the optimal elemental image generating unit obtains weights for reducing an energy function indicating a difference between the target image and an overlap image viewed from the predetermined viewpoint and calculated from the light ray distributions.