1. A gaze detection system comprising: a head mounted display, mounted on a user's head for use, comprising, a first infrared light source unit illuminating a user's right eye with infrared light; a second infrared light source unit illuminating a user's left eye with infrared light; a first image capturing device imaging the right eye illuminated by the infrared light; a second image capturing device imaging the left eye illuminated by the infrared light; and a display unit displaying a three-dimensional image, and a gaze detection device detecting a gaze of the user comprising, a first detection unit detecting a gaze direction of the right eye based on the image captured by the first image capturing device; a second detection unit detecting a gaze direction of the left eye based on the image captured by the second image capturing device; and a tracking unit that determines a gaze point of the user in the three-dimensional image on the basis of the gaze direction of the right eye and the gaze direction of the left eye, in which the first detection unit calculates the right eye gaze vector indicating the gaze direction of the right eye, and the second detection unit calculates the left eye gaze vector indicating the gaze direction of the left eye, wherein the display unit displays a three-dimensional image consisting of a plurality of layers in a depth direction, and the tracking unit identifies the layer that the user is gazing at by selecting the layer having a shortest distance between intersection points of the right eye gaze vector and the left eye gaze vector with each layer.