BACKGROUND

The present disclosure generally relates to head-mounted displays (HMDs), and specifically relates providing gaze contingent content on HMDs.

HMD's generate displays that depend on user motion. For example, a scene presented by a display of an HMD may move with detected changes in the user's eye position to create an immersive virtual reality (VR) environment. Eye tracking refers to the process of detecting the direction of a user's gaze, which may comprise detecting the position or angular orientation of the user's eyes. However, there is typically a (e.g., processing, data communication, etc.) latency between eye tracking data detection and the corresponding gaze contingent content being output on the display.

SUMMARY

Techniques for compensating for latency between eye tracking information detection and gaze contingent content display are discussed herein. A neural network model that predicts the future state of a user's eyes is used to generate input for gaze contingent content rendering. Programmatic eye movement prediction is a challenging problem because of the complexity of human eye movements over time. A neural network is used to create the neural network model of eye movements. In particular, a recurrent neural network is used to generate “predicted eye tracking information” including future eye positions and the future vergence depth of the eyes based on using input eye tracking information of previous eye positions or other prior states (e.g., previous vergence depth, head motion, etc.). The input eye tracking information includes a sequence of measured eye positions and vergence depths, and thus can be used to predict the future state of the eyes at a future time (or timestep). The predicted eye tracking information is a predicted future state of the eyes, which is generated by the recurrent neural network based on analyzing the current state, and prior states of the eyes. The predicted eye tracking information is used as input for rendering the gaze contingent content for output to a display.

For example, some embodiments may include a system comprising: a head mounted display (HMD) comprising: an electronic display that outputs gaze contingent content in accordance with display instructions, a plurality of cameras configured to capture images of a first eye of a user of the HMD and a second eye of a user of the HMD, and an optics block configured to direct the gaze contingent content to respective exit pupils of the HMD corresponding to locations of the first eye and the second eye of the user of the HMD. The system further includes an eye tracking controller configured to: determine eye tracking information defining positions of a first eye and a second eye of a user using the captured images, and determine, based on supplying the eye tracking information to a neural network model, predicted eye tracking information defining predicted positions of the first eye and the second eye. The system further includes a console configured to: generate the display instructions using the predicted eye tracking information, the display instructions including content for presentation by the display, and provide the display instructions to the HMD. In various embodiments, the eye tracking controller may be part of the HMD, the console, or a remote machine (e.g., of a central system).

In some embodiments, the neural network model may include a recurrent neural network model. The neural network model may define an algorithmic relationship between the eye tracking information at first timesteps and the predicted eye tracking information at a second timestep subsequent to the first timesteps.

The location of the eye tracking controller within the system may vary. In some embodiments, the eye tracking controller is integrated with the console. In other embodiments, the eye tracking controller is integrated with the HMD.

