1. A head mounted display (HMD) system comprising: an image capture device for tracking an eye pose of an eye of a wearer of the HMD system in an eye image of the eye of the wearer, wherein the eye pose comprises a direction toward which the eye is looking; non-transitory memory configured to store the eye image; a display for providing virtual image information to the wearer of the HMD system based on the eye pose of the wearer in the eye image; and a hardware processor programmed to: receive the eye image from the image capture device; map a pupil in the eye image to an equivalent frontal view to provide a remapped eye image; identify an eye feature based at least partly on the remapped eye image; determine a pitch and a yaw of the eye based at least partly on the remapped eye image; determine a roll of the eye based at least partly on the eye feature in the remapped eye image; determine the eye pose of the eye based at least partly on the pitch, the yaw, and the roll; determine the virtual image information to be provided to the wearer of the HMD using the eye pose of the eye; and cause the display to provide the virtual image information to the wearer of the HMD system.