1. A method performed by data processing apparatus, the method comprising: receiving an image of a sequence of images, the image depicting a face of a user; for each particular facial feature point of a plurality of facial feature points of the face depicted in the image: comparing a template image for the particular facial feature point to one or more image portions of the image, the template image for the particular facial feature point including a portion of a previous image of the sequence of images that depicted the facial feature point; identifying, based on the comparison, a matching image portion of the image that matches the template image for the particular facial feature point; and identifying, in the image, a location of the matching image portion; and determining a direction in which the user is looking based on the identified location for each template image.