Technologies of recognizing and predicting feelings and behaviors of humans using machines are widely needed for society. For example, in order to realize robots more friendly with humans, it is necessary to recognize the feeling of other person. When whether or not a person behaves suspiciously can be determined from behavior prediction based on camera images from a monitoring camera, the number of staffs performing monitoring tasks can be reduced to save effort. An example of a key important to recognize the feelings of humans and to predict the behaviors thereof includes the line of sight of a person. This is because, though humans think or determine behaviors based on information obtained from external fields, most of the information is visual information.
For example, it is assumed that a camera is pointed at a driver of a vehicle and the line of sight of the driver is estimated from a camera image obtained using the pointed camera. In such a situation, it is assumed that it is found from the estimated line of sight of the driver that the driver intends to turn the vehicle to the left without seeing a motorcycle approaching from the left side of the vehicle. In this case, a behavior such as involved driving can be predicted. In addition, it is assumed that a camera is pointed at a viewer of a commercial display and the line of sight of the person standing in front of the display is estimated from a camera image obtained using the pointed camera. In such a situation, it is assumed that it is found from the estimated line of sight of the person that the point of gaze thereof is focused. In this case, it is possible to obtain information that the viewer shows an interest in information displayed at a position of the point of gaze.
The line of sight is a straight line in a three-dimensional space. Therefore, in order to define the line of sight, it is necessary to set two points in the three-dimensional space. Various methods involving defining the line of sight are expected. A first method is based on the idea that an eyeball is assumed as a structure in which a sphere forming a crystalline lens and a partial sphere forming a cornea portion are combined and a center of the cornea partial sphere and a center of a pupil are assumed as two points for defining the line of sight. The first method is normally called a “corneal reflection method” and disclosed in, for example, JP 07-055941 A (hereinafter referred to as “Patent Document 1”). According to the corneal reflection method, a rotation angle of the eyeball in a coordinate system fixed to a head is calculated based on a reflection light image (cornea reflection image or Purkinje image) observed in an eye region when the eyeball is irradiated with near-infrared illumination light, the center of the pupil, and an illumination relative position. A representation of a head posture in a world coordinate system which is estimated using any method and an eyeball rotation are combined to estimate the line of sight in the world coordinate system. Note that the “world coordinate system” means a coordinate system arbitrarily determined without depending on both a person and a camera.
Hereinafter, a three-dimensional coordinate system determined in association with a camera photographing a person is referred to as a “camera coordinate system”, and a coordinate system determined in association with the head of the person is referred to as a “head coordinate system”. The eyeball can reflect light from various light sources, and hence it is normally difficult to obtain a correspondence between an illumination light source and the cornea reflection image. Therefore, the corneal reflection method is effective in a case where the camera photographing the eye region and the illumination light source are significantly close to the person (for example, camera and illumination light source are incorporated in head mount display). However, the corneal reflection method has a problem that the person is required to wear the camera or the like and thus has inconvenience.
On the other hand, the second method is based on the idea that the eyeball is assumed as a single sphere and the center of the eyeball and the center of the pupil are assumed as two points for defining the line of sight. According to this idea (second method), the line of sight can be estimated without depending on positions of the camera and the illumination light source with respect to the person. Hereinafter, a combination of an eyeball central position and an eyeball radius is referred to as eyeball parameters.
The pupil is a feature point appearing on a surface of the eyeball, and hence a pupil central position can be detected from the camera image. However, the center of the eyeball is positioned in an inner portion of a human, and hence the eyeball central position cannot be directly detected from the camera image.
Various related technologies of estimating the eyeball central position have been known.
According to a face image processing system described in, for example, JP 2004-504684 A (WO 02/009025 A1) (hereinafter referred to as “Patent Document 2”), during a calibration operation for estimating the eyeball central position, a person to be examined gazes on the point of gaze with known coordinates in a plurality of postures. The eyeball central position is estimated based on the fact that the line of sight passes through the set point of gaze in each of the postures.
In a system described in JP 2003-015816 A (hereinafter referred to as “Patent Document 3”), it is assumed that the center of the eyeball is positioned on a straight line which passes through a middle point between the inner corner of the eye and the outer corner of the eye and has a directional vector determined from a head posture.