CROSS REFERENCE TO RELATED APPLICATIONS AND CLAIM OF PRIORITY

This application claims benefit under 35 U.S.C. 119(e), 120, 121, or 365(c), and is a National Stage entry from International Application No. PCT/KR2018/004562 filed on Apr. 19, 2018, which claims priority to the benefit of Korean Patent Application Nos. 10-2017-0117059 filed on Sep. 13, 2017 and 10-2017-0167334 filed on Dec. 7, 2017 in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to an eye tracking technology.

BACKGROUND ART

Eye tracking is a technology for tracking the location of a gaze by sensing the ocular movement of a user, and video analysis methods, contact lens methods, sensor attachment methods, or the like may be used as eye tracking methods. According to a video analysis method, the movement of a pupil is detected through real-time camera image analysis, and the direction of a gaze is calculated on the basis of a fixed position which reflects onto the cornea. According to a contact lens method, the reflected light of a mirror contact lens, the magnetic field of a coil contact lens, or the like is used. The contact lens method has low convenience but high accuracy. According to a sensor attachment method, a sensor is attached to the vicinity of an eye, and ocular movement is sensed using a change in electric field caused by the movement of the eye. It is possible to detect ocular movement even when the eye is closed (during sleep or the like).

Recently, the eye tracking technology is gradually being applied to a greater variety of equipment and wider application fields. Accordingly, in providing an advertising service in a terminal, such as a smart phone, attempts to use the eye tracking technology are increasing. However, in order to efficiently provide an advertising service, it is necessary to improve the accuracy of eye tracking and efficiently configure a bidding method, a reward method, etc. according to advertisement watching.

SUMMARY

The present invention is directed to providing a means for improving the accuracy of eye tracking when eye tracking is performed on the basis of deep learning.

One aspect of the present invention provides a user terminal including an imaging device configured to capture a face image of a user and an eye tracking unit configured to acquire a vector representing a direction that a face of the user faces and an ocular image of the user from the face image on the basis of set rules and track a gaze of the user by inputting the face image, the vector, and the ocular image to a set deep learning model.

The user terminal may further include a training data collection unit configured to collect training data including a face image of a viewer captured at a time point of receiving a set action and location information of a set point when the set action is received from the viewer who looks at the set point in a screen, and the eye tracking unit may train the deep learning model with the training data and track the gaze of the user using the deep learning model which has learned the training data.

When the viewer touches the point, the training data collection unit may collect the training data at a time point at which the touch is made.

The training data collection unit may collect the training data by operating the imaging device at the time point at which the viewer touches the point.

The training data collection unit may transmit the training data collected at the time point at which the viewer touches the point to a server.

When the viewer touches the point while the imaging device is operating, the training data collection unit may separately collect the training data at the time point at which the touch is made and time points a set time before and after the time point at which the touch is made.

The training data collection unit may change a visual element of the point after the viewer touches the point so that a gaze of the viewer may remain at the point even after the touch.

The training data collection unit may display set text at the point and collect the training data at a time point at which an utterance of the viewer is started when the viewer speaks.

The eye tracking unit may acquire ocular location coordinates and face location coordinates of the user from the face image on the basis of the rules and additionally input the ocular location coordinates and the face location coordinates to the deep learning model together with the vector representing the direction that the face of the user faces.

The user terminal may further include a content providing unit configured to display advertising content on the screen, the eye tracking unit may determine whether the user is watching the advertising content on the basis of a detected gaze of the user and a location of the advertising content in the screen, and the content providing unit may change the location of the advertising content in the screen by considering the location of the advertising content in the screen and a time period for which the user has watched the advertising content.

Another aspect of the present invention provides an eye tracking method including: capturing, by an imaging device, a face image of a user; acquiring, by an eye tracking unit, a vector representing a direction that a face of the user faces and an ocular image of the user from the face image on the basis of set rules; and inputting, by the eye tracking unit, the face image, the vector, and the ocular image to a set deep learning model to track a gaze of the user.

The eye tracking method may further include: when a set action is received from a viewer who looks at a set point in a screen, collecting, by a training data collection unit, training data including a face image of the viewer captured at a time point of receiving the set action and location information of the set point; and training, by the eye tracking unit, the deep learning model with the training data. The tracking of the gaze of the user may include tracking the gaze of the user by using the deep learning model which has learned the training data.

The collecting of the training data may include, when the viewer touches the point, collecting the training data at a time point at which the touch is made.

The collecting of the training data may include collecting the training data by operating the imaging device at the time point at which the viewer touches the point.

The eye tracking method may further include transmitting, by the training data collection unit, the training data collected at the time point at which the viewer touches the point to a server.

The collecting of the training data may include, when the viewer touches the point while the imaging device is operating, separately collecting the training data at the time point at which the touch is made and time points a set time before and after the time point at which the touch is made.

The eye tracking method may further include changing, by the training data collection unit, a visual element of the point after the viewer touches the point so that a gaze of the viewer may remain at the point even after the touch.

The collecting of the training data may include displaying set text at the point and collecting, when the viewer speaks, the training data at a time point at which an utterance of the viewer is started.

The eye tracking method may further include acquiring, by the eye tracking unit, ocular location coordinates and face location coordinates of the user from the face image on the basis of the rules, and the tracking of the gaze of the user may include additionally inputting the ocular location coordinates and the face location coordinates to the deep learning model together with the vector representing the direction that the face of the user faces.

The eye tracking method may further include: displaying, by a content providing unit, advertising content on the screen; determining, by the eye tracking unit, whether the user is watching the advertising content on the basis of a detected gaze of the user and a location of the advertising content in the screen; and changing, by the content providing unit, the location of the advertising content in the screen by considering the location of the advertising content in the screen and a time period for which the user has watched the advertising content.

According to embodiments of the present invention, it is possible to improve the accuracy of eye tracking by using a vector representing a direction that the face of a user faces as well as a face image and an ocular image of the user as input data of a deep learning model in the case of eye tracking based on the deep learning model.

Also, according to the embodiments of the present invention, when an action, such as a touch or an utterance, is input from a viewer who is looking at a set point in a screen, a face image of the viewer captured at a time point at which the action is input and location information of the point is used as training data of a deep learning model for eye tracking. Consequently, it is possible to improve the accuracy and reliability of eye tracking.

