Deep learning for three dimensional (3d) gaze prediction

Abstract

Techniques for generating 3D gaze predictions based on a deep learning system are described. In an example, the deep learning system includes a neural network. A scaled image is generated from 2D image showing a user face based on a rough distance between the user eyes and a camera that generated the 2D image. Image crops at different resolutions are generated from the scaled image and include a crop around each of the user eyes and a crop around the user face. These crops are input to the neural network. In response, the neural network outputs a distance correction and a 2D gaze vector per user eye. A corrected eye-to-camera distance is generated by correcting the rough distance based on the distance correction. A 3D gaze vector for each of the user eyes is generated based on the corresponding 2D gaze vector and the corrected distance.

What is claimed is: 
     
         1 . A computer-implemented method for detecting three dimensional (3D) gaze, the computer-implemented method comprising:
 generating, by an eye tracking system based on a two dimensional (2D) image, a warped image centered around a user eye from the 2D image, wherein the 2D image is generated by a camera associated with the eye tracking system;   inputting, by the eye tracking system to a neural network, the warped image, wherein the neural network predicts a distance correction, a 2D gaze origin of the user eye in the warped image, and a 2D gaze direction of the user eye in the warped image based on the warped image;   generating, by the eye tracking system, a corrected distance between the user eye and the camera by at least updating an estimated distance based on the distance correction; and   generating, by the eye tracking system, 3D gaze information for the user eye based on the 2D gaze origin, the 2D gaze direction, and the corrected distance.