Training of a neural network for three dimensional (3d) gaze prediction

Abstract

Techniques for generating 3D gaze predictions based on a deep learning system are described. In an example, the deep learning system includes a neural network. The neural network is trained with training images generated by cameras and showing eyes of user while gazing at stimulus points. Some of the stimulus points are in the planes of the camera. Remaining stimulus points are not un the planes of the cameras. The training includes inputting a first training image associated with a stimulus point in a camera plane and inputting a second training image associated with a stimulus point outside the camera plane. The training minimizes a loss function of the neural network based on a distance between at least one of the stimulus points and a gaze line.

What is claimed is: 
     
         1 . A non-transitory computer-readable storage medium storing instructions, that upon execution on a computer system, cause the computer system to perform operations comprising:
 accessing training images that comprise a first set of training images and a second set of training images, wherein the first set of training images show user eyes associated with gaze points in a plane of a camera, and wherein the second set of training images show user eyes associated with gaze points outside the plane of the camera; and   training a neural network based on the training images, wherein the training comprises:
 inputting a first training image to a neural network, wherein the first training image belongs to the first set of training images; 
 inputting a second training image to the neural network, wherein the second training image belongs to the second set of training images; 
 minimizing a loss function of the neural network based on a distance between a gaze point and a gaze line, wherein the gaze point is associated with one of the first training image or the second training image, and wherein the gaze line is predicted by the neural network for a user eye shown in the one of the first training image or the second training image.