Neural network training for three dimensional (3d) gaze prediction with calibration parameters

Abstract

Techniques for generating 3D gaze predictions based on a deep learning system are described. In an example, the deep learning system includes a neural network. The neural network is trained with training images. During the training, calibration parameters are initialized and input to the neural network, and are updated through the training. Accordingly, the network parameters of the neural network are updated based in part on the calibration parameters. Upon completion of the training, the neural network is calibrated for a user. This calibration includes initializing and inputting the calibration parameters along with calibration images showing an eye of the user to the neural network. The calibration includes updating the calibration parameters without changing the network parameters by minimizing the loss function of the neural network based on the calibration images. Upon completion of the calibration, the neural network is used to generate 3D gaze information for the user.

What is claimed is: 
     
         1 . A computer-implemented method comprising:
 training, by a computer system, a neural network by at least:
 inputting a training image and a first calibration parameter to the neural network, the training image showing an eye of a person, and 
 updating the first calibration parameter and a network parameter of the neural network based on minimizing a loss function of the neural network, wherein the loss function is minimized based on the training image and the first calibration parameter; 
   upon completion of the training, calibrating by the computer system the neural network for a user by at least:
 inputting a calibration image and a second calibration parameter to the neural network, the calibration image showing an eye of the user based on image data generated by a camera associated with an eye tracking system of the user, and 
 updating the second calibration parameter without updating the network parameter of the neural network, wherein the second calibration parameter is updated by at least minimizing the loss function based on the calibration image and the second calibration parameter; and 
   upon completion of the calibrating, generating by the computer system three dimensional (3D) gaze information for the user by at least:
 inputting an image and the second calibration parameter to the neural network, the image showing the eye of the user based on additional image data generated by the camera, and 
 receiving a prediction from the neural network based on the image and the second calibration parameter, wherein the prediction comprises a distance correction, a two dimensional (2D) gaze origin of the eye of the user in the image, and a 2D gaze direction of the eye of the user in the image.