Methods, devices and systems enabling determination of eye state variables

Abstract

Methods, devices and systems for generating data suitable for determining at least one eye state variable of at least one eye of a subject, and methods and systems for determining such eye state variables are provided. The at least one eye state variable being derivable from at least one image of the eve taken with a camera of known camera intrinsics. Synthetic image data of a first 3D model eye which models corneal refraction can be generated for different sets of eye state variables and may be used to determine said eye state variables using a further 3D eye model comprising at least one parameter. A characteristic of the pupil image in the synthetic images can be determined for later use to determine eye state variables based on image data of real eyes and under use of the further 3D eye model.

1 - 30 . (canceled) 
     
     
         31 . A method for generating data suitable for determining at least one eye state variable of at least one eye of a subject, the eye comprising an eyeball, an iris defining a pupil, and a cornea, the at least one eye state variable being derivable from at least one image of the eye taken with a camera of known camera intrinsics, the method comprising:
 providing a first 3D eye model modeling corneal refraction;   generating, using the known camera intrinsics, synthetic images of several model eyes according to the first 3D eye model, for a plurality of given values of at least one eye state variable;   using a given algorithm to calculate the at least one eye state variable using one or more of the synthetic images and a further 3D eye model having at least one parameter;   determining a characteristic of the image of the pupil within each of the synthetic images;   determining one or more hypothetically optimal values of the at least one parameter of the further 3D eye model that minimize the error between the value(s) of the at least one given eye state variable and the value(s) of the corresponding eye state variable obtained when applying the given algorithm; and   establishing a relationship between the one or more hypothetically optimal values of the at least one parameter of the further 3D eye model and the characteristic of the pupil image.   
     
     
         32 . The method of  claim 31 , wherein the characteristic of the image of the pupil is a measure of the circularity of the pupil area or outline, in particular a ratio of minor to major axis length of an ellipse fit to the pupil image area or outline, a measure of variation of the curvature of the pupil outline, a measure of elongation or a measure of the bounding box of the pupil area. 
     
     
         33 . The method of  claim 31 , wherein the relationship between the hypothetically optimal values of the at least one further 3D eye model parameter and the characteristic of the pupil image is chosen from the list of a constant value, in particular a constant value smaller or larger than the corresponding average parameter of the first 3D eye model, a linear relationship, a polynomial relationship, or another non-linear relationship, in particular a relationship derived via a regression fit. 
     
     
         34 . The method of  claim 31 , wherein the further 3D eye model has at most one parameter. 
     
     
         35 . The method of  claim 31 , wherein the further 3D eye model has multiple parameters and a relationship is established for more than one of them. 
     
     
         36 . The method of  claim 31 , wherein any parameter of the first and/or of the further 3D eye model is/are selected from the list of a distance between a center of an eyeball, in particular a rotational, geometrical or optical center, and a center of a pupil or cornea, a size measure of an eyeball, a cornea or an iris such as an eyeball radius, a cornea radius, an iris diameter, a distance pupil center to cornea center, a distance cornea center to eyeball center, a distance pupil center to limbus center, a distance crystalline lens to eyeball center, to cornea center and/or to corneal apex, a refractive property of an eye structure such as an index of refraction of a cornea, vitreous humor or crystalline lens, an ellipsoidal shape measure of an eyeball or cornea, and a degree of astigmatism. 
     
     
         37 . The method of  claim 31 , wherein said relationship is the same for all eye state variables, or wherein a different relationship between a parameter of the further 3D eye model and the characteristic of the pupil image is established for each eye state variable or for groups of eye state variables. 
     
     
         38 . The method of  claim 31 , wherein the eye state variable is selected from the list of a pose of an eye such as a location of an eye, in particular an eyeball center, and/or an orientation of an eye, in particular a gaze vector, optical axis orientation or visual axis orientation, a 3D circle center line, a 3D eye intersecting line, and a size measure of a pupil of an eye, such as a pupil radius or diameter. 
     
     
         39 . A method for determining at least one eye state variable of at least one eye of a subject, the eye comprising an eyeball, an iris defining a pupil, and a cornea, the at least one eye state variable being derivable from at least one image of the eye taken with a camera of known camera intrinsics, the method comprising:
 receiving image data of the at least one eye from a camera of known camera intrinsics and defining an image plane;   determining a characteristic of the image of the pupil within the image data;   providing a 3D eye model having at least one parameter, the at least one parameter depending in a pre-determined relationship on the characteristic;   using a given algorithm to calculate the at least one eye state variable using the image data and the 3D eye model including the at least one parameter.   
     
     
         40 . The method of  claim 39 , wherein the characteristic of the image of the pupil is a measure of the circularity of the pupil area or outline, in particular a ratio of minor to major axis length of an ellipse fit to the pupil image area or outline, a measure of variation of the curvature of the pupil outline, a measure of elongation or a measure of the bounding box of the pupil area. 
     
     
         41 . The method of  claim 39 , wherein the pre-determined relationship between the at least one parameter of the 3D eye model and the characteristic of the pupil image is chosen from the list of a constant value, a linear relationship, a polynomial relationship, or another non-linear relationship, in particular a relationship derived via a regression fit, in particular wherein the relationship is stored in analytical form and evaluated on-the-fly for given image data or stored as a lookup-table. 
     
     
         42 . The method of  claim 39 , wherein the 3D eye model has either only one parameter, or wherein the 3D eye model has multiple parameters and a pre-determined relationship between any of them and the characteristic is used for at least one of the parameters. 
     
     
         43 . The method of  claim 39 , wherein the respective parameter of the 3D eye model is selected from the list of a distance between a center of an eyeball, in particular a rotational, geometrical or optical center, and a center of a pupil or cornea, a size measure of an eyeball, a cornea or an iris such as an eyeball radius, a cornea radius, an iris diameter, a distance pupil-center to cornea-center, a distance cornea-center to eyeball-center, a distance pupil-center to limbus center, a distance crystalline lens to eyeball-center, to cornea center and/or to corneal apex, a refractive property of an eye structure such as an index of refraction of a cornea, vitreous humor or crystalline lens, an ellipsoidal shape measure of an eyeball or cornea, and a degree of astigmatism. 
     
     
         44 . The method of  claim 39 , wherein said relationship is the same for all eye state variables, or wherein a different pre-determined relationship between a parameter of the 3D eye model and the characteristic of the pupil image is used for each eye state variable or for groups of eye state variables. 
     
     
         45 . The method of any of  claim 39 , wherein the eye state variable is selected from the list of a pose of an eye such as a location of an eye, in particular an eyeball center, and/or an orientation of an eye, in particular a gaze vector, optical axis orientation or visual axis orientation, a 3D circle center line, a 3D eye intersecting line, and a size measure of a pupil of an eye, such as a pupil radius or diameter. 
     
     
         46 . The method of  claim 31 , wherein the given algorithm does not take into account a glint from the eye for calculating the at least one eye state variable, wherein the algorithm is glint-free, and/or wherein the algorithm does not require structured light and/or special purpose illumination to derive eye state variables, and/or wherein the given algorithm calculates the at least one eye state variable in a non-iterative way. 
     
     
         47 . The method of  claim 31 , the given algorithm including:
 determining a first ellipse in the image data, the first ellipse at least substantially representing a border of the pupil of the at least one eye at a first time;   using the camera intrinsics and the first ellipse to determine a 3D orientation vector of a first circle in 3D and a first center line on which a center of the first circle is located in 3D, so that a projection of the first circle, in a direction parallel to the first center line, onto the image plane is expected to reproduce the first ellipse; and   determining a first eye intersecting line in 3D expected to intersect a 3D center of the eyeball at the corresponding time as a line which is, in the direction of the orientation vector, parallel-shifted to the first center line by an expected distance between the center of the eyeball and a center of the pupil.   
     
     
         48 . The method of  claim 47 , further comprising at least one of:
 receiving image data of a further eye of the subject at a time, substantially corresponding to the first times, from a camera of known camera intrinsics and defining an image plane, the further eye comprising a further eyeball, a further iris defining a further pupil, and a further cornea, the given algorithm further including:
 determining a further ellipse in the image data, the further ellipse at least substantially representing the border of the further pupil of the further eye at the corresponding time; 
 using the camera intrinsics and the further ellipse to determine a 3D orientation vector of a further circle in 3D and a further center line on which a center of the further circle is located in 3D, so that a projection of the further circle, in a direction parallel to the further center line, onto the image plane is expected to reproduce the further ellipse; 
 determine a further eye intersecting line in 3D expected to intersect a 3D center of the further eyeball at the corresponding time as a line which is, in the direction of the 3D orientation vector of the further circle, parallel-shifted to the further center line by an expected distance between the center of the further eyeball and a center of the further pupil; 
 receiving second image data of the at least one eye at a second time from the camera; 
   the given algorithm further including:
 determining a second ellipse in the second image data, the second ellipse at least substantially representing the border of the pupil at the second time; 
 using the camera intrinsics and the second ellipse to determine an orientation vector of a second circle and a second center line on which a center of the second circle is located, so that a projection of the second circle, in a direction parallel to the second center line, onto the image plane is expected to reproduce the second ellipse; and 
 determine a second eye intersecting line expected to intersect the center of the eyeball at the second time as a line which is, in the direction of the orientation vector of the second circle, parallel-shifted to the second center line by the expected distance. 
   
     
     
         49 . The method of  claim 48 , wherein the given algorithm further includes using the first eye intersecting line and the second eye intersecting line, respectively the first eye intersecting line and the further eye intersecting line to determine other eye state variables such as co-ordinates of the center of the eyeball of the at least one eye respectively of the at least one eye and the further eye, a gaze direction, an optical axis, an orientation, a visual axis, a size of the pupil and/or a radius of the pupil of the at least one eye and/or of the further eye, wherein the expected distance between the center of the eyeball and the center of the pupil is a parameter of the 3D eye model respectively of the further 3D eye model, depending in the pre-determined relationship on the characteristic of the image of the pupil of the corresponding eye, wherein the respective center line and/or the respective eye intersecting line is determined using a model of the camera and/or the 3D eye model respectively the further 3D eye model, wherein the camera is modeled as a pinhole camera, and/or wherein the model of the camera comprises at least one of a focal length, a shift of a central image pixel, a shear parameter, and a distortion parameter. 
     
     
         50 . A computer program product or a non-volatile computer-readable storage medium comprising instructions which, when executed by a one or more processors of a system, cause the system to carry out the following steps:
 providing a first 3D eye model modeling corneal refraction;   generating, using known camera intrinsics of a camera, synthetic images of several model eyes according to the first 3D eye model, for a plurality of given values of at least one eye state variable, the at least one eye state variable being derivable from at least one image of an eye of a subject taken with the camera;   using a given algorithm to calculate the at least one eye state variable using one or more of the synthetic images and a further 3D eye model having at least one parameter;   determining a characteristic of an image of a pupil within each of the synthetic images;   determining one or more hypothetically optimal values of the at least one parameter of the further 3D eye model that minimize the error between the value(s) of the at least one given eye state variable and the value(s) of the corresponding eye state variable obtained when applying the given algorithm; and   establishing a relationship between the one or more hypothetically optimal values of the at least one parameter of the further 3D eye model and the characteristic of the pupil image.