Calibrated sensitivity model approximating the eye

Abstract

In one embodiment, a method includes projecting a source image onto a surface using a lens approximation component, where the surface is associated with sampling points approximating photoreceptors of an eye, where each sampling point has a corresponding photoreceptor type, sampling color information from the projected source image at the sampling points, where the color information sampled at each sampling point depends on the corresponding photoreceptor type, accessing pooling units approximating retinal ganglion cells (RGCs) of the eye, where each pooling unit is associated with groups of one or more of the sampling points, calculating weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with each pooling unit, and computing a perception profile for the source image based on the weighted aggregations associated with each of the pooling units.

1 - 20 . (canceled) 
     
     
         21 . A method comprising:
 generating a first image using a machine-learning model;   computing a first perception profile corresponding to the first image using a virtual eye model, wherein the virtual eye model comprises a lens-approximating component, a retina-approximating surface, and retinal ganglion cell (RGC)-approximating pooling units;   accessing a ground truth image corresponding to the first image as a second image;   computing a second perception profile corresponding to the second image using the virtual eye model;   measuring differences between the first perception profile and the second perception profile; and   updating trainable variables of the machine-learning model based on the measured differences.   
     
     
         22 . The method of  claim 21 , wherein computing the first perception profile corresponding to the first image using the virtual eye model comprising:
 projecting the first image onto the retina-approximating surface using the lens-approximating component, wherein the retina-approximating surface is associated with sampling points approximating photoreceptors of an eye, wherein each sampling point has a corresponding photoreceptor type;   sampling color information from the projected source image at the sampling points, wherein the color information sampled at each sampling point depends on the corresponding photoreceptor type;   accessing the RGC-approximating pooling units, wherein each pooling unit is associated with groups of one or more of the sampling points;   calculating, for each of the RGC-approximating pooling units, weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with the RGC-approximating pooling unit; and   computing a perception profile for the first image based on the weighted aggregations associated with each of the RGC-approximating pooling units.   
     
     
         23 . The method of  claim 22 , wherein the lens-approximating component projects the first image onto a curved shape surface and maps the first image projected onto the curved shape surface into an image on the retina-approximating surface. 
     
     
         24 . The method of  claim 23 , wherein the lens-approximating component comprises an Optical Transfer Function (OTF) and a warping operator, wherein the OTF convolves a matrix representing pixel information of the first image with convolution templates and multiplies with a coding matrix, wherein the convolution templates and the coding matrix are obtained by a matrix factorization, and wherein the warping operator maps the first image projected onto the curved shape surface into the image on the retina-approximating surface. 
     
     
         25 . The method of  claim 22 , wherein a distribution of a type of the sampling points is associated with a corresponding blue noise mask. 
     
     
         26 . The method of  claim 22 , wherein each RGC-approximating pooling unit has a corresponding RGC type. 
     
     
         27 . The method of  claim 26 , wherein a distribution of a type of the RGC-approximating pooling units is associated with a corresponding blue noise mask. 
     
     
         28 . The method of  claim 22 , wherein a group of one or more of the sampling points associated with an RGC-approximating pooling unit is determined based on a field weighting function. 
     
     
         29 . The method of  claim 28 , wherein the field weighting function is a Gaussian receptive field function. 
     
     
         30 . The method of  claim 22 , wherein the perception profile for the first image is computed using Rectified Linear Unit (ReLU) activation functions. 
     
     
         31 . The method of  claim 21 , wherein the first image corresponds to a frame of a video stream. 
     
     
         32 . The method of  claim 21 , wherein the first image can be of any resolution, and wherein a resolution of the first image is pre-determined. 
     
     
         33 . The method of  claim 21 , wherein the machine-learning model is a foveated-rendering machine-learning model. 
     
     
         34 . The method of  claim 21 , wherein the machine-learning model is a metamers-generating machine-learning model. 
     
     
         35 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 generate a first image using a machine-learning model;   compute a first perception profile corresponding to the first image using a virtual eye model, wherein the virtual eye model comprises a lens-approximating component, a retina-approximating surface, and retinal ganglion cell (RGC)-approximating pooling units;   access a ground truth image corresponding to the first image as a second image;   compute a second perception profile corresponding to the second image using the virtual eye model;   measure differences between the first perception profile and the second perception profile; and   update trainable variables of the machine-learning model based on the measured differences.   
     
     
         36 . The media of  claim 35 , wherein computing the first perception profile corresponding to the first image using the virtual eye model comprising:
 projecting the first image onto the retina-approximating surface using the lens-approximating component, wherein the retina-approximating surface is associated with sampling points approximating photoreceptors of an eye, wherein each sampling point has a corresponding photoreceptor type;   sampling color information from the projected source image at the sampling points, wherein the color information sampled at each sampling point depends on the corresponding photoreceptor type;   accessing the RGC-approximating pooling units, wherein each pooling unit is associated with groups of one or more of the sampling points;   calculating, for each of the RGC-approximating pooling units, weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with the RGC-approximating pooling unit; and   computing a perception profile for the first image based on the weighted aggregations associated with each of the RGC-approximating pooling units.   
     
     
         37 . The media of  claim 36 , wherein the lens-approximating component projects the first image onto a curved shape surface and maps the first image projected onto the curved shape surface into an image on the retina-approximating surface. 
     
     
         38 . The media of  claim 36 , wherein the lens-approximating component comprises an Optical Transfer Function (OTF) and a warping operator, wherein the OTF convolves a matrix representing pixel information of the first image with convolution templates and multiplies with a coding matrix, wherein the convolution templates and the coding matrix are obtained by a matrix factorization, and wherein the warping operator maps the first image projected onto the curved shape surface into the image on the retina-approximating surface. 
     
     
         39 . The media of  claim 36 , wherein a distribution of a type of the sampling points is associated with a corresponding blue noise mask. 
     
     
         40 . A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
 generate a first image using a machine-learning model;   compute a first perception profile corresponding to the first image using a virtual eye model, wherein the virtual eye model comprises a lens-approximating component, a retina-approximating surface, and retinal ganglion cell (RGC)-approximating pooling units;   access a ground truth image corresponding to the first image as a second image;   compute a second perception profile corresponding to the second image using the virtual eye model;   measure differences between the first perception profile and the second perception profile; and   update trainable variables of the machine-learning model based on the measured differences.