Display apparatus and method of displaying using gaze prediction and image steering

Abstract

A display apparatus including configuration of gaze sensors; gaze predictor module configured to process sensor data collected by aforesaid configuration to determine current gaze location and gaze velocity and/or acceleration, and to predict gaze location and gaze velocity and/or acceleration of user; image processing module configured to process input image for generating first image having first resolution and second image having second resolution, second resolution being higher than first resolution; first and second image renderers that render first and second image, respectively; optical combiner for optically combining projections of first and second images; and image steering unit configured to determine region of optical combiner onto which projection of second image is to be focused, and to make adjustment to focus projection of second image on said region; wherein second image renderer is switched off or dimmed during adjusting phase of image steering unit when said unit is making adjustment.

1 . A display apparatus comprising:
 a configuration of gaze sensors;   a gaze predictor module configured to process sensor data collected by the configuration of gaze sensors to determine a current gaze location and a current gaze velocity and/or acceleration of a user, and to predict a gaze location and a gaze velocity and/or acceleration of the user, based at least partially upon the current gaze location and the current gaze velocity and/or acceleration;   an image processing module configured to process an input image, based upon the predicted gaze location and the predicted gaze velocity and/or acceleration, to generate at least a first image and a second image, wherein the first image has a first resolution, while the second image has a second resolution, the second resolution being higher than the first resolution;   at least one first image renderer and at least one second image renderer that, in operation, render the first image and the second image, respectively;   at least one optical combiner for optically combining a projection of the first image with a projection of the second image; and   an image steering unit configured to determine, based upon the predicted gaze location and the predicted gaze velocity and/or acceleration, a region of the at least one optical combiner onto which the projection of the second image is to be focused, and to make an adjustment to focus the projection of the second image on said region of the at least one optical combiner;   
       wherein the at least one second image renderer is to be switched off or dimmed during an adjusting phase of the image steering unit when the image steering unit is making the adjustment. 
     
     
         2 . The display apparatus of  claim 1 , wherein the at least one second image renderer is to be switched on or brightened during a focus phase of the image steering unit when the adjustment has been made. 
     
     
         3 . The display apparatus of  claim 1 , wherein the image processing module is configured to process a subsequent input image to generate only a single image, when the predicted gaze velocity and/or acceleration is indicative of a saccadic movement of the user's eyes, and wherein the at least one first image renderer, in operation, renders the single image, further wherein the at least one second image renderer is to be switched off or dimmed during the saccadic movement of the user's eyes. 
     
     
         4 . The display apparatus of  claim 1 , wherein the gaze predictor module is configured to predict the gaze location and the gaze velocity and/or acceleration of the user, based also upon information pertaining to a visual scene being presented to the user. 
     
     
         5 . The display apparatus of  claim 4 , wherein the information pertaining to the visual scene comprises information indicative of a location of an object present in the visual scene that has at least one of: an audio feature of interest, a visual feature of interest, a physical interaction with another object present in the visual scene. 
     
     
         6 . The display apparatus of  claim 1 , wherein the image processing module is configured to determine a region of interest of the input image, based upon the predicted gaze location and the predicted gaze velocity and/or acceleration, wherein the first image and the second image are to be generated in a manner that the second image substantially corresponds to the region of interest of the input image, whilst the first image substantially corresponds to an entirety of the input image. 
     
     
         7 . The display apparatus of  claim 6 , wherein the first image is to be generated in a manner that a region of the first image that substantially corresponds to the region of interest of the input image is masked, wherein the projection of the second image is to substantially overlap with a projection of the masked region of the first image on the at least one optical combiner. 
     
     
         8 . The display apparatus of  claim 1 , wherein the image processing module is configured to generate the first image and the second image, based also upon an operational status of the at least one first image renderer and the at least one second image renderer. 
     
     
         9 . The display apparatus of  claim 1 , wherein the image steering unit comprises at least one actuator for moving at least one of:
 the at least one second image renderer with respect to the at least one optical combiner,   the at least one optical combiner,   at least one optical element positioned on an optical path between the at least one second image renderer and the at least one optical combiner.   
     
     
         10 . A method of displaying, via a display apparatus, the method comprising:
 processing sensor data collected by a configuration of gaze sensors of the display apparatus to determine a current gaze location and a current gaze velocity and/or acceleration of a user;   predicting a gaze location and a gaze velocity and/or acceleration of the user, based at least partially upon the current gaze location and the current gaze velocity and/or acceleration;   processing an input image, based upon the predicted gaze location and the predicted gaze velocity and/or acceleration, to generate at least a first image and a second image, wherein the first image has a first resolution, while the second image has a second resolution, the second resolution being higher than the first resolution;   rendering, via at least one first image renderer and at least one second image renderer of the display apparatus, the first image and the second image, respectively, wherein a projection of the first image is optically combined with a projection of the second image using at least one optical combiner of the display apparatus;   determining, based upon the predicted gaze location and the predicted gaze velocity and/or acceleration, a region of the at least one optical combiner onto which the projection of the second image is to be focused;   employing an image steering unit of the display apparatus to make an adjustment to focus the projection of the second image on said region of the at least one optical combiner; and   switching off or dimming the at least one second image renderer during an adjusting phase of the image steering unit when the image steering unit is making the adjustment.   
     
     
         11 . The method of  claim 10 , further comprising switching on or brightening the at least one second image renderer during a focus phase of the image steering unit when the adjustment has been made. 
     
     
         12 . The method of  claim 10 , further comprising:
 processing a subsequent input image to generate only a single image, when the predicted gaze velocity and/or acceleration is indicative of a saccadic movement of the user's eyes;   rendering the single image via the at least one first image renderer; and   switching off or dimming the at least one second image renderer during the saccadic movement of the user's eyes.   
     
     
         13 . The method of  claim 10 , wherein the step of predicting the gaze location and the gaze velocity and/or acceleration of the user is performed based also upon information pertaining to a visual scene being presented to the user. 
     
     
         14 . The method of  claim 13 , wherein the information pertaining to the visual scene comprises information indicative of a location of an object present in the visual scene that has at least one of: an audio feature of interest, a visual feature of interest, a physical interaction with another object present in the visual scene. 
     
     
         15 . The method of  claim 10 , further comprising determining a region of interest of the input image, based upon the predicted gaze location and the predicted gaze velocity and/or acceleration, wherein the first image and the second image are generated in a manner that the second image substantially corresponds to the region of interest of the input image, whilst the first image substantially corresponds to an entirety of the input image. 
     
     
         16 . The method of  claim 15 , wherein the first image is generated in a manner that a region of the first image that substantially corresponds to the region of interest of the input image is masked, wherein the projection of the second image substantially overlaps with a projection of the masked region of the first image on the at least one optical combiner. 
     
     
         17 . The method of  claim 10 , wherein the step of generating the first image and the second image is performed based also upon an operational status of the at least one first image renderer and the at least one second image renderer. 
     
     
         18 . The method of  claim 10 , further comprising employing at least one actuator of the image steering unit to move at least one of:
 the at least one second image renderer with respect to the at least one optical combiner,   the at least one optical combiner,   at least one optical element positioned on an optical path between the at least one second image renderer and the at least one optical combiner.