Artificial reality system with varifocal display of artificial reality content

Abstract

The disclosure describes artificial reality systems and techniques for providing artificial reality content to a user. For example, an artificial reality system includes a head-mounted display (HMD) configured to output artificial reality content, the HMD including a set of second image capture devices configured to capture image data indicative of a focal point of a gaze of the user and a varifocal display having a focal length that is modifiable based on the focal point of the user. Additionally, the system includes a depth engine configured to generate, based on the real-world image data and depth data associated with the real-world image data, a three-dimensional (3D) scene of the physical environment of the user and generate artificial reality content as an overlay to the 3D scene of the physical environment for display on the varifocal display of the HMD based on the focal point of the user.

What is claimed is: 
     
         1 . An artificial reality system comprising:
 a head-mounted display (HMD) configured to output artificial reality content, the HMD comprising:
 a set of image capture devices configured to capture image data indicative of a focal point of a user; 
 a varifocal display having a focal length that is modifiable based on the focal point of the user; and 
 a motor configured to control a position of the varifocal display to modify the focal length; and 
   a depth engine configured to:
 determine the focal length for the varifocal display to match the focal point of the user; 
 control the motor in order to move the varifocal display to achieve the determined focal length to match the focal point of the user; and 
 generate content for display on the varifocal display of the HMD based on the focal point of the user. 
   
     
     
         2 . The artificial reality system of  claim 1 , further comprising:
 a first image capture device configured to capture real-world image data representative of a physical environment of a user, wherein the set of image capture devices of the HMD is a set of second image capture devices,   wherein to generate the content for display on the varifocal display, the depth engine is configured to:
 generate, based on the real-world image data and depth data associated with the real-world image data, a three-dimensional (3D) scene of the physical environment of the user; and 
 generate the content as an overlay to the 3D scene of the physical environment for display on the varifocal display of the HMD, wherein the content is artificial reality content. 
   
     
     
         3 . The system of  claim 2 , wherein the first image capture device comprises a pass-through camera configured to capture the real-world image data in color. 
     
     
         4 . The system of  claim 3 , wherein the HMD comprises one or more processors configured to execute the depth engine, wherein the one or more processors are located proximate to the pass-through camera. 
     
     
         5 . The system of  claim 2 , wherein the HMD further comprises a depth sensor configured to generate the depth data, and wherein to generate the 3D scene of the physical environment, the depth engine is configured to:
 generate, using the depth data, a 3D mesh of a real-world scene; and   overlay at least a portion of the real-world image data onto the 3D mesh of the real-world scene.   
     
     
         6 . The system of  claim 5 , wherein the real-world image data represents a stream of real-world image data, wherein the depth data represents a stream of depth data, and wherein to generate the artificial reality content, the depth engine is further configured to:
 generate, using the stream of real-world image data and the stream of depth data, the artificial reality content in near real-time based on a position and an orientation of the HMD with respect to the 3D mesh of the real-world scene.   
     
     
         7 . The system of  claim 1 , wherein the set of image capture devices comprises:
 a first eye-tracking camera configured to capture a first set of image data including a position of a first pupil of the user in relation to a first set of reference points; and   a second eye-tracking camera configured to capture a second set of image data including a position of a second pupil of the user in relation to a second set of reference points,   wherein the depth engine is configured to determine the focal point of the user based on the position of the first pupil in relation to the first set of reference points and the position of the second pupil in relation to the second set of reference points.   
     
     
         8 . The system of  claim 7 , further comprising a gaze tracker configured to:
 track the position of the first pupil in relation to the first set of reference points over a period of time;   track the position of the second pupil in relation to the second set of reference points over the period of time;   determine, based on a movement of the position of the first pupil in relation to the first set of reference points over the period of time, a projected future movement of the position of the first pupil in relation to the first set of reference points; and   determine, based on a movement of the position of the second pupil in relation to the second set of reference points over the period of time, a projected future movement of the position of the second pupil in relation to the second set of reference points,   wherein the depth engine is further configured to determine, based on the projected future movement of the position of the first pupil and the projected future movement of the position of the second pupil, a projected future focal point of the user.   
     
     
         9 . The system of  claim 1 , wherein the image data is further indicative of a depth of field of the user, and
 wherein to generate the artificial reality content, the depth engine is configured to blur portions of the artificial reality content that are outside of the depth of field of the user.   
     
     
         10 . The system of  claim 1 , wherein the depth engine is configured to control the motor in order to move the varifocal display along a longitudinal of the HMD relative to a lens. 
     
     
         11 . A method comprising:
 capturing, by a set of image capturing devices of a head-mounted display (HMD) configured to output artificial reality content, image data indicative of a focal point of a user; and   modifying, by a depth engine of the HMD, a focal length of a varifocal display of the HMD based on the focal point of the user, wherein modifying the focal length of the varifocal display comprises:
 determining, by the depth engine, the focal length for the varifocal display to match the focal point of the user; 
 controlling, by the depth engine, a motor of the HMD in order to move the varifocal display to achieve the determined focal length to match the focal point of the user, wherein the motor is configured to control a position of the varifocal display to modify the focal length; and 
 generating, by the depth engine, content for display on the varifocal display of the HMD based on the focal point of the user. 
   
     
     
         12 . The method of  claim 11 , further comprising:
 capturing, by a first image capture device, real-world image data representative of a physical environment of a user, wherein the set of image capture devices of the HMD is a set of second image capture devices,   wherein generating the content for display on the varifocal display comprises:
 generating, based on the real-world image data and depth data associated with the real-world image data, a three-dimensional (3D) scene of the physical environment of the user; and 
 generating the content as an overlay to the 3D scene of the physical environment for display on the varifocal display of the HMD, wherein the content is artificial reality content. 
   
     
     
         13 . The method of  claim 12 , wherein the first image capture device comprises a pass-through camera, and wherein the method comprises capturing, by the pass-through camera, the real-world image data in color. 
     
     
         14 . The method of  claim 13 , further comprising executing, by one or more processors of the HMD, the depth engine, wherein the one or more processors are located proximate to the pass-through camera. 
     
     
         15 . The method of  claim 12 , further comprising:
 generating, by a depth sensor of the HMD, the depth data,   wherein generating the 3D scene of the physical environment comprises:
 generating, using the depth data, a 3D mesh of a real-world scene; and 
 overlaying at least a portion of the real-world image data onto the 3D mesh of the real-world scene. 
   
     
     
         16 . The method of  claim 15 , wherein the real-world image data represents a stream of real-world image data, wherein the depth data represents a stream of depth data, and wherein generating the artificial reality content comprises:
 generating, using the stream of real-world image data and the stream of depth data, the artificial reality content in near real-time based on a position and an orientation of the HMD with respect to the 3D mesh of the real-world scene.   
     
     
         17 . The method of  claim 11 , further comprising:
 capturing, by a first eye-tracking camera of the set of image capture devices, a first set of image data including a position of a first pupil of the user in relation to a first set of reference points;   capturing, by a second eye-tracking camera of the set of image capture devices, a second set of image data including a position of a second pupil of the user in relation to a second set of reference points; and   determining, by the depth engine, the focal point of the user based on the position of the first pupil in relation to the first set of reference points and the position of the second pupil in relation to the second set of reference points.   
     
     
         18 . The method of  claim 17 , further comprising:
 tracking, by a gaze tracker, the position of the first pupil in relation to the first set of reference points over a period of time;   tracking, by the gaze tracker, the position of the second pupil in relation to the second set of reference points over the period of time;   determining, by the gaze tracker based on a movement of the position of the first pupil in relation to the first set of reference points over the period of time, a projected future movement of the position of the first pupil in relation to the first set of reference points;   determining, by the gaze tracker based on a movement of the position of the second pupil in relation to the second set of reference points over the period of time, a projected future movement of the position of the second pupil in relation to the second set of reference points; and   determining, by the depth engine based on the projected future movement of the position of the first pupil and the projected future movement of the position of the second pupil, a projected future focal point of the user.   
     
     
         19 . The method of  claim 11 , wherein the image data is further indicative of a depth of field of the user, and
 wherein generating the artificial reality content comprises blurring portions of the artificial reality content that are outside of the depth of field of the user.   
     
     
         20 . A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to:
 capture image data indicative of a focal point of a user; and   modify a focal length of a varifocal display of a head-mounted display (HMD) based on the focal point of the user, wherein to modify the focal length of the varifocal display, the instructions cause the one or more processors to:
 determine the focal length for the varifocal display of the HMD to match the focal point of the user; 
 control a motor of the HMD in order to move the varifocal display to achieve the determined focal length to match the focal point of the user, wherein the motor is configured to control a position of the varifocal display to modify the focal length; and 
 generate content for display on the varifocal display based on the focal point of the user.