Reconstruction of Essential Visual Cues in Mixed Reality Applications

Abstract

A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

What is claimed is: 
     
         1 . A computer-implemented method, comprising:
 receiving two or more images of a real-world scene, wherein the real-world scene includes one or more objects;   generating at least one two-dimensional (2D) geometry, each corresponding to one or more of the two or more images and representing a view of the real-world scene;   generating a three-dimensional (3D) geometry based on the at least one 2D geometry, the 3D geometry comprising a partial representation of the one or more objects of the real-world scene;   generating additional 3D vertices for the 3D geometry based on extrapolating from 3D vertices included in the 3D geometry; and   rendering, based on the 3D geometry, one or more graphical representations of the real-world scene for display.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 identifying, in the two or more images, a first set of one or more features associated with the one or more objects and a second set of one or more features associated with the one or more objects.   
     
     
         3 . The computer-implemented method of  claim 2 ,
 wherein identifying the first set of one or more features comprises identifying at least one of an edge, a corner, or a surface within a first image of the two or more images; and   wherein identifying the second set of one or more features comprises identifying at least one of an edge, a corner or a surface within a second image of the two or more images.   
     
     
         4 . The computer-implemented method of  claim 2 , wherein generating a first of the at least one 2D geometry is based on the first set of one or more features, and wherein generating a second of the at least one 2D geometry is based on the second set of one or more features. 
     
     
         5 . The computer-implemented method of  claim 4 , wherein generating the 3D geometry includes determining a plurality of depth values based on A) the first of the at least one 2D geometry and B) the second of the at least one 2D geometry. 
     
     
         6 . The computer-implemented method of  claim 2 , wherein generating the at least one 2D geometry includes generating, for each feature of the first set of one or more features, a corresponding portion of the at least one 2D geometry. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein generating the 3D geometry includes generating a triangulated mesh based on the at least one 2D geometry. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein generating the 3D geometry further includes generating the 3D vertices included in the 3D geometry based on the at least one 2D geometry. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein rendering the one or more graphical representations includes extracting one or more textures from one or more of the two or more images. 
     
     
         10 . The computer-implemented method of  claim 1 , further comprising:
 receiving one or more additional images of the real-world scene;   updating the 3D geometry based on the one or more additional images; and   rendering, based on the updated 3D geometry, one or more second graphical representations of the real-world scene for display.   
     
     
         11 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process comprising:
 receiving one or more images of a real-world scene, wherein the real-world scene includes one or more objects;   generating at least one two-dimensional (2D) geometry, each corresponding to one or more of the one or more images and representing a view of the real-world scene;   generating a three-dimensional (3D) geometry based on the at least one 2D geometry, the 3D geometry comprising a partial representation of the one or more objects of the real-world scene;   generating additional 3D vertices for the 3D geometry based on extrapolating from 3D vertices included in the 3D geometry; and   rendering, based on the 3D geometry, one or more graphical representations of the real-world scene for display.   
     
     
         12 . The computer-readable storage medium of  claim 11 , wherein the process further comprises:
 identifying, in the one or more images, a first set of one or more features associated with the one or more objects and a second set of one or more features associated with the one or more objects.   
     
     
         13 . The computer-readable storage medium of  claim 12 ,
 wherein identifying the first set of one or more features comprises identifying at least one of an edge, a corner, or a surface within a first image of the one or more images; and   wherein identifying the second set of one or more features comprises identifying at least one of an edge, a corner or a surface within a second image of the one or more images.   
     
     
         14 . The computer-readable storage medium of  claim 12 , wherein generating a first of the at least one 2D geometry is based on the first set of one or more features, and wherein generating a second of the at least one 2D geometry is based on the second set of one or more features. 
     
     
         15 . The computer-readable storage medium of  claim 14 , wherein generating the 3D geometry includes determining a plurality of depth values based on A) the first of the at least one 2D geometry and B) the second of the at least one 2D geometry. 
     
     
         16 . The computer-readable storage medium of  claim 12 , wherein generating the at least one 2D geometry includes generating, for each feature of the first set of one or more features, a corresponding portion of the at least one 2D geometry. 
     
     
         17 . The computer-readable storage medium of  claim 11 , wherein generating the 3D geometry includes generating a triangulated mesh based on the at least one 2D geometry. 
     
     
         18 . A computing system comprising:
 one or more processors; and   one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
 receiving one or more images of a real-world scene, wherein the real-world scene includes one or more objects; 
 generating at least one two-dimensional (2D) geometry, each corresponding to one or more of the one or more images and representing a view of the real-world scene; 
 generating a three-dimensional (3D) geometry based on the at least one 2D geometry, the 3D geometry comprising a partial representation of the one or more objects of the real-world scene; 
 generating additional 3D vertices for the 3D geometry based on extrapolating from 3D vertices included in the 3D geometry; and 
 rendering, based on the 3D geometry, one or more graphical representations of the real-world scene for display. 
   
     
     
         19 . The computing system of  claim 18 , wherein rendering the one or more graphical representations includes extracting one or more textures from the one or more images. 
     
     
         20 . The computing system of  claim 18 , wherein the process further comprises:
 receiving one or more additional images of the real-world scene;   updating the 3D geometry based on the one or more additional images; and   rendering, based on the updated 3D geometry, one or more second graphical representations of the real-world scene for display.