Video See-Through Augmented Reality

Abstract

In one embodiment, a method includes capturing, using a pair of cameras of a video see-through AR system, stereo images of a predetermined calibration object having a predetermined pose and obtaining, by a position sensor, position data for each of the pair of cameras at a time when the stereo images are captured. The method further includes generating a 3D reconstruction of the calibration object. The method further includes performing a registration process by generating, using parameters of each of a pair of stereo virtual cameras, stereo virtual views that include the predetermined calibration object and determining, for each of the stereo virtual views, one or more differences between the predetermined calibration object in the virtual images and a virtual rendering of the calibration object, and then using those differences to either adjust the virtual-camera parameters and repeat the registration process or to store the parameters for the system.

What is claimed is: 
     
         1 . A method comprising:
 capturing, using a pair of cameras of a video see-through AR system, stereo images of a predetermined calibration object having a predetermined pose;   obtaining, by a position sensor, position data for each of the pair of cameras at a time when the stereo images are captured;   generating a 3D reconstruction of the calibration object based on the stereo images and the position data;   performing a virtual object registration process comprising:
 generating, using parameters of each of a pair of stereo virtual cameras, stereo virtual views comprising the predetermined calibration object based on the captured stereo images, the position data, and the 3D reconstruction, wherein each virtual camera of the pair of stereo virtual cameras is located at an eye position for viewing content on the video see-through AR system; 
 determining, for each of the stereo virtual views, one or more differences between the predetermined calibration object in the virtual images and a virtual rendering of the calibration object; 
   when at least one of the determined one or more differences is greater than a corresponding threshold, then adjusting one or more parameters of at least one of the pair of stereo virtual cameras and re-executing the virtual object registration process; and   when none of the one or more differences is greater than a corresponding threshold, then storing the parameters of each of the pair of stereo virtual cameras in association with the video see-through AR system.   
     
     
         2 . The method of  claim 1 , further comprising:
 determining depth information of the virtual calibration object based on the stereo images and a depth scale factor;   determining a difference between the depth information of the virtual object and known depth information of the predetermined calibration object; and   adjusting the depth scale factor based on the difference.   
     
     
         3 . The method of  claim 1 , wherein the position sensor comprises an inertial measurement unit. 
     
     
         4 . The method of  claim 1 , wherein the parameters of each of the pair of stereo virtual cameras comprise a camera matrix for each virtual camera and a distortion model for each virtual camera. 
     
     
         5 . The method of  claim 1 , wherein the one or more differences comprise a difference in position. 
     
     
         6 . The method of  claim 1 , wherein the one or more differences comprise a difference in orientation. 
     
     
         7 . The method of  claim 1 , wherein the one or more differences comprise a difference in distortion. 
     
     
         8 . One or more non-transitory computer readable storage media embodying instructions and coupled to one or more processors that are operable to execute the instructions to:
 capture, using a pair of cameras of a video see-through AR system, stereo images of a predetermined calibration object having a predetermined pose;   obtain, by a position sensor, position data for each of the pair of cameras at a time when the stereo images are captured;   generate a 3D reconstruction of the calibration object based on the stereo images and the position data;   execute a virtual object registration process comprising:
 generating, using parameters of each of a pair of stereo virtual cameras, stereo virtual views comprising the predetermined calibration object based on the captured stereo images, the position data, and the 3D reconstruction, wherein each virtual camera of the pair of stereo virtual cameras is located at an eye position for viewing content on the video see-through AR system; 
 determining, for each of the stereo virtual views, one or more differences between the predetermined calibration object in the virtual images and a virtual rendering of the calibration object; 
   when at least one of the determined one or more differences is greater than a corresponding threshold, then adjust one or more parameters of at least one of the pair of stereo virtual cameras and re-executing the virtual object registration process; and   when none of the one or more differences is greater than a corresponding threshold, then store the parameters of each of the pair of stereo virtual cameras in association with the video see-through AR system.   
     
     
         9 . The media of  claim 8 , wherein the media further embodies instructions that the one or more processors are operable to execute to:
 determine depth information of the virtual calibration object based on the stereo images and a depth scale factor;   determine a difference between the depth information of the virtual object and known depth information of the predetermined calibration object; and   adjust the depth scale factor based on the difference.   
     
     
         10 . The media of  claim 8 , wherein the parameters of each of the pair of stereo virtual cameras comprise a camera matrix for each virtual camera and a distortion model for each virtual camera. 
     
     
         11 . The media of  claim 8 , wherein the one or more differences comprise a difference in position. 
     
     
         12 . A system comprising:
 one or more non-transitory computer readable storage media embodying instructions; and   one or more processors coupled to the non-transitory computer readable storage media, the one or more processors being operable to execute the instructions to:   capture, using a pair of cameras of a video see-through AR system, stereo images of a predetermined calibration object having a predetermined pose;   obtain, by a position sensor, position data for each of the pair of cameras at a time when the stereo images are captured;   generate a 3D reconstruction of the calibration object based on the stereo images and the position data;   execute a virtual object registration process comprising:
 generating, using parameters of each of a pair of stereo virtual cameras, stereo virtual views comprising the predetermined calibration object based on the captured stereo images, the position data, and the 3D reconstruction, wherein each virtual camera of the pair of stereo virtual cameras is located at an eye position for viewing content on the video see-through AR system; 
 determining, for each of the stereo virtual views, one or more differences between the predetermined calibration object in the virtual images and a virtual rendering of the calibration object; 
   when at least one of the determined one or more differences is greater than a corresponding threshold, then adjust one or more parameters of at least one of the pair of stereo virtual cameras and re-executing the virtual object registration process; and   when none of the one or more differences is greater than a corresponding threshold, then store the parameters of each of the pair of stereo virtual cameras in association with the video see-through AR system.   
     
     
         13 . The system of  claim 12 , wherein the media further embodies instructions that the one or more processors are operable to execute to:
 determining depth information of the virtual calibration object based on the stereo images and a depth scale factor;   determining a difference between the depth information of the virtual object and known depth information of the predetermined calibration object; and   adjusting the depth scale factor based on the difference.   
     
     
         14 . The system of  claim 12 , wherein the parameters of each of the pair of stereo virtual cameras comprise a camera matrix for each virtual camera and a distortion model for each virtual camera. 
     
     
         15 . A method comprising:
 accessing an image of a real-world scene captured by a see-through camera of a video see-through AR system;   rendering one or more virtual objects for display within the real-world scene; and   rendering for display on a display of the video see-through AR system an image of the real-world scene blended with the one or more virtual objects, wherein the rendering is based on one or more predetermined parameters of a virtual camera associated with the display.   
     
     
         16 . The method of  claim 15 , wherein the one or more predetermined parameters of the virtual camera have been determined by a process comprising:
 capturing, using a pair of cameras of the video see-through AR system, stereo images of a predetermined calibration object having a predetermined pose;   obtaining, by a position sensor, position data for each of the pair of cameras at a time when the stereo images are captured;   generating a 3D reconstruction of the calibration object based on the stereo images and the position data;   performing a virtual object registration process comprising:
 generating, using parameters of each of a pair of stereo virtual cameras, stereo virtual views comprising the predetermined calibration object based on the captured stereo images, the position data, and the 3D reconstruction, wherein each virtual camera of the pair of stereo virtual cameras is located at an eye position for viewing content on the video see-through AR system; 
 determining, for each of the stereo virtual views, one or more differences between the predetermined calibration object in the virtual images and a virtual rendering of the calibration object; 
   when at least one of the determined one or more differences is greater than a corresponding threshold, then adjusting one or more parameters of at least one of the pair of stereo virtual cameras and re-executing the virtual object registration process; and   when none of the one or more differences is greater than a corresponding threshold, then storing the parameters of each of the pair of stereo virtual cameras in association with the video see-through AR system.   
     
     
         17 . The method of  claim 16 , wherein the parameters of each of the pair of stereo virtual cameras comprise a camera matrix for each virtual camera and a distortion model for each virtual camera. 
     
     
         18 . The method of  claim 16 , wherein the one or more differences comprise a difference in position. 
     
     
         19 . The method of  claim 16 , wherein the one or more differences comprise a difference in orientation. 
     
     
         20 . The method of  claim 16 , wherein the one or more differences comprise a difference in distortion.