Video See-Through Augmented Reality

Abstract

In one embodiment, a method includes accessing an image captured by a camera of a pair of stereoscopic cameras of a video see-through augmented-reality (AR) system. The method further includes determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image and generating, based on the modification to the image, a transformation map for the camera that captured the accessed image. The transformation map identifies frame-independent transformations to apply for rendering a scene based on one or more subsequent images captured by that camera.

What is claimed is: 
     
         1 . A method comprising:
 accessing an image captured by a camera of a pair of stereoscopic cameras of a video see-through augmented-reality (AR) system;   determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image;   generating, based on the modification to the image, a transformation map for the camera that captured the accessed image, wherein the transformation map identifies frame-independent transformations to apply for rendering a scene based on one or more subsequent images captured by that camera; and   storing, in a memory associated with the video see-through AR system, a copy of the transformation map for that system.   
     
     
         2 . The method of  claim 1 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises determining, using a model of the see-through camera that is based on one or more parameters of that camera, a correction to a distortion of the image caused by that camera. 
     
     
         3 . The method of  claim 1 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises determining a rectification for the image based on one or more of:
 a model of the see-through camera that is based on one or more parameters of that camera;   a model of the pair of stereoscopic cameras of the video see-through AR system; or   a model of a virtual camera located at an eye position for viewing content on the video see-through AR system.   
     
     
         4 . The method of  claim 3 , wherein:
 the model of the pair of stereoscopic cameras of the video see-through AR system is used for (1) transforming stereo image pair captured by the stereo camera pair onto a plane and (2) making corresponding epipolar lines in the stereo image pair collinear; and   the model of virtual camera maps image coordinates onto an image plane shared by the stereo camera pair.   
     
     
         5 . The method of  claim 1 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises transforming a coordinate system of the image from a viewpoint of the camera to a viewpoint of a virtual camera located at an eye position for viewing content on the video see-through AR system based on: (1) a model of the virtual camera that is based on one or more parameters of the virtual camera and (2) a model of a rotation and a translation of the virtual camera relative to the camera of the pair of stereoscopic cameras 
     
     
         6 . The method of  claim 1 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises determining, using a model of a display lens of the video see-through AR system, a correction to a distortion of the image caused by that lens. 
     
     
         7 . The method of  claim 1 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises:
 determining, using a model of the see-through camera that is based on one or more parameters of that camera, a correction to a distortion of the image caused by that camera;   determining a rectification for the image based on each of:
 a model of the see-through camera that is based on one or more parameters of that camera; 
 a model of the pair of stereoscopic cameras of the video see-through AR system; and 
 a model of a virtual camera located at an eye position for viewing content on the video see-through AR system; 
   transforming a coordinate system of the image from a viewpoint of the camera to viewpoint of a virtual camera located at an eye position for viewing content on the video see-through AR system based on:
 a model of the virtual camera that is based on one or more parameters of the virtual camera; and 
 a model of a rotation and a translation of the virtual camera relative to the camera of the pair of stereoscopic cameras; and 
   determining, using a model of a display lens of the video see-through AR system, a correction to a distortion of the image caused by that lens.   
     
     
         8 . The method of  claim 1 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, the modification to the image comprises:
 determining a mesh representing a plurality of coordinates of the image; and   determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the mesh.   
     
     
         9 . One or more non-transitory computer readable storage media embodying instructions and coupled to one or more processors that are operable to execute the instructions to:
 access an image captured by a camera of a pair of stereoscopic cameras of a video see-through augmented-reality (AR) system;   determine, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image; and   generate, based on the modification to the image, a transformation map for the camera that captured the accessed image, wherein the transformation map identifies frame-independent transformations to apply for rendering a scene based on one or more subsequent images captured by that camera.   
     
     
         10 . The media of  claim 9 , wherein the media further embodies instructions that the one or more processors are operable to execute to:
 determine, using a model of the see-through camera that is based on one or more parameters of that camera, a correction to a distortion of the image caused by that camera;   determine a rectification for the image based on each of:
 a model of the see-through camera that is based on one or more parameters of that camera; 
 a model of the pair of stereoscopic cameras of the video see-through AR system; and 
 a model of a virtual camera located at an eye position for viewing content on the video see-through AR system; 
   transform a coordinate system of the image from a viewpoint of the camera to viewpoint of a virtual camera located at an eye position for viewing content on the video see-through AR system based on:
 a model of the virtual camera that is based on one or more parameters of the virtual camera; and 
 a model of a rotation and a translation of the virtual camera relative to the camera of the pair of stereoscopic cameras; and 
   determine, using a model of a display lens of the video see-through AR system, a correction to a distortion of the image caused by that lens.   
     
     
         11 . A system comprising:
 one or more non-transitory computer readable storage media embodying instructions; and   one or more processors coupled to the non-transitory computer readable storage media, the one or more processors being operable to execute the instructions to:   access an image captured by a camera of a pair of stereoscopic cameras of a video see-through augmented-reality (AR) system;   determine, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image; and   generate, based on the modification to the image, a transformation map for the camera that captured the accessed image, wherein the transformation map identifies frame-independent transformations to apply for rendering a scene based on one or more subsequent images captured by that camera.   
     
     
         12 . The system of  claim 11 , wherein the media further embodies instructions that the one or more processors are operable to execute to:
 determine, using a model of the see-through camera that is based on one or more parameters of that camera, a correction to a distortion of the image caused by that camera;   determine a rectification for the image based on each of:
 a model of the see-through camera that is based on one or more parameters of that camera; 
 a model of the pair of stereoscopic cameras of the video see-through AR system; and 
 a model of a virtual camera located at an eye position for viewing content on the video see-through AR system; 
   transform a coordinate system of the image from a viewpoint of the camera to viewpoint of a virtual camera located at an eye position for viewing content on the video see-through AR system based on:
 a model of the virtual camera that is based on one or more parameters of the virtual camera; and 
 a model of a rotation and a translation of the virtual camera relative to the camera of the pair of stereoscopic cameras; and 
   determine, using a model of a display lens of the video see-through AR system, a correction to a distortion of the image caused by that lens.   
     
     
         13 . A method comprising:
 accessing an image captured by a camera of a pair of stereoscopic cameras of a video see-through augmented-reality (AR) system;   transforming the image using a predetermined transformation map that identifies frame-independent transformations to apply to images captured by the camera; and   displaying, on a display on the video see-through AR system, the transformed image.   
     
     
         14 . The method of  claim 13 , wherein the transformation map is predetermined by:
 using a previous image captured by the camera of a pair of stereoscopic cameras of a video see-through augmented-reality (AR) system;   determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image; and   generating, based on the modification to the image, the transformation map for the camera that captured the previous image.   
     
     
         15 . The method of  claim 14 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises determining, using a model of the see-through camera that is based on one or more parameters of that camera, a correction to a distortion of the image caused by that camera. 
     
     
         16 . The method of  claim 14 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises determining a rectification for the image based on one or more of:
 a model of the see-through camera that is based on one or more parameters of that camera;   a model of the pair of stereoscopic cameras of the video see-through AR system; or   a model of a virtual camera located at an eye position for viewing content on the video see-through AR system.   
     
     
         17 . The method of  claim 16 , wherein:
 the model of the pair of stereoscopic cameras of the video see-through AR system is used for (1) transforming stereo image pair captured by the stereo camera pair onto a plane and (2) making corresponding epipolar lines in the stereo image pair collinear; and   the model of virtual camera maps image coordinates onto an image plane shared by the stereo camera pair.   
     
     
         18 . The method of  claim 16 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises transforming a coordinate system of the image from a viewpoint of the camera to a viewpoint of a virtual camera located at an eye position for viewing content on the video see-through AR system based on: (1) a model of the virtual camera that is based on one or more parameters of the virtual camera and (2) a model of a rotation and a translation of the virtual camera relative to the camera of the pair of stereoscopic cameras 
     
     
         19 . The method of  claim 14 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises determining, using a model of a display lens of the video see-through AR system, a correction to a distortion of the image caused by that lens. 
     
     
         20 . The method of  claim 14 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the image comprises:
 determining, using a model of the see-through camera that is based on one or more parameters of that camera, a correction to a distortion of the image caused by that camera;   determining a rectification for the image based on each of:
 a model of the see-through camera that is based on one or more parameters of that camera; 
 a model of the pair of stereoscopic cameras of the video see-through AR system; and 
 a model of a virtual camera located at an eye position for viewing content on the video see-through AR system; 
   transforming a coordinate system of the image from a viewpoint of the camera to viewpoint of a virtual camera located at an eye position for viewing content on the video see-through AR system based on:
 a model of the virtual camera that is based on one or more parameters of the virtual camera; and 
 a model of a rotation and a translation of the virtual camera relative to the camera of the pair of stereoscopic cameras; and 
   determining, using a model of a display lens of the video see-through AR system, a correction to a distortion of the image caused by that lens.   
     
     
         21 . The method of  claim 14 , wherein determining, based on each of one or more models of one or more components of the video see-through AR system, the modification to the image comprises:
 determining a mesh representing a plurality of coordinates of the image; and   determining, based on each of one or more models of one or more components of the video see-through AR system, a modification to the mesh.