Joint Environmental Reconstruction and Camera Calibration

Abstract

In one embodiment, a method includes accessing a calibration model for a camera rig. The method includes accessing multiple observations of an environment captured by the camera rig from multiple poses in the environment. The method includes generating an environmental model including geometry of the environment based on at least the observations, the poses, and the calibration model. The method includes determining, for one or more of the poses, one or more predicted observations of the environment based on the environmental model and the poses. The method includes comparing the predicted observations to the observations corresponding to the poses from which the predicted observations were determined. The method includes revising the calibration model based on the comparison. The method includes revising the environmental model based on at least a set of observations of the environment and the revised calibration model.

1 - 20 . (canceled) 
     
     
         21 . A method comprising, by a computing device associated with a camera rig:
 determining that a calibration model for the camera rig needs to be revised;   calculating one or more predictions for observations of an environment based on an environment model generated based at least on a first plurality of observations captured by the camera rig from a plurality of poses in the environment;   revising the calibration model to reduce a sum of squared errors in comparison between the calculated one or more predictions and one or more of a second plurality of observations of the environment corresponding to the one or more poses from which the calculated one or more predictions were determined; and   revising the environmental model based at least on the second plurality of observations of the environment and the revised calibration model.   
     
     
         22 . The method of  claim 21 , further comprising capturing the first plurality of captured observations by:
 projecting, by the camera rig, a structured light pattern into the environment;   detecting, by a camera of the camera rig, the projected structured light pattern in the environment; and   comparing the detected structured light pattern to a template structured light pattern.   
     
     
         23 . The method of  claim 21 , further comprising capturing the first plurality of captured observations by:
 projecting, by the camera rig, a structured light pattern comprising a plurality of points into the environment;   detecting, by a camera of the camera rig, one or more points of the structured light pattern in the environment;   identifying each of the detected points; and   for each identified point, comparing a location of the identified point to a corresponding expected bearing of the identified point in the structured light pattern.   
     
     
         24 . The method of  claim 21 , further comprising:
 capturing a pose of the camera rig from which each first observation was captured by receiving the pose of the camera rig from a localization system of the camera rig.   
     
     
         25 . The method of  claim 24 , further comprising:
 initializing the localization system of the camera rig based on receiving a movement of the camera rig from a movement sensor of the camera rig.   
     
     
         26 . The method of  claim 25 , wherein the movement sensor of the camera rig comprises:
 an accelerometer;   a gyroscope;   an ultra-sonic movement sensor;   a magnetometer; or   an optical movement sensor.   
     
     
         27 . The method of  claim 21 , further comprising:
 capturing a pose of the camera rig from which each observation was captured by localizing the camera rig based at least on the observation and the calibration model.   
     
     
         28 . The method of  claim 21 , wherein the environmental model comprising geometry of the environment, and the environment model is generated by:
 generating an estimation of the geometry of the environment from each observation and the pose of the camera rig from which the observation was captured; and   combining the estimations generated from the first plurality of observation to form the environmental model.   
     
     
         29 . The method of  claim 21 , wherein the calculated one or more predictions and the first plurality of captured observations comprise location information for points of a structured light pattern; and
 wherein the comparison between the calculated one or more predictions and the one or more of the first plurality of captured observations corresponding to the one or more poses from which the calculated one or more predictions were determined comprises comparing the location information of each calculated prediction with the location information of the respective corresponding first observation.   
     
     
         30 . The method of  claim 21 , wherein the calibration model for the camera rig comprises:
 intrinsic parameters for a camera of the camera rig;   intrinsic parameters for an emitter of the camera rig;   parameters for performing localization of the camera rig in an environment; or   parameters associated with a relationship between the camera and emitter of the camera rig.   
     
     
         31 . The method of  claim 30 , wherein revising the calibration model comprises:
 modifying one or more of the parameters of the calibration model to minimize a difference between the calculated one or more predictions and the one or more of the first plurality of captured observations corresponding to the one or more poses from which the calculated one or more predictions were determined.   
     
     
         32 . The method of  claim 21 , wherein revising the calibration model comprises:
 calculating a proposed revised calibration model;   capturing the one or more of the second plurality of observations with the camera rig using the proposed revised calibration model from one or more poses in the environment;   calculating a model error in comparison between the calculated one or more predictions and the one or more of the second plurality of observations; and   determining that the calculated model error satisfies a revision threshold corresponding to a likelihood that revising the calibration model will improve the calibration model.   
     
     
         33 . The method of  claim 21 , wherein revising the environmental model based at least on the second plurality of observations of the environment and the revised calibration model comprises:
 generating a proposed revised environmental model from the second plurality of observations, a second plurality of poses from which the observations were collected, and the revised calibration model; and   comparing the environmental model and the proposed revised environmental model.   
     
     
         34 . The method of  claim 21 , further comprising:
 receiving a movement of the camera rig from the movement sensor of the camera rig, wherein the environmental model is revised responsive to receiving the movement of the camera rig from the movement sensor of the camera rig.   
     
     
         35 . The method of  claim 34 , further comprising:
 prior to revising the environmental model, determining that the movement is not an erroneous movement by comparing the movement of the camera rig to movement of the camera rig determined based at least on the revised camera calibration model and the second plurality of observations.   
     
     
         36 . The method of  claim 21 , wherein a camera of the camera rig is configured to detect infrared light or ultraviolet light. 
     
     
         37 . The method of  claim 21 , wherein the camera rig is incorporated into a head-mounted device. 
     
     
         38 . The method of  claim 21 , wherein the camera rig is incorporated into a hand-held computing device. 
     
     
         39 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 determine that a calibration model for the camera rig needs to be revised;   calculate one or more predictions for observations of an environment based on an environment model generated based at least on a first plurality of observations captured by the camera rig from a plurality of poses in the environment;   revise the calibration model to reduce a sum of squared errors in comparison between the calculated one or more predictions and one or more of a second plurality of observations of the environment corresponding to the one or more poses from which the calculated one or more predictions were determined; and   revise the environmental model based at least on the second plurality of observations of the environment and the revised calibration model.   
     
     
         40 . A system comprising:
 one or more processors; and   one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
 determine that a calibration model for the camera rig needs to be revised; 
 calculate one or more predictions for observations of an environment based on an environment model generated based at least on a first plurality of observations captured by the camera rig from a plurality of poses in the environment; 
 revise the calibration model to reduce a sum of squared errors in comparison between the calculated one or more predictions and one or more of a second plurality of observations of the environment corresponding to the one or more poses from which the calculated one or more predictions were determined; and 
 revise the environmental model based at least on the second plurality of observations of the environment and the revised calibration model.