Sensor calibration with environment map

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sensor calibration with environment map. In some implementations, a three-dimensional surfel representation of a real-world environment is obtained. One or more surfels of the surfel representation having a particular classification of the different classifications are selected. Input sensor data from one or more sensors installed on an autonomous or semi-autonomous vehicle are received. The input sensor data is compared to the surfel representation to identify one or more differences between the observation and the surfel representation. At least one sensor of the one or more sensors is calibrated using the one or more differences between the observation and the surfel representation.

What is claimed is: 
     
         1 . A computer implemented method comprising:
 obtaining a three-dimensional surfel representation of a real-world environment generated from initial sensor data captured by a plurality of vehicles traveling in the real-world environment, the surfel representation comprising a plurality of surfels,   receiving input sensor data from one or more sensors installed on a first vehicle, the input sensor data indicating an observation of the real-world environment;
 determining, from the input sensor data, one or more matching features in the sensor data that match corresponding features in the surfel representation; 
 computing, based on the one or more matching features, one or more differences between the sensor data and the surfel representation; and 
   calibrating at least one sensor of the one or more sensors using the one or more differences between the sensor data and the surfel representation.   
     
     
         2 . The method of  claim 1 , comprising updating the surfel representation based on the one or more surfels,
 wherein comparing the input sensor data to the surfel representation to identify the one or more differences between the observation and the surfel representation comprises comparing the input sensor data to the updated surfel representation to identify one or more differences between the observation and the updated surfel representation.   
     
     
         3 . The method of  claim 2 , wherein updating the surfel representation based on the one or more surfels comprises filtering the surfel representation to remove or hide one or more surfels having a particular classification. 
     
     
         4 . The method of  claim 2 , wherein updating the surfel representation based on the one or more surfels comprises filtering the surfel representation to remove or hide all surfels in the representation that have a particular classification. 
     
     
         5 . The method of  claim 2 , wherein updating the surfel representation based on the one or more surfels comprises filtering the surfel representation to remove or hide all surfels in the surfel representation that have been associated with a particular object of the real-world environment. 
     
     
         6 . The method of  claim 2 , wherein updating the surfel representation based on the one or more surfels comprises filtering the surfel representation to remove or hide the one or more surfels. 
     
     
         7 . The method of  claim 2 , wherein updating the surfel representation based on the one or more surfels comprises filtering the surfel representation to remove or hide the one or more surfels in the surfel representation that have been identified as untrustworthy. 
     
     
         8 . The method of  claim 2 , wherein updating the surfel representation based on the one or more surfels comprises attaching a tag to surfels in the surfel representation other than the one or more surfels, wherein the tag indicates that the corresponding surfels are sufficiently reliable. 
     
     
         9 . The method of  claim 2 , wherein updating the surfel representation based on the one or more surfels comprises attaching a tag to the one or more surfels, wherein the tag indicates that the corresponding surfels are not sufficiently reliable. 
     
     
         10 . The method of  claim 1 , wherein selecting the one or more surfels of the surfel representation comprises selecting all surfels in the surfel representation that are associated with vegetation. 
     
     
         11 . The method of  claim 1 , wherein the one or more matching features comprise one or more of the following types of objects:
 buildings;   roadways;   road signs; or   road barriers.   
     
     
         12 . The method of  claim 1 , wherein:
 receiving input sensor data from the sensors installed on the first vehicle comprises receiving laser data from a lidar system, and   the one or more sensors installed on the first vehicle include the lidar system.   
     
     
         13 . The method of  claim 12 , wherein:
 comparing the input sensor data to the surfel representation to identify the one or more differences between the observation and the surfel representation comprises:
 estimating three-dimensional point data from the laser data, wherein the observation is the three-dimensional point data; and 
 performing dense image matching using the three-dimensional point data to identify point matches between the observation and the surfel representation, 
   the one or more differences between the observation and the surfel representation include one or more distances between the matching points in the point data and corresponding surfels in the surfel representation, and   calibrating the at least one sensor comprises calibrating the lidar system using the one or more distances.   
     
     
         14 . The method of  claim 1 , wherein:
 receiving input sensor data from the one or more sensors installed on the autonomous or semi-autonomous vehicle comprises receiving image data from one or more cameras, and   the one or more sensors installed on the autonomous or semi-autonomous vehicle include the one or more cameras.   
     
     
         15 . The method of  claim 14 , wherein:
 comparing the input sensor data to the surfel representation to identify the one or more differences between the observation and the surfel representation comprises performing feature matching using the image data and the surfel representation to identify one or more features in the image data that match corresponding one or more features in the surfel representation,   the one or more features in the surfel representation correspond to one or more groupings of surfels in the surfel representation,   the one or more differences between the observation and the three-dimensional representation include one or more distances between the one or more features in the image data and corresponding one or more groupings of surfels in the surfel representation, and   calibrating the at least one sensor comprises calibrating the one or more cameras using the one or more distances.   
     
     
         16 . The method of  claim 15 , wherein calibrating the one or more cameras comprises adjusting one or more of the following:
 an offset of a camera of the one or more cameras;   a rotation a camera of the one or more cameras;   radial distortion parameters of a lens of a camera of the one or more cameras; or   a focal length of a lens of a camera of the one or more cameras.   
     
     
         17 . The method of  claim 1 , comprising determining a type of environment of the real-world environment using the surfel representation,
 wherein calibrating the at least one sensor comprises determining that calibration is viable based on the type of environment.   
     
     
         18 . The method of  claim 17 , wherein determining the type of environment of the real-world environment using the surfel representation comprises:
 analyzing the surfel representation or a portion of the surfel representation to identify classifications of surfels in the surfel representation or the portion of the surfel representation; and   based on percentages of the classifications of the surfels in the surfel representation or the portion of the surfel representation, determining a type of environment from multiple types of environments.   
     
     
         19 . A system comprising:
 one or more computers; and   one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
 obtaining a three-dimensional surfel representation of a real-world environment generated from initial sensor data captured by a plurality of vehicles traveling in the real-world environment, the surfel representation comprising a plurality of surfels, 
 receiving input sensor data from one or more sensors installed on a first vehicle, the input sensor data indicating an observation of the real-world environment; 
 determining, from the input sensor data, one or more matching features in the sensor data that match corresponding one or more features in the surfel representation; 
 computing, based on the one or more matching features, one or more differences between the sensor data and the surfel representation; and 
 calibrating at least one sensor of the one or more sensors using the one or more differences between the sensor data and the surfel representation. 
   
     
     
         20 . A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform operations comprising:
 obtaining a three-dimensional surfel representation of a real-world environment generated from initial sensor data captured by a plurality of vehicles traveling in the real-world environment, the surfel representation comprising a plurality of surfels,   receiving input sensor data from one or more sensors installed on a first vehicle, the input sensor data indicating an observation of the real-world environment;   determining, from the input sensor data, one or more matching features in the surfel representation;   computing, based on the one or more matching features, one or more differences between the sensor data and the surfel representation; and   calibrating at least one sensor of the one or more sensors using the one or more differences between the sensor data and the surfel representation.