Annotating high definition map data with semantic labels

Abstract

According to an aspect of an embodiment, a method may include obtaining multiple sets of camera images and light detection and ranging (LIDAR) point clouds along a track within a geographic sector of a map. The method may include applying a learning model to the camera images to characterize objects within the camera images within classes of objects to generate segmented images. The method may additionally include mapping the sets of camera images and the LIDAR point clouds to three dimensional points of the geographic sector of the map. The method may also include projecting the three dimensional points onto the segmented images to obtain corresponding classes for the three dimensional points of the geographic sector of the map.

What is claimed is: 
     
         1 . A computer-implemented method, comprising:
 obtaining a set of sensor data along a track traversing a geographic sector of a map;   applying a learning model to the sensor data to characterize objects within the sensor data to generate segmented sensor data, the objects characterized based on classes of objects; and   projecting the three dimensional points onto the segmented sensor data to obtain corresponding classes for the three dimensional points of the geographic sector of the map.   
     
     
         2 . The method of  claim 1 , wherein the sensor data includes at least one of camera images and light detection and ranging (LIDAR) point clouds. 
     
     
         3 . The method of  claim 2 , wherein applying the learning model to the sensor data includes applying a first learning model to the camera images and a second learning model to the LIDAR point clouds. 
     
     
         4 . The method of  claim 3 , further comprising resolving a conflict between a first classification of a given three dimensional point by the first learning model and a second classification of the given three dimensional point by the second learning model. 
     
     
         5 . The method of  claim 4 , wherein resolving the conflict includes selecting one of the first classification and the second classification instead of the other of the first classification and the second classification based on an accuracy of the first learning model relative to the second learning model in identifying a certain class. 
     
     
         6 . The method of  claim 4 , wherein resolving the conflict includes using the first classification based on the camera image if available and otherwise using the second classification of the LIDAR point clouds. 
     
     
         7 . The method of  claim 1 , further comprising storing the corresponding classes for the three dimensional points of the geographic sector of the map such that the geographic sector of the map, including the corresponding classes, is recallable by a vehicle traversing a physical location corresponding to the geographic sector of the map to facilitate localization of the vehicle. 
     
     
         8 . The method of  claim 7 , further comprising removing classes of dynamic objects from the map prior to storing the corresponding classes for the three dimensional points of the geographic sector of the map, the classes of dynamic objects including at least one of bicycles, vehicles, and pedestrians. 
     
     
         9 . The method of  claim 8 , further comprising storing an indication of a frequency of the dynamic objects occurring at a location within the geographic sector of the map. 
     
     
         10 . One or more non-transitory computer readable media storing instructions that in response to being executed by one or more processors, cause a computer system to perform operations, the operations comprising:
 obtaining a plurality of sets of camera images and light detection and ranging (LIDAR) point clouds along a track within a geographic sector of a map;   applying a learning model to the camera images to characterize objects within the camera images within classes of objects to generate segmented images;   mapping the plurality of sets of camera images and the LIDAR point clouds to three dimensional points of the geographic sector of the map; and   projecting the three dimensional points onto the segmented images to obtain corresponding classes for the three dimensional points of the geographic sector of the map.   
     
     
         11 . The computer readable media of  claim 10 , wherein the operations further comprise combining multiple classes across multiple segmented images for a single three dimensional point to obtain a single class for the single three dimensional point. 
     
     
         12 . The computer readable media of  claim 10 , wherein the mapping the plurality of sets of camera images and the LIDAR point clouds includes selecting representative sets of the plurality of sets that are captured at least a threshold distance apart. 
     
     
         13 . The computer readable media of  claim 10 , wherein the projecting the three dimensional points onto the segmented images to obtain corresponding classes for the three dimensional points of the geographic sector of the map includes, for a given set of a given camera image and a given LIDAR point cloud, ignoring the three dimensional points outside of the given camera image. 
     
     
         14 . The computer readable media of  claim 10 , wherein the classes includes types of objects in an urban environment, the classes including at least one of buildings, roads, sidewalks, fences, poles, traffic signs, vegetation, terrain, bicycles, vehicles, and pedestrians. 
     
     
         15 . The computer readable media of  claim 10 , wherein the classes includes colors or edges such that the object characterization detects colors or edges of objects. 
     
     
         16 . The computer readable media of  claim 10 , wherein the operations further comprise storing the corresponding classes for the three dimensional points of the geographic sector of the map such that the geographic sector of the map, including the corresponding classes, is recallable by a vehicle traversing a physical location corresponding to the geographic sector of the map to facilitate localization of the vehicle. 
     
     
         17 . The computer readable media of  claim 16 , wherein the operations further comprise removing classes of dynamic objects from the map prior to storing the corresponding classes for the three dimensional points of the geographic sector of the map, the classes of dynamic objects including at least one of bicycles, vehicles, and pedestrians. 
     
     
         18 . The computer readable media of  claim 17 , wherein the operations further comprise storing an indication of a frequency of the dynamic objects occurring at a location within the geographic sector of the map. 
     
     
         19 . A computer system comprising:
 one or more processors; and   one or more non-transitory computer readable media storing instructions that in response to being executed by the one or more processors, cause the computer system to perform operations, the operations comprising:
 obtaining a plurality of sets of camera images and light detection and ranging (LIDAR) point clouds along a track within a geographic sector of a map; 
 applying a learning model to the camera images to characterize objects within the camera images within classes of objects to generate segmented images; 
 mapping the plurality of sets of camera images and the LIDAR point clouds to three dimensional points of the geographic sector of the map; and 
 projecting the three dimensional points onto the segmented images to obtain corresponding classes for the three dimensional points of the geographic sector of the map. 
   
     
     
         20 . The computer system of  claim 19 , wherein the operations further comprise combining multiple classes across multiple segmented images for a single three dimensional point to obtain a single class for the single three dimensional point. 
     
     
         21 . The computer system of  claim 19 , wherein the mapping the plurality of sets of camera images and the LIDAR point clouds includes selecting representative sets of the plurality of sets that are captured at least a threshold distance apart. 
     
     
         22 . The computer system of  claim 19 , wherein the projecting the three dimensional points onto the segmented images to obtain corresponding classes for the three dimensional points of the geographic sector of the map includes, for a given set of a given camera image and a given LIDAR point cloud, ignoring the three dimensional points outside of the given camera image. 
     
     
         23 . The computer system of  claim 19 , wherein the classes includes types of objects in an urban environment, the classes including at least one of buildings, roads, sidewalks, fences, poles, traffic signs, vegetation, terrain, bicycles, vehicles, and pedestrians. 
     
     
         24 . The computer system of  claim 19 , wherein the classes includes colors or edges such that the object characterization detects colors or edges of objects. 
     
     
         25 . The computer system of  claim 19 , wherein the operations further comprise storing the corresponding classes for the three dimensional points of the geographic sector of the map such that the geographic sector of the map, including the corresponding classes, is recallable by a vehicle traversing a physical location corresponding to the geographic sector of the map to facilitate localization of the vehicle. 
     
     
         26 . The computer system of  claim 25 , wherein the operations further comprise removing classes of dynamic objects from the map prior to storing the corresponding classes for the three dimensional points of the geographic sector of the map, the classes of dynamic objects including at least one of bicycles, vehicles, and pedestrians. 
     
     
         27 . The computer system of  claim 26 , wherein the operations further comprise storing an indication of a frequency of the dynamic objects occurring at a location within the geographic sector of the map.