Map creation and localization for autonomous driving applications

Abstract

An end-to-end system for data generation, map creation using the generated data, and localization to the created map is disclosed. Mapstreams—or streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory data—corresponding to any number of drives by any number of vehicles may be generated and uploaded to the cloud. The mapstreams may be used to generate map data—and ultimately a fused high definition (HD) map—that represents data generated over a plurality of drives. When localizing to the fused HD map, individual localization results may be generated based on comparisons of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process may be repeated for any number of sensor modalities and the results may be fused together to determine a final fused localization result.

What is claimed is: 
     
         1 . A method comprising:
 receiving first data associated with a first sensor modality and second data associated with a second sensor modality;   generating, based at least on the first data, a first map associated with the first sensor modality;   generating, based at least on the second data, a second map associated with the second sensor modality; and   sending at least one of the first map or the second map to a machine to cause the machine to navigate within an environment.   
     
     
         2 . The method of  claim 1 , further comprising:
 generating, based at least on the first map and the second map, a third map associated with the environment,   wherein the sending the at least one of the first map or the second map to the machine comprises sending the third map to the machine to cause the machine to navigate within the environment.   
     
     
         3 . The method of  claim 2 , wherein:
 the third map comprises multiple layers;   a first layer of the multiple layers of the third map is generated using the first map, the first layer being associated with the first sensor modality; and   a second layer of the multiple layers of the third map is generated using the second map, the second layer being associated with the second sensor modality.   
     
     
         4 . The method of  claim 2 , wherein the generating the third map associated with the environment comprises:
 determining, based at least on first location information associated with the first map and second location information associated with the second map, that at least a portion of the first map overlaps with at least a portion of the second map; and   generating the third map based at least on the at least the portion of the first map and the at least the portion of the second map.   
     
     
         5 . The method of  claim 2 , wherein the generating the third map associated with the environment comprises:
 determining that the first map represents one or more landmarks also represented by the second map; and   generating the third map to represent at least the one or more landmarks.   
     
     
         6 . The method of  claim 1 , wherein:
 the first data represents one or more first mapstreams associated with the first sensor modality; and   the second data represents one or more second mapstreams associated with the second sensor modality.   
     
     
         7 . The method of  claim 6 , wherein:
 the generating the first map comprises converting the one or more first mapstreams from a first format associated with the first sensor modality to a second format associated with maps; and   the generating the second map comprises converting the one or more second mapstreams from a third format associated with the second sensor modality to the second format associated with the maps.   
     
     
         8 . The method of  claim 1 , wherein:
 the first data comprises first sensor data generated using one or more first sensors associated with the first sensor modality; and   the second data comprises second sensor data generated using one or more second sensors associated with the second sensor modality.   
     
     
         9 . The method of  claim 1 , wherein:
 the first map comprises a first top-down map representing one or more first elevation values; and   the second map comprises a second top-down map representing one or more second elevation values.   
     
     
         10 . A system comprising:
 one or more processing units to:
 generate, based at least on first data associated with a first sensor modality, a first map associated with the first sensor modality; 
 generate, based at least on second data associated with a second sensor modality, a second map associated with the second sensor modality; and 
 send at least one of the first map or the second map to a machine to cause the machine to navigate within an environment. 
   
     
     
         11 . The system of  claim 10 , wherein the one or more processing units are further to:
 generate, based at least on the first map and the second map, a third map associated with the environment,   wherein the third map is sent to the machine and causes the machine to navigate based at least on the third map.   
     
     
         12 . The system of  claim 11 , wherein:
 the third map comprises multiple layers;   a first layer of the multiple layers of the third map is generated using the first map, the first layer being associated with the first sensor modality; and   a second layer of the multiple layers of the third map is generated using the second map, the second layer being associated with the second sensor modality.   
     
     
         13 . The system of  claim 11 , wherein the generation of the third map associated with the environment comprises:
 determining, based at least on first location information associated with the first map and second location information associated with the second map, that at least a portion of the first map overlaps with at least a portion of the second map; and   generating the third map based at least on the at least the portion of the first map and the at least the portion of the second map.   
     
     
         14 . The system of  claim 11 , wherein the generation of the third map associated with the environment comprises:
 determining that the first map represents one or more landmarks also represented by the second map; and   generating the third map to represent at least the one or more landmarks.   
     
     
         15 . The system of  claim 10 , wherein:
 the first data represents one or more first mapstreams associated with the first sensor modality; and   the second data represents one or more second mapstreams associated with the second sensor modality.   
     
     
         16 . The system of  claim 10 , wherein:
 the first data comprises first sensor data generated using one or more first sensors associated with the first sensor modality; and   the second data comprises second sensor data generated using one or more second sensors associated with the second sensor modality.   
     
     
         17 . The system of  claim 10 , wherein the system is located in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing deep learning operations;   a system implemented using an edge device;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         18 . A system comprising:
 one or more processing units to cause performance of one or more localization operations of a machine based at least on a multi-layer map, the multi-layer map including a first map layer associated with a first sensor modality and a second map layer associated with a second sensor modality different from the first sensor modality, wherein the one or more localization operations are performed based at least on a first localization output determined based at least on a first comparison of the first map layer to first sensor data of the first sensor modality and a second localization output determined based at least on a second comparison of the second map layer to second sensor data of the second sensor modality.   
     
     
         19 . The system of  claim 18 , wherein a cloud-hosted version of the multi-layer map includes a third map layer associated with a third sensor modality different from both the first sensor modality and the second sensor modality, and the multi-layer map used to perform the one or more localization operations of the machine does not include the third map layer based at least on the machine not including one or more sensors of the third sensor modality. 
     
     
         20 . The system of  claim 18 , wherein the system is located in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing deep learning operations;   a system implemented using an edge device;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.