Map creation and localization for autonomous driving applications

Abstract

An end-to-end system for data generation, map creation using the generated data, and localization to the created map is disclosed. Mapstreams—or streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory data—corresponding to any number of drives by any number of vehicles may be generated and uploaded to the cloud. The mapstreams may be used to generate map data—and ultimately a fused high definition (HD) map—that represents data generated over a plurality of drives. When localizing to the fused HD map, individual localization results may be generated based on comparisons of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process may be repeated for any number of sensor modalities and the results may be fused together to determine a final fused localization result.

What is claimed is: 
     
         1 . A method comprising:
 obtaining, while a machine is navigating along a path of an environment, a first type of sensor data using a first type of sensor of the machine and a second type of sensor data using a second type of sensor of the machine;   determining, based at least on the first type of sensor data, one or more first locations of one or more first landmarks;   determining, based at least on the second type of sensor data, one or more second locations of one or more second landmarks; and   sending, to a remote system, first data associated with the one or more first locations of the one or more first landmarks and second data associated with the one or more second locations of the one or more second landmarks.   
     
     
         2 . The method of  claim 1 , wherein at least one of:
 a first landmark of the one or more first landmarks is a same landmark as a second landmark of the one or more second landmarks; or   a third landmark of the one or more first landmarks is a different landmark as compared to a fourth landmark of the one or more second landmarks.   
     
     
         3 . The method of  claim 1 , further comprising:
 obtaining the first data based at least on encoding the one or more first locations of the one or more first landmarks; and   obtaining the second data based at least on encoding the one or more second locations of the one or more second landmarks.   
     
     
         4 . The method of  claim 1 , further comprising:
 determining, based at least on the one or more first locations of the one or more first landmarks, one or more first three-dimensional (3D) locations of the one or more first landmarks; and   determining, based at least on the one or more second locations of the one or more second landmarks, one or more second 3D locations of the one or more second landmarks,   wherein the first data is representative of the one or more first 3D locations of the one or more first landmarks and the second data is representative of the one or more second 3D locations of the one or more second landmarks.   
     
     
         5 . The method of  claim 1 , further comprising:
 determining at least a first location associated with the machine when obtaining the first type of sensor data and a second location associated with the machine when obtaining the second type of sensor data; and   sending, to the remote system, third data representative of the first location associated with the machine and fourth data representative of the second location associated with the machine.   
     
     
         6 . The method of  claim 5 , further comprising:
 determining, based at least on motion of the machine, the second location associated with the machine as defined relative to the first location associated with the machine,   wherein the fourth data is representative of the second location associated with the machine as defined relative to the first location associated with the machine.   
     
     
         7 . The method of  claim 1 , wherein the first data is used by the remote system to generate, at least in part, a first layer of a map using the one or more first locations of the one or more first landmarks and the second data is used by the remote system to generate, at least in part, a second layer of the map using the one or more second locations of the one or more second landmarks. 
     
     
         8 . The method of  claim 1 , further comprising causing a second machine to navigate within at least a portion of the environment based at least on a map, the map being generated based at least on the first data and the second data. 
     
     
         9 . The method of  claim 1 , further comprising:
 determining, based at least on the first type of sensor data, at least one of one or more poses associated with the one or more first landmarks or one or more geometries associated with the one or more first landmarks,   wherein the first data is further representative of the at least one of the one or more poses associated with the one or more first landmarks or the one or more geometries associated with the one or more first landmarks.   
     
     
         10 . The method of  claim 1 , wherein at least one of the one or more first landmarks or the one or more second landmarks include:
 a lane divider;   a road boundary;   a sign;   a pole;   a wait condition;   a vertical structure;   a road user;   a static object; or   a dynamic object.   
     
     
         11 . A system comprising:
 one or more processing units to:
 determine, based at least on first sensor data obtained using one or more sensors of a machine while located at a first location, one or more first locations of one or more first landmarks; 
 determine, based at least on second sensor data obtained using the one or more sensors of the machine, one or more second locations of one or more second landmarks; 
 determine, based at least on motion of the machine, a second location of the machine that is relative to the first location of the machine; and 
 send, to a remote system, first data representative of the first location of the machine and the one or more first locations of the one or more first landmarks and second data representative of the second location of the machine and the one or more second locations of the one or more second landmarks. 
   
     
     
         12 . The system of  claim 11 , wherein the one or more processing units are further to:
 obtain the first data based at least on encoding the first location of the machine and the one or more first locations of the one or more first landmarks; and   obtain the second data based at least on encoding the second location of the machine and the one or more second locations of the one or more second landmarks.   
     
     
         13 . The system of  claim 11 , wherein the one or more processing units are further to:
 determine, based at least on the one or more first locations of the one or more first landmarks, one or more first three-dimensional (3D) locations of the one or more first landmarks; and   determine, based at least on the one or more second locations of the one or more second landmarks, one or more second 3D locations of the one or more second landmarks,   wherein the first data is representative of the first location of the machine and the one or more first 3D location of the one or more first landmarks and the second data is representative of the second location of the machine and the one or more second 3D locations of the one or more second landmarks.   
     
     
         14 . The system of  claim 11 , wherein the one or more processing units are further to:
 determine, based at least on the first type of sensor data, at least one of one or more poses associated with the one or more first landmarks or one or more geometries associated with the one or more first landmarks,   wherein the first data is further representative of the at least one of the one or more poses associated with the one or more first landmarks or the one or more geometries associated with the one or more first landmarks.   
     
     
         15 . The system of  claim 11 , wherein the one or more processing units are further to:
 determine, based at least on the motion, at least one of a translation or a rotation of the machine relative to the first location,   wherein the second location of the second machine is defined using the at least one of the rotation or the translation with respect to the first location.   
     
     
         16 . The system of  claim 11 , wherein the first data and the second data are used by the remote system to generate a map that stores the one or more first locations of the one or more first landmarks and the one or more second locations of the one or more second landmarks. 
     
     
         17 . The system of  claim 11 , wherein the one or more processing units are further to cause a second machine to navigate within at least a portion of the environment based at least on a map that indicates at least one of the one or more first locations of the one or more first landmarks and the one or more second locations of the one or more second landmarks. 
     
     
         18 . The system of  claim 11 , wherein the system is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing deep learning operations;   a system implemented using an edge device;   a system implemented using a robot;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         19 . A processor comprising:
 one or more processing units to send, to a remote system, first data representative of one or more first locations of one or more first landmarks and one or more second locations of one or more second landmarks, wherein the one or more first locations of the one or more first landmarks are determined using a first type of sensor data obtained using a first sensor modality of a machine and the one or more second locations of the one or more second landmarks are determined using a second type of sensor data obtained using a second sensor modality of the machine.   
     
     
         20 . The processor of  claim 19 , wherein the processor is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing deep learning operations;   a system implemented using an edge device;   a system implemented using a robot;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.