Data integration from multiple sensors

Abstract

Disclosed are methods and devices related to autonomous driving. In one aspect, a method is disclosed. The method includes determining three-dimensional bounding indicators for one or more first objects in road target information captured by a light detection and ranging (LIDAR) sensor; determining camera bounding indicators for one or more second objects in road image information captured by a camera sensor; processing the road image information to generate a camera matrix; determining projected bounding indicators from the camera matrix and the three-dimensional bounding indicators; determining, from the projected bounding indicators and the camera bounding indicators, associations between the one or more first objects and the one or more second objects to generate combined target information; and applying, by the autonomous driving system, the combined target information to produce a vehicle control signal.

1 .- 20 . (canceled) 
     
     
         21 . A system for generating autonomous driving data, comprising:
 a first vehicle that comprises a first light detection and ranging (LIDAR) sensor,   a second vehicle that comprises a second camera sensor, the second vehicle being at a first distance from the first vehicle while the first vehicle and the second vehicle are in motion, and   at least one processor,
 wherein the first LIDAR sensor is configured to capture three-dimensional data of a first moving object positioned between the first vehicle and the second vehicle while the first vehicle and the second vehicle are in motion, 
 wherein the second camera sensor is configured to capture two-dimensional data of the first moving object, and 
 wherein the at least one processor is configured to generate a first set of target data for the first moving object by combining data captured by the first LIDAR sensor of the first vehicle and data captured by the second camera sensor of the second vehicle. 
   
     
     
         22 . The system of  claim 21 , wherein the first vehicle is positioned in front of the second vehicle while the first vehicle and the second vehicle are in motion, and wherein the first LIDAR sensor comprises a rear facing LIDAR sensor. 
     
     
         23 . The system of  claim 21 , wherein the first vehicle further comprises a first camera sensor, the first camera sensor being a rear facing camera sensor, and wherein the second vehicle further comprises a second LIDAR sensor. 
     
     
         24 . The system of  claim 21 , wherein a second moving object is positioned between the first vehicle and the second vehicle while the first vehicle and the second vehicle are in motion, and wherein the second moving object is positioned outside of a range of the first LIDAR sensor, and wherein the at least one processor is configured to:
 generate target data for the second moving object based on data captured by the second camera sensor of the second vehicle only.   
     
     
         25 . The system of  claim 21 , wherein a third moving object is positioned between the first vehicle and the second vehicle while the first vehicle and the second vehicle are in motion, and wherein the third moving object is positioned outside of a range of the second camera sensor, and wherein the at least one processor is configured to:
 generate target data for the third moving object based on data captured by the first LIDAR sensor of the second vehicle only.   
     
     
         26 . The system of  claim 21 , wherein the at least one processor is configured to combine the data captured by the first LIDAR sensor of the first vehicle and the data captured by the second camera sensor of the second vehicle based on:
 determining three-dimensional bounding information of the at least one moving object based on the captured by the first LIDAR sensor of the first vehicle;   determining two-dimensional bounding information of the at least one moving object based on the data captured by the second camera sensor of the second vehicle; and   associating the three-dimensional bounding information with the two-dimensional bounding information.   
     
     
         27 . The system of  claim 26 , wherein the at least one processor is configured to associate the three-dimensional bounding information with the two-dimensional bounding information using one of an intersection over union technique, a dice coefficient, or a generalized intersection over union technique. 
     
     
         28 . The system of  claim 26 , wherein the at least one processor is configured to associate the three-dimensional bounding information with the two-dimensional bounding information based on:
 determining a camera matrix based on the two-dimensional data captured by the second camera sensor of the second vehicle, and   applying the camera matrix to the three-dimensional data captured by the first LIDAR sensor of the first vehicle.   
     
     
         29 . The system of  claim 21 , wherein the first vehicle and the second vehicle are at different distances from each other while the first vehicle and the second vehicle are in motion, and wherein the at least one processor is configured to generate different sets of target data for the first moving object to obtain ground truth data representing the first moving object. 
     
     
         30 . The system of  claim 21 , wherein the first distance between the first vehicle and the second vehicle is greater than a range of the first LIDAR sensor and is also greater than a range of the second camera sensor. 
     
     
         31 . The system of  claim 21 , wherein the at least one processor is configured to generate a vehicle control signal based on the first set of target data. 
     
     
         32 . A method for generating autonomous driving data, comprising:
 operating a first vehicle and a second vehicle at a first distance from each other on a road, wherein the first vehicle comprises a first light detection and ranging (LIDAR) sensor and the second vehicle comprises a second camera sensor, wherein at least one moving object is positioned between the first vehicle and the second vehicle while the first vehicle and the second vehicle are in motion; and   generating a first set of target data for the at least one moving object by combining data captured by the first LIDAR sensor of the first vehicle and data captured by the second camera sensor of the second vehicle.   
     
     
         33 . The method of  claim 32 , further comprising:
 operating the first vehicle and the second vehicle at a second distance from each other on the road;   generating a second set of target data for the least one moving object by combining data captured by the first LIDAR sensor of the first vehicle and data captured by the second camera sensor of the second vehicle.   
     
     
         34 . The method of  claim 33 , further comprising:
 obtaining ground truth data representing the at least one object by combining the first set of target data and the second set of target data.   
     
     
         35 . The method of  claim 32 , the combining of the data captured by the first LIDAR sensor of the first vehicle and the data captured by the second camera sensor of the second vehicle comprises:
 determining three-dimensional bounding information of the at least one moving object based on the captured by the first LIDAR sensor of the first vehicle;   determining two-dimensional bounding information of the at least one moving object based on the data captured by the second camera sensor of the second vehicle; and   associating the three-dimensional bounding information with the two-dimensional bounding information.   
     
     
         36 . The method of  claim 35 , wherein the associating of the three-dimensional bounding information with the two-dimensional bounding information comprises applying one of an intersection over union technique, a dice coefficient, or a generalized intersection over union technique. 
     
     
         37 . The method of  claim 35 , wherein the associating of the three-dimensional bounding information with the two-dimensional bounding information comprises:
 determining a camera matrix based on the two-dimensional data captured by the second camera sensor of the second vehicle, and   applying the camera matrix to the three-dimensional data captured by the first LIDAR sensor of the first vehicle.   
     
     
         38 . The method of  claim 37 , wherein the camera matrix includes extrinsic matrix parameters and intrinsic matrix parameters. 
     
     
         39 . The method of  claim 38 , wherein the intrinsic matrix parameters include one or more relationships between pixel coordinates and camera coordinates, wherein the extrinsic matrix parameters include information about location and orientation of the second camera sensor. 
     
     
         40 . The method of  claim 32 , further comprising:
 generating a vehicle control signal based on the first set of target data.