Lidar point selection using image segmentation

Abstract

The subject disclosure relates to techniques for selecting points of an image for processing with LiDAR data. A process of the disclosed technology can include steps for receiving an image comprising a first image object and a second image object, processing the image to place a bounding box around the first image object and the second image object, and processing an image area within the bounding box to identify a first image mask corresponding with a first pixel region of the first image object and a second image mask corresponding with a second pixel region of the second image object. Systems and machine-readable media are also provided.

What is claimed is: 
     
         1 . A computer-implemented method, comprising:
 receiving, from a first data set recorded by one or more cameras, an image comprising one or more pixels making up a first image object and a second image object, wherein the first image object is partially occluded by the second image object;   processing the image to place a bounding box around the first image object;   processing the one or more pixels to determine a depth of the first image object and a depth of the second image object based on LiDAR data; and   classifying the first image object and the second image object.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the classifying the first image object and the second image object further comprises:
 associating the first image object with a first semantic label.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the first semantic label being used in 3-dimensional transformation of the image for controlling operation of an Autonomous Vehicle (AV). 
     
     
         4 . The computer-implemented method of  claim 1 , further comprising:
 processing the one or more pixels within the bounding box to identify a first image mask corresponding to the first image object, and to identify a second image mask corresponding to the second image object.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein processing the image to place the bounding box around the image object is performed using a first machine-learning model. 
     
     
         6 . The computer-implemented method of  claim 4 , wherein processing the one or more pixels within the bounding box to identify the first image mask is performed using a second machine-learning model. 
     
     
         7 . The computer-implemented method of  claim 1 , further comprising:
 associating a range with the image object based on LiDAR data.   
     
     
         8 . A system, comprising:
 one or more processors; and   a computer-readable medium comprising instructions stored therein, which when executed by the processors, cause the processors to:
 receive, from a first data set recorded by one or more LiDAR cameras outputting LiDAR data, an image comprising a first image object and a second image object; 
 process the image to place a bounding box around the first image object and the second image object; 
 process the image in the bounding box to determine a depth of the first image and a depth the second image object based on the LiDAR data; 
 processing one or more pixels to determine a first classification label for the first image object; and 
 processing the one or more pixels to determine a second classification label for the second image object. 
   
     
     
         9 . The system of  claim 8 , wherein the one or more processors are configured to execute the computer-readable instructions to:
 process the image within the bounding box to identify a first image mask corresponding with a first pixel region making up the first image object and a second image mask corresponding with a second pixel region making up the second image object; and   identify one or more first pixels in the first image object and the second image object using the bounding box, the first image mask, and the second image mask.   
     
     
         10 . The system of  claim 8 , wherein the one or more processors are configured to execute the computer-readable instructions to:
 associate the first image object with the first classification label, wherein the first classification label is a semantic label.   
     
     
         11 . The system of  claim 10 , wherein the first classification label being used in 3-dimensional transformation of the image for controlling operation of an Autonomous Vehicle (AV). 
     
     
         12 . The system of  claim 8 , wherein the one or more processors are configured to execute the computer-readable instructions to process the image to place the bounding box around the first image object and the second image object is performed using a first machine-learning model. 
     
     
         13 . The system of  claim 9 , wherein the one or more processors are configured to execute the computer-readable instructions to process the image within the bounding box to identify the first image mask is performed using a second machine-learning model. 
     
     
         14 . The system of  claim 8 , wherein the one or more processors are configured to execute the computer-readable instructions to:
 associate a first range with the first image object and a second range with the second image object based on the LiDAR data.   
     
     
         15 . A non-transitory computer-readable medium comprising instructions stored therein, which when executed by one or more processors, causes the one or more processors to:
 receive, from a first data set recorded by one or more LiDAR cameras outputting LiDAR data, an image comprising a first image object and a second image object;   process the image to place a bounding box around the first image object and the second image object;   process the image in the bounding box to determine a depth of the first image and a depth the second image object based on the LiDAR data;   processing one or more pixels to determine a first classification label for the first image object; and   processing the one or more pixels to determine a second classification label for the second image object.   
     
     
         16 . The non-transitory computer-readable medium of  claim 15 , wherein the one or more processors are configured to execute the computer-readable instructions to:
 process the image within the bounding box to identify a first image mask corresponding with a first pixel region making up the first image object and a second image mask corresponding with a second pixel region making up the second image object; and   identify one or more first pixels in the first image object and the second image object using the bounding box, the first image mask, and the second image mask.   
     
     
         17 . The non-transitory computer-readable medium of  claim 15 , wherein the one or more processors are configured to execute the computer-readable instructions to:
 associate the first image object with the first classification label, wherein the first classification label is a semantic label.   
     
     
         18 . The non-transitory computer-readable medium of  claim 17 , wherein the first classification label being used in 3-dimensional transformation of the image for controlling operation of an Autonomous Vehicle (AV). 
     
     
         19 . The non-transitory computer-readable medium of  claim 15 , wherein the one or more processors are configured to execute the computer-readable instructions to process the image to place the bounding box around the first image object and the second image object is performed using a first machine-learning model. 
     
     
         20 . The non-transitory computer-readable medium of  claim 16 , wherein the one or more processors are configured to execute the computer-readable instructions to process the image within the bounding box to identify the first image mask is performed using a second machine-learning model.