Using lidar objectness to address vehicle slicing in supervised depth estimation

Abstract

Example systems and techniques are described for controlling operation of a vehicle and training a machine learning model for controlling operation of a vehicle. A system includes memory configured to store point cloud data associated with the vehicle and one or more processors communicatively coupled to the memory. The one or more processors are configured to determine a depth map indicative of distance of one or more objects to the vehicle and control operation of a vehicle based on the depth map. The depth map is based on executing a machine learning model, the machine learning model being trained with a slice loss function determined from training point cloud data having a respective depth that is greater than the average depth for a set of points of the point cloud data plus a threshold.

What is claimed is: 
     
         1 . A system for controlling operation of a vehicle comprising:
 memory configured to a depth map indicative of distance of one or more objects to the vehicle; and   one or more processors communicatively coupled to the memory, the one or more processors being configured to:
 determine the depth map; and 
 control operation of a vehicle based on the depth map, 
 wherein the one or more processors are configured to determine the depth map based on executing a machine learning model, the machine learning model having been trained with a slice loss function determined from training point cloud data having a respective depth that is greater than the average depth for a set of points of the training point cloud data plus a threshold. 
   
     
     
         2 . The system of  claim 1 , further comprising a monocular camera configured to sense image data, wherein the one or more processors are configured to determine the depth map based on executing the machine learning model on the image data. 
     
     
         3 . The system of  claim 1 , wherein the machine learning model has been further trained with at least one of a depth gradient loss function, a depth supervision loss function, or a photometric loss function. 
     
     
         4 . A system for training a machine learning model, the system comprising:
 memory configured to store the machine learning model; and   one or more processors communicatively coupled to the memory, the one or more processors being configured to:
 determine a slice loss function from training point cloud data having a respective depth that is greater than the average depth for a set of points of the training point cloud data plus a threshold; and 
 train the machine learning model with the slice loss function. 
   
     
     
         5 . The system of  claim 4 , wherein as part of training the machine learning model, the one or more processors are configured to process image data from a monocular camera. 
     
     
         6 . The system of  claim 4 , wherein the one or more processors are further configured train the machine learning model with at least one of a depth gradient loss function, a depth supervision loss function, or a photometric loss function. 
     
     
         7 . The system of  claim 4 , wherein as part of determining the slice loss function, the one or more processors are configured to:
 generate a two-dimensional (2D) instance label based on one or more LiDAR annotations, the 2D instance label being associated with an instance of an object of a neighboring vehicle;   transform three-dimensional (3D) points of the training point cloud data to 2D points in a frame to generate the instance of the object;   determine boundary points of the instance of the object;   generate a mask based at least in part on the boundary points, the mask defining the set of points, the set of points being associated with the instance of the object; and   determine the slice loss for points associated with the instance of the object having an associated depth that is greater than an average depth for the points associated with the instance of the object plus a threshold.   
     
     
         8 . The system of  claim 7 , wherein the one or more processors are configured to filter points within the instance of the object that are within a predetermined distance of a LiDAR sensor of the vehicle or the vehicle prior to determining the boundary points to generate filtered points, wherein the one or more processors are configured to determine the slice loss based on the filtered points and no other points of the training point cloud data. 
     
     
         9 . The system of  claim 7 , wherein as part of generating the 2D instance label based on one or more LiDAR annotations, the one or more processors are configured to generate a bounding box based on the one or more LiDAR annotations. 
     
     
         10 . The system of  claim 9 , wherein as part of transforming 3D points of the point cloud data to 2D points in the frame, the one or more processors are configured to:
 generate a transformation matrix based on camera intrinsics and extrinsics;   apply the transformation matrix to bounding box coordinates to generate corners of the bounding box; and   determine, for each point in the point cloud if the point is inside the bounding box.   
     
     
         11 . The system of  claim 9 , wherein as part of determining the boundary points for the instance of the object, the one or more processors are configured to apply a hull algorithm to the points inside the bounding box. 
     
     
         12 . The system of  claim 11 , wherein the hull algorithm is a non-convex hull algorithm, and wherein the non-convex hull algorithm uses an alpha value. 
     
     
         13 . The system of  claim 12 , wherein the one or more processors are configured to:
 apply a first predetermined alpha value to generate first boundary points;   determine whether the first boundary points meet a predetermined condition; and   based on the first boundary points not meeting the predetermined condition, apply a second predetermined alpha value to generate second boundary points, wherein the second boundary points comprise the boundary points.   
     
     
         14 . The system of  claim 13 , wherein the first predetermined alpha value comprises a value in the range of 0.1 and 0.2 inclusive. 
     
     
         15 . The system of  claim 14 , wherein the first predetermined alpha value is 0.157. 
     
     
         16 . The system of  claim 13 , wherein the second predetermined alpha value comprises a value less than 0.1. 
     
     
         17 . The system of  claim 16 , wherein the second predetermined alpha value is 0.02. 
     
     
         18 . The system of  claim 9 , wherein the one or more processors are further configured remove one or more ground points from the bounding box, wherein to remove the one or more ground points, the one or more processors are configured to lift a lower boundary of the bounding box by a distance. 
     
     
         19 . The system of  claim 18 , wherein the distance is at least one of 0.2 meters or based on a diameter of a wheel of the neighboring vehicle associated with the bounding box. 
     
     
         20 . A method for controlling operation of a vehicle, the method comprising:
 determining a depth map indicative of the distance to the vehicle; and   controlling operation of a vehicle based on the depth map,   wherein determining the depth map comprises executing a machine learning model, the machine learning model having been trained with a slice loss function determined from training point cloud data having a respective depth that is greater than the average depth for a set of points of the training point cloud data plus a threshold.   
     
     
         21 . The method of  claim 20 , wherein determining the depth map is based on image data from a monocular camera. 
     
     
         22 . The method of  claim 20 , wherein the machine learning model has been further trained with at least one of a depth gradient loss function, a depth supervision loss function, or a photometric loss function. 
     
     
         23 . A method for training a machine learning model, the method comprising:
 determining a slice loss function from training point cloud data having a respective depth that is greater than the average depth for a set of points of the training point cloud data plus a threshold; and   training the machine learning model with the slice loss function.   
     
     
         24 . The method of  claim 23 , wherein training the machine learning model comprises processing image data from a monocular camera. 
     
     
         25 . The method of  claim 23 , further comprising training the machine learning model with at least one of a depth gradient loss function, a depth supervision loss function, or a photometric loss function. 
     
     
         26 . The method of  claim 23 , wherein determining the slice loss function comprises:
 generating a two-dimensional (2D) instance label based on one or more LiDAR annotations, the 2D instance label being associated with an instance of an object of a neighboring vehicle;   transforming three-dimensional (3D) points of the training point cloud data to 2D points in a frame to generate the instance of the object;   determining boundary points of the instance of the object;   generating a mask based at least in part on the boundary points, the mask defining the set of points, the set of points being associated with the instance of the object; and   determining the slice loss for points associated with the instance of the object having an associated depth that is greater than an average depth for the points associated with the instance of the object plus a threshold.   
     
     
         27 . The method of  claim 26 , further comprising filtering points within the instance of the object that are within a predetermined distance of a LiDAR sensor of a vehicle or the vehicle prior to determining the boundary points to generate filtered points, wherein determining the slice loss is based on the filtered points and no other points of the training point cloud data. 
     
     
         28 . The method of  claim 26 , wherein generating the 2D instance label based on one or more LiDAR annotations comprises generating a bounding box based on the one or more LiDAR annotations. 
     
     
         29 . The method of  claim 28 , wherein transforming 3D points of the point cloud data to 2D points in a frame comprises:
 generating a transformation matrix based on camera intrinsics and extrinsics;   applying the transformation matrix to bounding box coordinates of the bounding box to generate corners of the bounding; and   determining, for each point in the point cloud if the point is inside the bounding box.   
     
     
         30 . The method of  claim 26 , wherein determining the boundary points for the instance of the object comprises applying a hull algorithm to the points inside the bounding box.