Method and electronic device for performing 3D point cloud object detection using neural network

Abstract

A method for performing a 3D point cloud object detection is provided. The method includes: receiving an object data comprising a point cloud of a plurality of sensed points; performing a grid-based key-points extraction on raw data points of the point cloud, so as to obtain one or more key-points; performing a hierarchical point-wise convolution on the raw data points of the point cloud according to the key-points, so as to obtain a plurality of target point-wise features; performing a location-aware 3D RoI pooling operation according to the target point-wise features, so as to obtain pooled point-wise features and pooled 3D voxels corresponding to the pooled point-wise features; performing 3D convolution operations on the pooled 3D voxels, so as to obtain represent data points, wherein each of the represent data points includes a represent feature; and inputting the represent data points into multi-layer perceptrons to obtain final predictions.

What is claimed is: 
     
       1. A computer-implemented method for performing a 3D point cloud object detection using grid-based key-points extraction and hierarchical point-wise voxelization by an electronic device, comprising:
 receiving, by a processor of the electronic device, an object data comprising a point cloud of a plurality of sensed points, wherein the point cloud comprises a plurality of raw data points respectively corresponding to the sensed points, and each of the raw data points comprises a point coordinate of corresponding sensed point and a corresponding point-wise feature with one or more channels; 
 performing, by the processor, a grid-based key-points extraction on the raw data points of the point cloud, so as to obtain a key-point set having one or more key-points among the raw data points; 
 performing, by the processor, a hierarchical point-wise convolution on the raw data points of the point cloud according to the key-point set by a neural network having one or more successive convolution blocks, so as to obtain a plurality of target point-wise features corresponding to a plurality of target data points; 
 performing, by the processor, a location-aware 3D RoI (Region of interest) pooling operation according to the target point-wise features, so as to obtain pooled point-wise features and pooled 3D voxels corresponding to the pooled point-wise features; 
 performing, by the processor, 3D convolution operations on the pooled 3D voxels, so as to obtain represent data points, wherein each of the represent data points includes a represent feature; and 
 inputting, by the processor, the represent data points into multi-layer perceptrons, so as to obtain final predictions, wherein the final predications is outputted as one or more 3D object detected from the point cloud. 
 
     
     
       2. The method of  claim 1 , wherein the step of performing the grid-based key-points extraction on the raw data points of the point cloud comprising:
 dividing the entire point cloud into regular grids with a predetermined grid length, wherein the predetermined grid length represents a down-sampling resolution; 
 calculating a grid index of each sensed points in the regular grids, wherein the grid index of one sensed point indicates which grid the said one sensed point belongs to; and 
 for each occupied grid, selecting one represent sensed point in each occupied grid as the key-point of the occupied grid, so as to obtain the key-point set in which each of the key-points has unique grid index. 
 
     
     
       3. The method of  claim 1 , wherein the step of performing the hierarchical point-wise convolution on the raw data points of the point cloud according to the key-point set comprising:
 querying one or more neighbouring points of each key-point according to a kernel radius and a kernel resolution; 
 converting the neighbouring points of each key-point into a set of 3D voxels, wherein a set of the 3D voxels of one key-point includes geometry information (v i ) which records local point cloud geometry around the said one key-point; and 
 performing one or more convolution operations on the sets of 3D voxels corresponding to the key-points by the successive convolution blocks, so as to obtain the target point-wise features. 
 
     
     
       4. The method of  claim 3 , wherein the step of querying the neighbouring points of each key-point according to the kernel radius and the kernel resolution comprising:
 dividing the entire point cloud into further regular grids with a resolution 
 
       
         
           
             
               
                 
                   2 
                   ⁢ 
                   R 
                 
                 k 
               
               , 
             
           
         
       
       wherein R is the kernel radius, and k is the kernel resolution; and
 performing a convolution by setting kernel window on different key-points to obtain different sets of the neighbouring points corresponding to the key-points, wherein a set of the neighbouring points of one key-point are included by a kernel window of which a center is that said one key-point. 
 
     
     
       5. The method of  claim 3 , wherein the step of converting the neighbouring points of each key-point into the set of 3D voxels comprising:
 for the neighbouring points of each key-point, fitting the neighbouring points into k×k×k 3D voxels, 
 wherein an average-pooling strategy is used for calculating average value of the neighbouring points occupying one set of 3D voxels. 
 
     
     
       6. The method of  claim 1 , after obtaining the target point-wise features, the method further comprises:
 inputting the target point-wise features into 1-D convolutional layers to obtain RoI proposals and binary foreground classification scores; and 
 identifying foreground data points corresponding to proposal regions according to the RoI proposals and the binary foreground classification scores. 
 
     
     
       7. The method of  claim 6 , wherein the step of performing the location-aware 3D RoI pooling operation according to the target point-wise features comprising:
 for one set of foreground data point corresponding to one proposal region, 
 dividing the set of foreground data points into pooling grids, wherein a pooling resolution of the pooling grids is 2×2; and 
 identifying one or more interior foreground data points of each occupied pooling grid; and 
 for each occupied pooling grid,
 calculating a pooling weight of each of the interior foreground data points; and 
 calculating a pooled point-wise feature of the occupied pooling grid according to calculated pooling weights and target point-wise features of the interior foreground data points. 
 
 
     
     
       8. The method of  claim 7 , wherein the pooling weight w i  of each of the interior foreground data points is calculated by formula below: 
       
         
           
             
               
                 w 
                 i 
               
               = 
               
                 e 
                 
                   1 
                   - 
                   
                     
                       d 
                       i 
                     
                     r 
                   
                 
               
             
           
         
         and the pooled point-wise feature f pg  of the occupied pooling grid is calculated by formula below: 
       
       
         
           
             
               
                 f 
                 
                   p 
                   ⁢ 
                   ℊ 
                 
               
               = 
               
                 
                   1 
                   n 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     n 
                   
                     
                   
                     
                       w 
                       i 
                     
                     ⁢ 
                     
                       f 
                       i 
                     
                   
                 
               
             
           
         
         where d i  is the distance between the interior data point and the grid center; r is the maximum dimension of pooling grid among three orientations; f i  is the target point-wise feature; n is the total number of interior data points in each occupied pooling grid. 
       
     
     
       9. An electronic device for performing a 3D point cloud object detection using grid-based key-points extraction and hierarchical point-wise voxel i zati on, comprising:
 a processor, configured to execute machine instructions to implement a computer-implemented method, the method comprising: 
 receiving, by the processor, an object data comprising a point cloud of a plurality of sensed points, wherein the point cloud comprises a plurality of raw data points respectively corresponding to the sensed points, and each of the raw data points comprises a point coordinate of corresponding sensed point and a corresponding point-wise feature with one or more channels; 
 performing, by the processor, a grid-based key-points extraction on the raw data points of the point cloud, so as to obtain a key-point set having one or more key-points among the raw data points; 
 performing, by the processor, a hierarchical point-wise convolution on the raw data points of the point cloud according to the key-point set by a neural network having one or more successive convolution blocks, so as to obtain a plurality of target point-wise features corresponding to a plurality of target data points; 
 performing, by the processor, a location-aware 3D RoI pooling operation according to the target point-wise features, so as to obtain pooled point-wise features and pooled 3D voxels corresponding to the pooled point-wise features; 
 performing, by the processor, 3D convolution operations on the pooled 3D voxels, so as to obtain represent data points, wherein each of the represent data points includes a represent feature; and 
 inputting, by the processor, the represent data points into multi-layer perceptrons, so as to obtain final predictions, wherein the final predications is outputted as one or more 3D object detected from the point cloud.