3d point cloud-based deep learning neural network acceleration apparatus and method

Abstract

Disclosed is a 3D point cloud-based deep learning neural network acceleration apparatus including a depth image input unit configured to receive a depth image, a depth data storage unit configured to store depth data derived from the depth image, a sampling unit configured to sample the depth image in units of a sampling window having a predetermined first size, a grouping unit configured to generate a grouping window having a predetermined second size and to group inner 3D point data by grouping window, and a convolution computation unit configured to separate point feature data and group feature data, among channel-direction data of 3D point data constituting the depth image, to perform convolution computation with respect to the point feature data and the group feature data, to sum the results of convolution computation by group grouped by the grouping unit, and to derive the final result.

What is claimed is: 
     
         1 . A 3D point cloud-based deep learning neural network acceleration apparatus comprising:
 a depth image input unit configured to receive a depth image, which is a training target;   a depth data storage unit configured to store depth data derived from the depth image;   a sampling unit configured to sample the depth image in units of a sampling window having a predetermined first size;   a grouping unit configured to generate a grouping window having a predetermined second size based on a result of sampling and to group inner 3D point data by grouping window; and   a convolution computation unit configured to separate point feature data, which are not affected by a group value, and group feature data, which are affected by the group value, among channel-direction data of 3D point data constituting the depth image, to perform convolution computation with respect to the point feature data and the group feature data, to sum a result of convolution computation with respect to the point feature data and a result of convolution computation with respect to the group feature data by group grouped by the grouping unit, and to derive a final result.   
     
     
         2 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 1 , wherein the sampling unit comprises:
 a sampling window generation unit configured to generate the sample window;   a first 3D point data conversion unit configured to load depth data stored in the depth data storage unit in units of the sampling window and to convert the depth data into 3D point data;   a first distance calculation unit configured to calculate a distance between each of all 3D point data included in the sampling window and arbitrary sampling reference 3D point data;   a sample point data determination unit configured to determine 3D point data having a distance from the reference 3D point data greater than a predetermined first distance critical value based on a result of distance calculation by the distance calculation unit to be representative 3D point data; and   a sampling controller configured to control operation of the sampling window generation unit, the first 3D point data conversion unit, the first distance calculation unit, and the sample point data determination unit according to a predetermined sampling algorithm.   
     
     
         3 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 2 , wherein the sampling window generation unit generates the sampling window such that the sampling window is disposed on the depth image at predetermined intervals while covering an entirety of the depth image and generates sampling windows equal in number to predetermined 3D point data in order to accelerate a 3D point cloud-based deep learning neural network. 
     
     
         4 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 2 , wherein the grouping unit comprises:
 a grouping window generation unit configured to generate the grouping window based on the representative 3D point data;   a second 3D point data conversion unit configured to load depth data constituting the depth image in units of the grouping window and to convert the depth data into 3D point data;   a second distance calculation unit configured to calculate a distance between each of all 3D point data included in the grouping window and the representative 3D point data;   a group determination unit configured to determine two or more 3D point data having a distance from the representative 3D point data less than a predetermined second distance critical value based on a result of distance calculation by the second distance calculation unit to be one group; and   a grouping controller configured to control operation of the grouping window generation unit, the second 3D point data conversion unit, the second distance calculation unit, and the group determination unit according to a predetermined grouping algorithm.   
     
     
         5 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 4 , wherein
 the sampling window generation unit and the grouping window generation unit are implemented by one window generator configured to generate a window based on input window generation information, and   the window generator generates the sampling window based on first setting information input from the sampling controller in order to generate the sampling window or generates the grouping window based on second setting information input from the grouping controller in order to generate the grouping window.   
     
     
         6 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 4 , wherein
 the first 3D point data conversion unit and the second 3D point data conversion unit are implemented by one data converter configured to convert input depth data into 3D point data, and   the data converter converts depth data loaded in units of the sampling window into 3D points under control of the sampling controller or converts depth data loaded in units of the grouping window into 3D points under control of the grouping controller.   
     
     
         7 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 4 , wherein
 the first distance calculation unit and the second distance calculation unit are implemented by one distance calculator configured to calculate a distance between input 3D points, and   the distance calculator calculates a distance between each of all 3D point data included in the sampling window and arbitrary sampling reference 3D point data under control of the sampling controller or calculates a distance between each of all 3D point data included in the grouping window and the representative 3D point data under control of the grouping controller.   
     
     
         8 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 4 , further comprising:
 an output buffer configured to temporarily store the result of sampling and the result of grouping, wherein   the output buffer stores indexes and coordinate values of the representative 3D point data in a separated state under control of the sampling controller or stores indexes and coordinate values of 3D point data included in the grouped group in a separated state under control of the grouping controller.   
     
     
         9 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 1 , wherein the convolution computation unit slices input data and weight, which are convolution computation targets, to upper bits and lower bits, respectively, performs convolution computation with respect to the sliced input data in chronological order, accumulates results of convolution computation, and produces a final result. 
     
     
         10 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 9 , wherein
 the convolution computation unit performs computation in a state in which L sliced input data are set to one computation unit, and   when the computation unit data are all 0, weight computation with respect to the computation unit data is skipped.   
     
     
         11 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 10 , wherein the convolution computation unit performs computation with respect to the upper bit data, among the sliced input data, first, predicts a position of a max-pooling maximum value based on a result value thereof, and skips computation with respect to the lower bit data at different positions from the position of the max-pooling maximum value. 
     
     
         12 . The 3D point cloud-based deep learning neural network acceleration apparatus according to  claim 1 , wherein the convolution computation unit is implemented in a pipeline structure such that convolution computation with respect to the point feature data and convolution computation with respect to the group feature data are performed by parallel processing. 
     
     
         13 . A 3D point cloud-based deep learning neural network acceleration method comprising:
 a sampling step of sampling a depth image, which is a training target, in units of a sampling window having a predetermined first size;   a grouping step of generating a grouping window having a predetermined second size based on a result of sampling and grouping inner 3D point data by grouping window;   a first convolution computation step of separating only point feature data, which are not affected by a group value, from channel-direction data of all 3D point data included in the depth image, performing convolution computation with respect to the point feature data, and storing a result of convolution computation;   a second convolution computation step of separating only group feature data, which are affected by the group value, from channel-direction data of 3D point data constituting each group by grouped group, performing convolution computation with respect to the group feature data, and storing a result of convolution computation; and   a final computation step of summing a result of computation in the first convolution computation step and a result of computation in the second convolution computation step with respect to each of the 3D point data constituting each group by group.   
     
     
         14 . The 3D point cloud-based deep learning neural network acceleration method according to  claim 13 , wherein the sampling step comprises:
 a first loading step of loading depth data constituting the depth image in units of the sampling window;   a first conversion step of converting the loaded depth data into 3D point data;   a first distance calculation step of calculating a distance between each of all 3D point data included in the sampling window and arbitrary sampling reference 3D point data; and   a first determination step of determining 3D point data having a distance from the reference 3D point data greater than a predetermined first distance critical value based on a result of distance calculation in the first distance calculation step to be representative 3D point data.   
     
     
         15 . The 3D point cloud-based deep learning neural network acceleration method according to  claim 14 , wherein the sampling step further comprises a sampling window generation step of generating the sampling window such that the sampling window is disposed on the depth image at predetermined intervals while covering an entirety of the depth image and generating sampling windows equal in number to predetermined 3D point data in order to accelerate a 3D point cloud-based deep learning neural network. 
     
     
         16 . The 3D point cloud-based deep learning neural network acceleration method according to  claim 14 , wherein the grouping step comprises:
 a grouping window generation step of generating a grouping window based on the representative 3D point data determined by sampling window;   a second loading step of loading depth data constituting the depth image in units of the grouping window;   a second conversion step of converting the loaded depth data into 3D point data;   a second distance calculation step of calculating a distance between each of all 3D point data included in the grouping window and the representative 3D point data, which are a center of the grouping window; and   a second determination step of determining two or more 3D point data having a distance from the representative 3D point data less than a predetermined second distance critical value based on a result of distance calculation in the second distance calculation step to be one group.   
     
     
         17 . The 3D point cloud-based deep learning neural network acceleration method according to  claim 13 , wherein each of the first and second convolution computation steps slices input data and weight, which are convolution computation targets, to upper bits and lower bits, respectively, performs convolution computation with respect to the sliced input data in chronological order, accumulates results of convolution computation, and produces a final result. 
     
     
         18 . The 3D point cloud-based deep learning neural network acceleration method according to  claim 17 , wherein
 each of the first and second convolution computation steps performs computation in a state in which L sliced input data are set to one computation unit, and   each of the first and second convolution computation steps comprises a sparse input skipping step of, when the computation unit data are all 0, skipping weight computation with respect to the computation unit data.   
     
     
         19 . The 3D point cloud-based deep learning neural network acceleration method according to  claim 18 , wherein each of the first and second convolution computation steps further comprises:
 a maximum value position prediction step of performing computation with respect to the upper bit data, among the sliced input data, first, and predicting a position of a max-pooling maximum value based on a result value thereof; and   an output computation skipping step of skipping computation with respect to the lower bit data at different positions from the position of the max-pooling maximum value.   
     
     
         20 . The 3D point cloud-based deep learning neural network acceleration method according to  claim 13 , wherein the first convolution computation step and the second convolution computation step temporarily overlap so as to be performed by parallel processing.