Method and apparatus with neural network quantization

Abstract

A method for neural network quantization includes performing a forward pass and a backward pass of a first neural network having a first bit precision with respect to each of a plurality of input data sets, obtaining profile information with respect to at least one of input gradients, weight gradients, and output gradients calculated for each layer of layers included in the first neural network in the process of performing the backward pass, determining one or more layers, from among the layers, to be quantized with a second bit precision less than the first bit precision, based on the obtained profile information, and generating a second neural network by quantizing the determined layers from among the layers with the second bit precision.

What is claimed is: 
     
         1 . A method for neural network quantization, comprising:
 performing a forward pass and a backward pass of a first neural network having a first bit precision with respect to each of a plurality of input data sets;   obtaining profile information with respect to at least one of input gradients, weight gradients, and output gradients calculated for each layer of layers included in the first neural network in the process of performing the backward pass;   determining one or more layers, from among the layers, to be quantized with a second bit precision less than the first bit precision, based on the obtained profile information; and   generating a second neural network by quantizing the determined layers from among the layers with the second bit precision.   
     
     
         2 . The method of  claim 1 , wherein the profile information includes a normalized statistic obtained by dividing an average of absolute values of the weight gradients by an average of absolute values of weights. 
     
     
         3 . The method of  claim 1 , wherein the profile information includes a normalized statistic obtained based on values of the weight gradients and weight values. 
     
     
         4 . The method of  claim 1 , wherein the profile information includes a statistic obtained by dividing a variance of absolute values of the input gradients by an average of the absolute values of the input gradients. 
     
     
         5 . The method of  claim 1 , wherein the profile information includes a normalized statistic obtained based on variance values of the input gradients and the input gradient values. 
     
     
         6 . The method of  claim 1 , wherein the profile information includes a normalized statistic obtained by dividing a variance of absolute values of weights for each of the layers by a number of parameters for each channel. 
     
     
         7 . The method of  claim 1 , wherein the profile information includes a normalized statistic obtained based on values of weights and a number of parameters for each of the layers. 
     
     
         8 . The method of  claim 1 , further comprising sorting the layers in an order of statistical size corresponding to the obtained profile information, wherein
 the determining of the layers comprises determining layers having a relatively small statistical size as one or more layers to be quantized from among the sorted layers.   
     
     
         9 . The method of  claim 8 , wherein the determining of the layers comprises determining one or more layers to be quantized by searching whether an accuracy loss of the second neural network is within a predetermined threshold value compared with the first neural network when some of the sorted layers are quantized with the second bit precision. 
     
     
         10 . The method of  claim 1 , wherein the first neural network corresponds to a neural network quantized from a third neural network having layers of floating-point parameters of a third bit precision that is greater than the first bit precision and having layers of fixed-point parameters of the first bit precision, and
 the second neural network corresponds to a neural network quantized such that the determined layers from among the layers have fixed-point parameters with the second bit precision and the remaining layers have fixed-point parameters with the first bit precision.   
     
     
         11 . The method of  claim 1 , wherein the method generates the second neural network by performing quantization for each channel of the determined layers of the first neural network based on the obtained profile information without retraining the second neural network. 
     
     
         12 . The method of  claim 1 , further comprising:
 performing a convolution operation between a quantized input feature map and a weight map based on a scale factor determined for each channel in an inference process using the generated second neural network;   reflecting the scale factor of the input feature map on the results of the convolution operation before calculating the partial sum for each channel of a output feature map; and   obtaining the output feature map by accumulating the results of the convolution operation on which the scale factor of the input feature map is reflected for each channel.   
     
     
         13 . The method of  claim 12 , further comprising quantizing the output feature map based on the scale factor of the weight map without determining a separate scale factor for the output feature map. 
     
     
         14 . A neural network quantization apparatus comprising:
 a memory storing at least one program; and   a processor configured to perform neural network quantization by executing the at least one program, wherein   the processor is further configured to perform, with respect to each of a plurality of input data sets, a forward pass and a backward pass of a first neural network having a first bit precision,   obtain profile information for at least one of input gradients, weight gradients, and output gradients calculated for each layer of layers included in the first neural network in the process of performing the backward pass,   determine one or more layers to be quantized with a second bit precision less than the first bit precision, among the layers, based on the obtained profile information, and   generate a second neural network by quantizing the determined layers from among the layers with the second bit precision.   
     
     
         15 . The neural network quantization apparatus of  claim 14 , wherein the profile information includes a normalized statistic by dividing an average of absolute values of the weight gradients by an average of absolute values of weights. 
     
     
         16 . The neural network quantization apparatus of  claim 14 , wherein the profile information includes a statistic obtained by dividing a variance of absolute values of the input gradients by an average of the absolute values of the input gradients. 
     
     
         17 . The neural network quantization apparatus of  claim 14 , wherein the profile information includes a normalized statistic obtained by dividing a variance of absolute values of weights for each layer of the layers by a number of parameters for each channel. 
     
     
         18 . The neural network quantization apparatus of  claim 14 , wherein the processor is further configured to sort the layers in an order of statistical size corresponding to the obtained profile information, and
 determine layers having a relatively small statistical size as one or more layers to be quantized, from among the sorted layers.   
     
     
         19 . The neural network quantization apparatus of  claim 14 , wherein the processor is further configured to determine one or more layers to be quantized by searching whether an accuracy loss of the second neural network is within a predetermined threshold value compared with the first neural network when some of the sorted layers are quantized with the second bit precision. 
     
     
         20 . The neural network quantization apparatus of  claim 14 , wherein the first neural network corresponds to a neural network quantized from a third neural network having layers of floating-point parameters of a third bit precision that is greater than the first bit precision and having layers of fixed-point parameters of the first bit precision, and
 the second neural network corresponds to a neural network quantized such that the determined layers from among the layers have fixed-point parameters with the second bit precision and the remaining layers have fixed-point parameters with the first bit precision.   
     
     
         21 . The neural network quantization apparatus of  claim 14 , wherein the processor is further configured to generate the second neural network is by performing quantization for each channel of the determined layers of the first neural network based on the obtained profile information without retraining the second neural network. 
     
     
         22 . The neural network quantization apparatus of  claim 14 , wherein the processor is further configured to:
 perform a convolution operation between a quantized input feature map and a weight map based on a scale factor determined for each channel in an inference process using the generated second neural network;   reflect the scale factor of the input feature map to the results of the convolution operation before calculating the partial sum for each channel of a output feature map; and   obtain the output feature map by accumulating the results of the convolution operation on which the scale factor of the input feature map is reflected for each channel.   
     
     
         23 . The neural network quantization apparatus of  claim 14 , wherein the processor is further configured to quantize the output feature map based on the scale factor of the weight map without determining a separate scale factor for the output feature map. 
     
     
         24 . The neural network quantization apparatus of  claim 14 , wherein a configuration of an electronic system is controlled or determined based on the neural network quantization apparatus.