Method and system for deep neural network model size reduction by layer and filter elimination

Abstract

The present invention generally relates to the field of deep learning, and, more particularly, to a method and system for deep neural network model size reduction by layer and filter elimination. Conventional methods result in zero weight parameters but does not effectively reduce the size of the models. Thus, the method of present disclosure imbibes capability of effective discovery of sparsified layers and systematically generates smaller size models close to the performance of base model. Further, the disclosed method gains regularization effect due to the removal of unnecessary parameters. The smaller size model reduces computational burden, energy consumption, and latency along with satisfying smaller memory requirements and thus, it leads to the possibility of deploying at different edge devices that provides the opportunity to develop niche and important applications in different domains including automobiles, robotics, healthcare.

1 . A processor implemented method, comprising:
 obtaining, via one or more hardware processors, a Deep Neural Network (DNN) model, a training dataset, and one or more constraints, wherein the one or more constraints comprise a target model size and one or more performance penalties; and   iteratively performing, via the one or more hardware processors, model size reduction on the DNN model until the one or more constraints are satisfied, to obtain a compact DNN model, wherein at each iteration, model size reduction comprises:
 pruning the DNN model using a pruning technique to obtain a sparse model, wherein the pruning technique sets one or more weights in the DNN model to zero; 
 determining a configuration of the sparse model, wherein the configuration provides state information of a plurality of layers comprised in the sparse model; 
 calculating a sparsity statistic of each of the plurality of layers; 
 updating the configuration to obtain a new configuration by i) eliminating one or more layers among the plurality of layers having the sparsity statistic greater than a predefined threshold value, and ii) determining number of filters for each of the plurality of layers except for the one or more layers that were eliminated, based on the associated sparsity statistics; 
 post-processing the new configuration to create a consistent model configuration; and 
 generating a new DNN model based on the consistent model configuration and training the new DNN model using the training dataset, wherein the new DNN model is used in a subsequent iteration, and wherein the new DNN model obtained when the one or more constraints are satisfied is determined as the compact DNN model. 
   
     
     
         2 . The method of  claim 1 , wherein the one or more performance penalties set a bound on performance parameters of the compact DNN model in terms of accuracy, sensitivity, specificity, and F1-score. 
     
     
         3 . The method of  claim 1 , wherein the configuration comprises one or more parameters of each of the plurality of layers, wherein the one or more parameters comprise depth of an input tensor, depth of an output tensor, a kernel size, a stride length, padding, and type of the layer. 
     
     
         4 . The method of  claim 1 , wherein sparsity statistic of each of the plurality of layers is calculated by dividing number of weights with zero magnitude by total number of weights in the layer. 
     
     
         5 . The method of  claim 1 , wherein post-processing the new configuration comprises:
 checking if depth of an input tensor to each of the plurality of layers is equal to depth of an output tensor of an immediate previous layer; and   setting the depth of the input tensor to each of the plurality of layers as the depth of the output tensor of the previous layer if the depth of the input tensor and the output tensor are not equal.   
     
     
         6 . A system, comprising:
 a memory storing instructions;   one or more Input/Output (1/O) interfaces; and   one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
 obtain a Deep Neural Network (DNN) model, a training dataset, and one or more constraints, wherein the one or more constraints comprise a target model size and one or more performance penalties; and 
 iteratively perform model size reduction on the DNN model until the one or more constraints are satisfied, to obtain a compact DNN model, wherein at each iteration, model size reduction comprises: 
 pruning the DNN model using a pruning technique to obtain a sparse model, wherein the pruning technique sets one or more weights in the DNN model to zero; 
 determine a configuration of the sparse model, wherein the configuration provides state information of a plurality of layers comprised in the sparse model; 
 calculate a sparsity statistic of each of the plurality of layers; 
 update the configuration to obtain a new configuration by i) eliminating one or more layers among the plurality of layers having the sparsity statistic greater than a predefined threshold value, and ii) determining number of filters for each of the plurality of layers except for the one or more layers that were eliminated, based on the associated sparsity statistics; 
 post-process the new configuration to create a consistent model configuration; and 
 generate a new DNN model based on the consistent model configuration and training the new DNN model using the training dataset, wherein the new DNN model is used in a subsequent iteration, and wherein the new DNN model obtained when the one or more constraints are satisfied is determined as the compact DNN model. 
   
     
     
         7 . The system of  claim 6 , wherein the one or more performance penalties set a bound on performance parameters of the compact DNN model in terms of accuracy, sensitivity, specificity, and F1-score. 
     
     
         8 . The system of  claim 6 , wherein the configuration comprises one or more parameters of each of the plurality of layers, wherein the one or more parameters comprise depth of an input tensor, depth of an output tensor, a kernel size, a stride length, padding, and type of the layer. 
     
     
         9 . The system of  claim 6 , wherein sparsity statistic of each of the plurality of layers is calculated by dividing number of weights with zero magnitude by total number of weights in the layer. 
     
     
         10 . The system of  claim 6 , wherein post-processing the new configuration comprises:
 checking if depth of an input tensor to each of the plurality of layers is equal to depth of an output tensor of an immediate previous layer; and   setting the depth of the input tensor to each of the plurality of layers as the depth of the output tensor of the previous layer if the depth of the input tensor and the output tensor are not equal.   
     
     
         11 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
 obtaining a Deep Neural Network (DNN) model, a training dataset, and one or more constraints, wherein the one or more constraints comprise a target model size and one or more performance penalties; and   iteratively performing model size reduction on the DNN model until the one or more constraints are satisfied, to obtain a compact DNN model, wherein at each iteration, model size reduction comprises:
 pruning the DNN model using a pruning technique to obtain a sparse model, wherein the pruning technique sets one or more weights in the DNN model to zero; 
 determining a configuration of the sparse model, wherein the configuration provides state information of a plurality of layers comprised in the sparse model; 
 calculating a sparsity statistic of each of the plurality of layers; 
 updating the configuration to obtain a new configuration by i) eliminating one or more layers among the plurality of layers having the sparsity statistic greater than a predefined threshold value, and ii) determining number of filters for each of the plurality of layers except for the one or more layers that were eliminated, based on the associated sparsity statistics; 
 post-processing the new configuration to create a consistent model configuration; and 
 generating a new DNN model based on the consistent model configuration and training the new DNN model using the training dataset, wherein the new DNN model is used in a subsequent iteration, and wherein the new DNN model obtained when the one or more constraints are satisfied is determined as the compact DNN model. 
   
     
     
         12 . The one or more non-transitory machine-readable information storage mediums of  claim 11 , wherein the one or more performance penalties set a bound on performance parameters of the compact DNN model in terms of accuracy, sensitivity, specificity, and F1-score. 
     
     
         13 . The one or more non-transitory machine-readable information storage mediums of  claim 11 , wherein the configuration comprises one or more parameters of each of the plurality of layers, wherein the one or more parameters comprise depth of an input tensor, depth of an output tensor, a kernel size, a stride length, padding, and type of the layer. 
     
     
         14 . The one or more non-transitory machine-readable information storage mediums of  claim 11 , wherein sparsity statistic of each of the plurality of layers is calculated by dividing number of weights with zero magnitude by total number of weights in the layer. 
     
     
         15 . The one or more non-transitory machine-readable information storage mediums of  claim 11 , wherein post-processing the new configuration comprises:
 checking if depth of an input tensor to each of the plurality of layers is equal to depth of an output tensor of an immediate previous layer; and   setting the depth of the input tensor to each of the plurality of layers as the depth of the output tensor of the previous layer if the depth of the input tensor and the output tensor are not equal.