Quantization of weights in a neural network

Abstract

The present description concerns a circuit configured to perform a multiply and accumulate operation in a layer of an artificial neural network, the operation taking, as an input, an input data value and a weight, and wherein the weight only has a value within a limited set only formed of value 0, of a plurality of values equal to 2n, where n is an integer, and of a plurality of values, each equal to the product of 2n by an odd number greater than or equal to 3.

1 . A circuit, comprising:
 a layer of an artificial neural network configured to perform a multiply and accumulate (MAC) operation;   wherein the MAC operation receives, as an input, an input data value and a weight; and   wherein the weight has a value within a limited set formed only of: value 0; a plurality of values equal to 2 n , where n is an integer; and a plurality of values that are each equal to a product of 2 n  by an odd number greater than or equal to 3.   
     
     
         2 . The circuit according to  claim 1 , comprising:
 a first sub-circuit configured to perform a multiplication of the input data value by said odd number, the first sub-circuit comprising a first shift circuit configured to perform a multiplication by 2 and an adder; and   a second sub-circuit configured to perform a multiplication by value 2 n , the second sub-circuit comprising a second shift circuit configured to perform a shift by n bits.   
     
     
         3 . The circuit according to  claim 2 , wherein the first sub-circuit further comprises a first multiplexer configured to supply the adder with a value selected from among: an output of the first shift circuit; and value 0; and
 wherein the first multiplexer is configured to select the output of the first shift circuit if the weight is equal to one of said plurality of values equal to the multiplication of 2 n  by an odd number greater than or equal to 3.   
     
     
         4 . The circuit according to  claim 2 , further comprising a second multiplexer configured to supply the first sub-circuit with a value selected from among: the input data value; and value 0; and
 wherein the second multiplexer is configured to select value 0 when the weight is equal to 0.   
     
     
         5 . The circuit according to  claim 2 , further comprising a third sub-circuit configured to apply, selectively, a negative sign to an output value of the first sub-circuit, the third sub-circuit comprising, for example, an inverter, an adder configured to add 1, and a third multiplexer configured to bypass, selectively, the inverter and the adder. 
     
     
         6 . The circuit according to  claim 2 , wherein each weight is encoded over at least 4 bits, the at least 4 bits comprising:
 one bit encoding the sign of the weight;   at least one bit indicating whether the value of the weight is a value from among the plurality of values equal to the product of 2 n  by an odd number greater than or equal to 3; and   at least two bits encoding the value of n.   
     
     
         7 . The circuit according to  claim 6 , wherein the second shift circuit is configured to perform a multiplication by a power of 2, based on the value of the at least two bits encoding an exponent of 2. 
     
     
         8 . The circuit according to  claim 1 , wherein integer n is smaller than 3. 
     
     
         9 . The circuit according to  claim 1 , wherein said plurality of values equal to the product of the plurality of values 2 n  by an odd number greater than or equal to 3 comprises:
 a first plurality of values equal to the product of the plurality of values 2 n  by a first odd number; and   a second plurality of values equal to product of the plurality of values 2 n  by a second odd number that is different from the first odd number.   
     
     
         10 . The circuit according to  claim 1 , wherein the odd number is number 3. 
     
     
         11 . The circuit according to  claim 1 , wherein the odd number is selected from the group consisting of: 3, 5 and 7. 
     
     
         12 . The circuit according to  claim 1 , wherein the weight value equal to 0 is encoded by a specific value comprising a binary sequence X111, where X is equal to 0 or 1. 
     
     
         13 . The circuit according to  claim 1 , wherein the layer is a convolutional layer. 
     
     
         14 . A method of generating, by a circuit, a weighted output value of a neuron of a layer of a neural network, the method comprising:
 multiplying an input data value by a weight, wherein the weight has a value within a limited set formed only of: value 0; a plurality of values equal to 2 n , where n is an integer; and a plurality of values that are each equal to the product of 2 n  by an odd number greater than or equal to 3.   
     
     
         15 . The method according to  claim 14 , wherein multiplying comprises:
 first multiplying the input data value by said odd number, the first multiplying comprising a first shifting configured to perform a multiplication by 2 and an addition; and   second multiplying by value 2 n , the second multiplying comprising a second shifting by n bits.   
     
     
         16 . The circuit according to  claim 15 , wherein the first multiplying supplies the addition with a value selected from among: an output of the first shifting; and value 0; and
 further comprising selecting the output of the first shifting when the weight is equal to one of said plurality of values equal to the multiplication of 2 n  by an odd number greater than or equal to 3.   
     
     
         17 . The circuit according to  claim 15 , further comprising selecting for the first multiplying a value selected from among: the input data value; and value 0; and
 further comprising selecting value 0 when the weight is equal to 0.   
     
     
         18 . The circuit according to  claim 15 , further comprising selectively applying a negative sign to an output value of the first multiplying by performing a 2's complement operation. 
     
     
         19 . A method for training an artificial neural network, comprising:
 determining weights of neurons;   wherein determining is performed by: selecting each weight within a limited set formed only of: value 0; a plurality of values equal to 2 n , where n is an integer; and a plurality of values that are each equal to the product of 2 n  by an odd number greater than or equal to 3.   
     
     
         20 . The method according to  claim 19 , wherein each weight is encoded over at least 4 bits, the at least 4 bits comprising:
 one bit encoding the sign of the weight;   at least one bit indicating whether the value of the weight is a value from among the plurality of values equal to the product of 2 n  by an odd number greater than or equal to 3; and   at least two bits encoding the value of n.   
     
     
         21 . The method according to  claim 20 ,
 wherein said plurality of values equal to the product of the plurality of values 2 n  by an odd number greater than or equal to 3 comprises:
 a first plurality of values equal to the product of the plurality of values 2 n  by a first odd number; and 
 a second plurality of values equal to product of the plurality of values 2 n  by a second odd number that is different from the first odd number. 
   
     
     
         22 . The method according to  claim 20 , wherein the odd number is number 3. 
     
     
         23 . The method according to  claim 20 , wherein the odd number is selected from the group consisting of: 3, 5 and 7. 
     
     
         24 . The method according to  claim 20 , wherein the weight value equal to 0 is encoded by a specific value comprising a binary sequence X111, where X is equal to 0 or 1.