Weight-sparse npu with fine-grained structured sparsity

Abstract

A neural processing unit is reconfigurable to process a fine-grain structured sparsity weight arrangement selected from N:M=1:4, 2:4, 2:8 and 4:8 fine-grain structured weight sparsity arrangements. A weight buffer stores weight values and a weight multiplexer array outputs one or more weight values stored in the weight buffer as first operand values based on a selected fine-grain structured sparsity weight arrangement. An activation buffer stores activation values and an activation multiplexer array outputs one or more activation values stored in the activation buffer as second operand values based on the selected fine-grain structured weight sparsity in which each respective second operand value and a corresponding first operand value forms an operand value pair. A multiplier array outputs a product value for each operand value pair.

What is claimed is: 
     
         1 . A neural processing unit, comprising:
 a weight buffer configured to store weight values in a fine-grain structured sparsity weight arrangement selected from a group of fine-grain structured sparsity weight arrangements comprising at least two arrangements of a 1:4 fine-grain structured sparsity weight arrangement, a 2:4 fine-grain structured sparsity weight arrangement, a 4:8 fine-grain structured sparsity weight arrangement, and a 2:8 fine-grain structured sparsity weight arrangement;   a weight multiplexer array configured to output one or more weight values stored in the weight buffer as first operand values based on the selected fine-grain structured sparsity weight arrangement;   an activation buffer configured to store activation values;   an activation multiplexer array comprising inputs to the activation multiplexer array coupled to the activation buffer, the activation multiplexer array configured to output one or more activation values stored in the activation buffer as second operand values, each respective second operand value and a corresponding first operand value forming an operand value pair; and   a multiplier array configured to output a product value for each operand value pair.   
     
     
         2 . The neural processing unit of  claim 1 , wherein the activation buffer comprises 8 activation registers to store 8 activation values,
 wherein the weight multiplexer array comprises a first weight multiplexer configured to select a weight register based on the selected fine-grain structured sparsity weight arrangement and to output the weight value stored in the selected weight register as a first operand value;   wherein the activation multiplexer array comprises a first activation multiplexer comprising seven inputs, each respective input of the first activation multiplexer being connected to a corresponding activation register within a first group of activation registers, the first activation multiplexer being configured to select an activation register in the first group of activation registers based on the selected fine-grain structured sparsity weight arrangement and to output the activation value stored in the selected activation register as a second operand value, the second operand value corresponding to the first operand value and forming a first operand value pair; and   wherein the multiplier array comprises a first a first multiplier unit configured to output a product value for the first operand value pair.   
     
     
         3 . The neural processing unit of  claim 2 , wherein the weight values are stored in the weight buffer in a 1:4 fine-grain structured sparsity weight arrangement, or in a 2:8 fine-grain structured sparsity weight arrangement, and
 wherein the first group of activation registers comprises 7 activation registers.   
     
     
         4 . The neural processing unit of  claim 2 , wherein the weight values are stored in the weight buffer in a 2:4 fine-grain structured sparsity weight arrangement, and
 wherein the first group of activation registers comprises 4 activation registers.   
     
     
         5 . The neural processing unit of  claim 2 , wherein the weight values are stored in the weight buffer in a 4:8 fine-grain structured sparsity weight arrangement, and
 wherein the first group of activation registers comprises 6 activation registers.   
     
     
         6 . The neural processing unit of  claim 1 , wherein the weight values are arranged in a 2:8 fine-grain structured sparsity configuration, and
 wherein the activation registers comprise two rows of four activation registers in which two output multiplexers are configured to select one activation register from each row.   
     
     
         7 . The neural processing unit of  claim 1 , wherein the weight values are arranged in a 2:4 fine-grained structured sparsity configuration, and
 wherein the activation registers comprise two rows of two activation registers in which two output multiplexers are configured to select one activation register from each row.   
     
     
         8 . A neural processing unit, comprising:
 a first weight buffer comprising an array of first weight registers, each first weight register being configured to store a weight value in a fine-grain structured sparsity weight arrangement selected from a group of fine-grain structured sparsity weight arrangements comprising at least two arrangements of a 1:4 fine-grain structured sparsity weight arrangement, a 2:4 fine-grain structured sparsity weight arrangement, a 4:8 fine-grain structured sparsity weight arrangement, and a 2:8 fine-grain structured sparsity weight arrangement;   a first weight multiplexer configured to select a first weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected first weight register as a first operand value;   a first activation buffer comprising a first predetermined number of first activation registers, each first activation register being configured to store an activation value; and   a first activation multiplexer comprising a second predetermined number of first activation multiplexer inputs, each respective input of the first activation multiplexer being connected to a corresponding first activation register within a first group of first activation registers, the first activation multiplexer being configured to select a first activation register based on the selected fine-grain structured sparsity weight arrangement and output the activation value stored in the selected first activation register as a second operand value, the activation value output as the second operand value corresponding to the weight value output as the first operand value; and   a first multiplier unit configured to output a first product value of the first operand value and the second operand value.   
     
     
         9 . The neural processing unit of  claim 8 , wherein the first predetermined number of first activation registers comprises 8, and the second predetermined number of activation inputs comprises 7. 
     
     
         10 . The neural processing unit of  claim 9 , wherein the weight values are arranged in a 1:4 fine-grain structured sparsity configuration. 
     
     
         11 . The neural processing unit of  claim 9 , wherein the weight values are arranged in a 2:4 fine-grain structured sparsity configuration. 
     
     
         12 . The neural processing unit of  claim 9 , wherein the weight values are arranged in a 4:8 fine-grain structured sparsity configuration. 
     
     
         13 . The neural processing unit of  claim 9 , wherein the weight values are arranged in a 2:8 fine-grain structured sparsity configuration. 
     
     
         14 . The neural processing unit of  claim 8 , further comprising:
 a second weight multiplexer configured to select a first weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected first weight register as a third operand value;   a second activation multiplexer comprising a second predetermined number of second activation multiplexer inputs, each respective input of the second activation multiplexer being connected to a corresponding first activation register within a second group of activation registers that is different from the first group of first activation registers, the second activation multiplexer being configured to select a first activation register based on the selected fine-grain structured sparsity weight arrangement and output the activation value stored in the selected first activation register as a fourth operand value, the activation value output as the fourth operand value corresponding to the weight value output as the third operand value; and   a second multiplier unit configured to output a second product value of the third operand value and the fourth operand value.   
     
     
         15 . The neural processing unit of  claim 14 , further comprising:
 a second weight buffer configured to store weight values of fine-grain structured sparsity weights based on the selected fine-grain structured sparsity weight arrangement;   a third weight multiplexer configured to select a second weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected second weight register as a fifth operand value;   a second activation buffer comprising a first predetermined number of second activation registers, each second activation register being configured to store an activation value;   a third activation multiplexer comprising a second predetermined number of third activation multiplexer inputs, each respective input of the third activation multiplexer being connected to a corresponding second activation register within a first group of second activation registers, the third activation multiplexer being configured to select a second activation register based on the selected fine-grain structured sparsity weight arrangement and output the activation value stored in the selected second activation register as a sixth operand value, the activation value output as the sixth operand value corresponding to the weight value output as the fifth operand value;   a third multiplier unit configured to output a third product value of the fifth operand value and the sixth operand value;   a fourth weight multiplexer configured to select a second weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected second weight register as a seventh operand value;   a fourth activation multiplexer comprising a second predetermined number of fourth activation multiplexer inputs, each respective input of the fourth activation multiplexer being connected to a corresponding second activation register within a fourth group of second activation registers that is different from the third group of activation registers, the fourth activation multiplexer being configured to select a second activation register based on the selected fine-grain structured sparsity weight arrangement and output the activation value stored in the selected second activation register as an eighth operand value; and   a fourth multiplier unit configured to output a fourth product value of the seventh operand value and the eighth operand value.   
     
     
         16 . The neural processing unit of  claim 15 , wherein the first predetermined number of first activation registers comprises 8 first activation registers,
 wherein the second predetermined number of first activation multiplexer inputs comprises 7 first activation multiplexer inputs,   wherein the second predetermined number of second activation multiplexer inputs comprises 7 second activation multiplexer inputs,   wherein the first predetermined number of second activation registers comprises 8 second activation registers,   wherein the second predetermined number of third activation multiplexer inputs comprises 7 third activation multiplexer inputs, and   wherein the second predetermined number of fourth activation multiplexer inputs comprises 7 fourth activation multiplexer inputs.   
     
     
         17 . The neural processing unit of  claim 16 , wherein the weight values are arranged in a 1:4 fine-grain structured sparsity configuration,
 wherein the first group of first activation registers comprises four first activation registers, and the second group of first activation registers comprises four first activation registers and is different from the first group of first activation registers, and   wherein the third group of second activation registers comprises four second activation registers, and the fourth group of second activation registers comprises four second activation registers and is different from the third group of second activation registers.   
     
     
         18 . The neural processing unit of  claim 16 , wherein the weight values are arranged in a 2:8 fine-grain structured sparsity configuration,
 wherein the first group of first activation registers comprises seven first activation registers, and the second group of first activation registers comprises seven first activation registers and is different from the first group of first activation registers, and   wherein the third group of second activation registers comprises seven second activation registers, and the fourth group of second activation registers comprises seven second activation registers and is different from the third group of second activation registers.   
     
     
         19 . The neural processing unit of  claim 16 , wherein the weight values are arranged in a 2:4 fine-grain structured sparsity configuration,
 wherein an activation value is stored in four first activation registers of the first activation buffer and is stored in four second activation registers of the second activation buffer,   wherein the first group of first activation registers comprises the four first activation registers storing activation values, and the second group of first activation registers comprises a same four first activation registers as the first group of second activation registers, and   wherein the third group of second activation registers comprises the four second activation registers storing activation values, and the fourth group of second activation registers comprises a same four second activation registers as the third group of activation registers.   
     
     
         20 . The neural processing unit of  claim 16 , wherein the weight values are arranged in a 4:8 fine-grain structured sparsity configuration,
 wherein an activation value is stored in six first activation registers of the first activation buffer and in six second activation registers of the second activation buffer,   wherein the first group of first activation registers comprises the six first activation registers storing activation values, and the second group of first activation registers comprises a same six activation registers as the first group of second activation registers, and   wherein the third group of second activation registers comprises six second activation registers storing activation values, and the fourth group of second activation registers comprises a same six second activation registers as the third group of second activation registers.   
     
     
         21 . The neural processing unit of  claim 15 , wherein the weight values are arranged in a 2:8 fine-grain structured sparsity configuration, and
 wherein the first activation registers comprise two rows of four activation registers in which two output multiplexers are configured to select one activation register from each row.   
     
     
         22 . The neural processing unit of  claim 15 , wherein the weight values are arranged in a 2:4 fine-grained structured sparsity configuration, and
 wherein the first activation registers comprise two rows of two activation registers in which two output multiplexers are configured to select one activation register from each row.