Neural network processor

Abstract

A method for neural network processing, the method may include applying, by a neural network processor, a group of neural network kernels of respective layers of a neural network to provide neural network output results. The applying may include applying a (2D) kernel of the group of kernels on 2D layer input values of a first layer of the neural network to provide first layer output values. The applying may include scanning the 2D layer input values with the 2D kernel. The scanning may include flattening the 2D kernel, flattening the 2D layer input values, and storing first layer output values in a 3D memory unit. The applying may also include applying a 3D kernel of the group of kernels, on 3D layer input values of a second layer of the neural network to provide second layer output values, wherein the applying comprises scanning at least one feature vector of the 3D layer input values with at least one vector of the 3D kernel.

We claim: 
     
         1 . A neural network processor comprising:
 a three-dimensional (3D) memory unit configured to store scalar values in a 3D array of storage elements, the 3D array of storage elements comprised of a plurality of two-dimensional (2D) rows and a 2D array of shift registers, the 3D memory unit including data paths between corresponding storage elements in adjacent rows of the 3D array; and   hardware processing circuitry configured to perform operations comprising:   applying a two-dimensional (2D) kernel on 2D layer input values of a first layer of a plurality of layers in a neural network;   generating, based on the applying of the 2D kernel, first layer output values;   storing the first layer output values in the 3D memory unit;   second applying a 3D kernel on 3D layer input values of a second layer of the plurality of layers in the neural network;   generating, based on the second applying of the 3D kernel, second layer output values;   storing the second layer output values in the 3D memory unit;   reading first scalar values from a first row of the plurality of layers of the 3D memory unit;   shifting, via the data paths, each of the scalar values stored in the 3D memory unit to a corresponding storage element in an adjacent row in response to a completion of the reading of the first scalar values from the first row; and   outputting layer output values of a layer of the neural network from the array of shift registers; and   wherein the shifting replaces all the first scalar values stored in the first row with second scalar values stored in an adjacent second row of the 3D memory unit, and replaces the second scalar values stored in the second row with third scalar values stored in a third row of the 3D memory unit, the third row adjacent to the second row, and wherein the shifting performs a certain number of shift operations across rows of the 3D memory unit, with the certain number of shift operations set based on a depth of next layer input values.   
     
     
         2 . The neural network processor of  claim 1 , wherein each shift register of the 2D array of shift registers comprises a plurality of storage elements, the data paths linking corresponding storage elements in adjacent shift registers. 
     
     
         3 . The neural network processor of  claim 1 , the operations further comprising scanning the 2D layer input values with the 2D kernel, wherein the scanning comprises flattening the 2D kernel, and flattening the 2D layer input values. 
     
     
         4 . The neural network processor of  claim 3 , wherein the flattening of the 2D kernel comprises converting the 2D kernel to a kernel vector, wherein the flattening of the 2D layer input values comprises converting the 2D layer input values to a layer input values vector, and wherein the scanning comprises multiplying the kernel vector by the layer input values vector to provide an intermediate result vector. 
     
     
         5 . The neural network processor of  claim 4 , the operations further comprising summing elements of the intermediate result vector to provide an output scalar. 
     
     
         6 . The neural network processor of  claim 3 , wherein the flattening of the 2D layer input values comprises reading multiple line segments of the 2D layer input values to an intermediate memory unit, and reading 2D segments of the 2D layer input values to a register to provide the 2D layer input values. 
     
     
         7 . The neural network processor of  claim 1 , the operations further comprising scanning at least one feature vector of the 3D layer input values with at least one vector of the 3D kernel. 
     
     
         8 . The neural network processor of  claim 1 , wherein the second applying of the 3D kernel comprises repeating, for each vector of the 3D kernel, scanning the 3D layer input values with a vector of the 3D kernel. 
     
     
         9 . A method for neural network processing, the method comprises:
 applying, by a neural network processor, a group of neural network kernels of respective layers of a neural network to provide neural network output results;   wherein the applying of the group of kernels comprises:   applying a two dimensional (2D) kernel of the group of kernels on 2D layer input values of a first layer of the neural network to provide first layer output values, wherein the applying comprises scanning the 2D layer input values with the 2D kernel, wherein the scanning comprises flattening the 2D kernel, flattening the 2D layer input values, and storing first layer output values in a three dimensional (3D) memory unit, wherein the 3D memory unit comprises an array of shift registers, and wherein some of the shift registers are coupled to some other shift registers to shift data values across rows of the 3D memory unit; and   applying a 3D kernel of the group of kernels, on 3D layer input values of a second layer of the neural network to provide second layer output values, wherein the applying comprises scanning at least one feature vector of the 3D layer input values with at least one vector of the 3D kernel; and   outputting layer output values of a layer of the neural network from the array of shift registers, wherein the outputting includes performing a certain number of shift operations within at least one target shift register, and wherein the certain number is set based on a depth of next layer input values.   
     
     
         10 . The method according to  claim 9 , wherein the outputting comprises performing shift operations between source shift registers of the array that are parallelly coupled to target shift registers of the array. 
     
     
         11 . The method according to  claim 9 , wherein the outputting comprises performing shift operations within at least one target shift register. 
     
     
         12 . The method according to  claim 9 , wherein the outputting comprises outputting the layer output values from only some of the shift registers of the array. 
     
     
         13 . The method according to  claim 9 , wherein the flattening of the 2D kernel comprises converting the 2D kernel to a kernel vector, wherein the flattening of the 2D layer input values comprises converting the 2D layer input values to a layer input values vector, and wherein the scanning comprises multiplying the kernel vector by the layer input values vector to provide an intermediate result vector. 
     
     
         14 . The method according to  claim 13 , comprising summing elements of the intermediate result vector to provide an output scalar. 
     
     
         15 . The method according to  claim 9 , wherein the flattening of the 2D layer input values comprises reading multiple line segments of the 2D layer input values to intermediate memory unit, and reading 2D segments of the 2D layer input values to a register to provide the 2D layer input values. 
     
     
         16 . The method according to  claim 9 , wherein the applying of the 3D kernel comprises repeating, for each vector of the 3D kernel, scanning the 3D layer input values with a vector of the 3D kernel. 
     
     
         17 . An integrated circuit comprising a three-dimensional (3D) memory unit, a processing circuit, a manipulator, and one or more memory units that are configured to cooperate in:
 applying a group of neural network kernels of respective layers of a neural network to provide neural network output results;   wherein the applying of the group of kernels comprises:   applying a two dimensional (2D) kernel of the group of kernels on 2D layer input values of a first layer of the neural network to provide first layer output values, wherein the applying comprising scanning the 2D layer input values with the 2D kernel, wherein the scanning comprises flattening the 2D kernel, flattening the 2D layer input values, and storing first layer output values in a three dimensional (3D) memory unit, and wherein the 3D memory unit comprises an array of shift registers, and wherein some of the shift registers are coupled to some other shift registers to shift data values across rows of the 3D memory unit; and   applying a 3D kernel of the group of kernels, on 3D layer input values of a second layer of the neural network to provide second layer output values, wherein the applying comprises scanning at least one feature vector of the 3D layer input values with at least one vector of the 3D kernel; and   outputting layer output values of a layer of the neural network from the array of shift registers, wherein the outputting includes performing a certain number of shift operations within at least one target shift register, and wherein the certain number is set based on a depth of next layer input values.   
     
     
         18 . A non-transitory computer readable medium that stores instructions, which when executed by an integrated circuit, cause the integrated circuit to perform operations comprising:
 applying, by a neural network processor, a group of neural network kernels of respective layers of a neural network to provide neural network output results, wherein the applying of the group of kernels comprises:   applying a two dimensional (2D) kernel of the group of kernels on 2D layer input values of a first layer of the neural network to provide first layer output values, wherein the applying comprising scanning the 2D layer input values with the 2D kernel, wherein the scanning comprises flattening the 2D kernel, flattening the 2D layer input values, and storing first layer output values in a three dimensional (3D) memory unit, wherein the 3D memory unit comprises an array of shift registers, and wherein some of the shift registers are coupled to some other shift registers to shift data values across rows of the 3D memory unit; and   applying a 3D kernel of the group of kernels, on 3D layer input values of a second layer of the neural network to provide second layer output values, wherein the applying comprises scanning at least one feature vector of the 3D layer input values with at least one vector of the 3D kernel; and   outputting layer output values of a layer of the neural network from the array of shift registers, wherein the outputting includes performing a certain number of shift operations within at least one target shift register, and wherein the certain number is set based on a depth of next layer input values.   
     
     
         19 . The non-transitory computer readable medium according to  claim 18 , wherein the outputting comprises performing shift operations between source shift registers of the array that are parallelly coupled to target shift registers of the array. 
     
     
         20 . The non-transitory computer readable medium according to  claim 18 , wherein the outputting comprises performing shift operations within at least one target shift register. 
     
     
         21 . The non-transitory computer readable medium according to  claim 18 , wherein the outputting comprises outputting the layer output values from only some of the shift registers of the array. 
     
     
         22 . The non-transitory computer readable medium according to  claim 18 , wherein the flattening of the 2D kernel comprises converting the 2D kernel to a kernel vector, wherein the flattening of the 2D layer input values comprises converting the 2D layer input values to a layer input values vector, and wherein the scanning comprises multiplying the kernel vector by the layer input values vector to provide an intermediate result vector. 
     
     
         23 . The non-transitory computer readable medium according to  claim 22 , that stores instructions for summing elements of the intermediate result vector to provide an output scalar. 
     
     
         24 . The non-transitory computer readable medium according to  claim 18 , wherein the flattening of the 2D layer input values comprises reading multiple line segments of the 2D layer input values to intermediate memory unit, and reading 2D segments of the 2D layer input values to a register to provide the 2D layer input values.