Array-based inference engine for machine learning

Abstract

An array-based inference engine includes a plurality of processing tiles arranged in a two-dimensional array of a plurality of rows and a plurality of columns. Each processing tile comprises at least one or more of an on-chip memory (OCM) configured to load and maintain data from the input data stream for local access by components in the processing tile and further configured to maintain and output result of the ML operation performed by the processing tile as an output data stream. The array includes a first processing unit (POD) configured to perform a dense and/or regular computation task of the ML operation on the data in the OCM. The array also includes a second processing unit/element (PE) configured to perform a sparse and/or irregular computation task of the ML operation on the data in the OCM and/or from the POD.

What is claimed is: 
     
         1 . An array-based inference engine configured to perform a machine learning (ML) operation on an input data stream, comprising:
 a plurality of processing tiles arranged in a two-dimensional array of a plurality of rows and a plurality of columns, wherein each processing tile of the plurality of processing tiles comprises at least one or more of
 an on-chip memory (OCM) configured to
 receive and maintain data from the input data stream for local access by components in the each processing tile; 
 maintain and output result of the ML operation performed by the each processing tile as an output data stream; 
 
 a first processing unit configured to perform a first type of computation task of the ML operation on the data in the OCM; and 
 a second processing unit configured to perform a second type of computation task of the ML operation on the data in the OCM and/or from the first processing unit. 
   
     
     
         2 . The system of  claim 1 , wherein:
 the data maintained in the OCM in the each processing tile is retrieved from and/or written to directly by the first processing unit and/or second processing unit in the each processing tile via at least one read port and/or at least one write port of the OCM, respectively.   
     
     
         3 . The system of  claim 1 , wherein:
 the input data stream includes data to be analyzed and inferred by the inference engine and/or training data used to train the inference engine for the ML operation.   
     
     
         4 . The system of  claim 1 , wherein:
 each processing tile is programmable by a set of programming instructions streamed to the inference engine.   
     
     
         5 . The system of  claim 4 , wherein:
 the set of programming instructions is configured to program the first processing unit and/or the second processing unit in the processing tile to perform one or more of: loading the data into the first processing unit and/or the second processing unit, performing the operations on the data by the first processing unit and/or the second processing unit, and writing output of the operations into the associated OCM of the processing tile.   
     
     
         6 . The system of  claim 5 , wherein:
 each processing tile is programmed to load, process, and output the input data stream and/or the output data stream via one streaming instruction, wherein the input data stream and/or the output data stream each comprises a plurality of data.   
     
     
         7 . The system of  claim 1 , further comprising:
 one or more processing blocks each including a set of the processing tiles coupled to one another via a routing element, wherein the processing blocks are coupled to one another via one or more routing elements.   
     
     
         8 . The system of  claim 1 , wherein:
 the first type of computation task of the ML operation is a dense and/or regular computation task.   
     
     
         9 . The system of  claim 1 , wherein:
 the second type of computation task of the ML operation is a sparse and/or irregular computation task.   
     
     
         10 . The system of  claim 8 , wherein:
 the first processing unit in each processing tile is configured to perform a matrix multiplication operation on the data in the OCM of the processing tile.   
     
     
         11 . The system of  claim 10 , wherein:
 the second processing unit in each processing tile is configured to perform one or more post matrix multiplication operation on output from the matrix multiplication operation by the first processing unit in the same processing tile.   
     
     
         12 . The system of  claim 11 , wherein:
 the inference engine is configured to integrate one or more post matrix multiplication operations with the matrix multiplication operation by the first processing unit in each processing tile so that the one or more post matrix multiplication operations are performed immediately on the output from the matrix multiplication by the first processing unit without having to transmit and save the output to the OCM first and to read from the OCM again for these post matrix multiplication operations.   
     
     
         13 . A method to support perform a machine learning (ML) operation on an input data stream via an inference engine, comprising:
 receiving and maintaining data from the input data stream for local access by an on-chip memory (OCM) of one of a plurality of processing tiles arranged in a two-dimensional array of a plurality of rows and a plurality of columns in the inference engine;   performing a first type of computation task of the ML operation on the data in the OCM via a first processing unit in the processing tile;   performing a second type of computation task of the ML operation on the data in the OCM and/or from the first processing unit via a second processing unit in the processing tile;   maintaining and outputting result of the ML operation performed by the processing tile as an output data stream from the OCM.   
     
     
         14 . The method of  claim 13 , further comprising:
 retrieving from and/or writing the data directly to the OCM in the each processing tile by the first processing unit and/or second processing unit in the each processing tile via at least one read port and/or at least one write port of the OCM, respectively.   
     
     
         15 . The method of  claim 13 , further comprising:
 including in the input data stream data to be analyzed and inferred by the inference engine and/or training data used to train the inference engine for the ML operation.   
     
     
         16 . The method of  claim 13 , further comprising:
 programming each processing tile by a set of programming instructions streamed to the inference engine.   
     
     
         17 . The method of  claim 16 , further comprising:
 programming the first processing unit and/or the second processing unit in the processing tile via a set of programming instructions to perform one or more of: loading the data into the first processing unit and/or the second processing unit, performing the operations on the data by the first processing unit and/or the second processing unit, and writing output of the operations into the OCM of the processing tile.   
     
     
         18 . The method of  claim 17 , further comprising:
 programming each processing tile to load, process, and output the input data stream and/or the output data stream via one streaming instruction, wherein the input data stream and/or the output data stream each comprises a plurality of data.   
     
     
         19 . The method of  claim 13 , further comprising:
 organizing one or more of the plurality of processing tiles into one of one or more processing blocks, wherein the one or more of the plurality of processing tiles in the processing block are coupled to one another via a routing element, wherein the processing blocks are coupled to one another via one or more routing elements.   
     
     
         20 . The method of  claim 13 , further comprising:
 performing a matrix multiplication operation on the data in the OCM of the processing tile via the first processing unit in the processing tile.   
     
     
         21 . The method of  claim 20 , further comprising:
 performing one or more post matrix multiplication operation on output from the matrix multiplication operation by the first processing unit in the same processing tile.   
     
     
         22 . The method of  claim 21 , further comprising:
 integrating one or more post matrix multiplication operations with the matrix multiplication operation by the first processing unit in each processing tile so that the one or more post matrix multiplication operations are performed immediately on the output from the matrix multiplication by the first processing unit without having to transmit and save the output to the OCM first and to read from the OCM again for these post matrix multiplication operations.   
     
     
         23 . A system configured to perform a machine learning (ML) operation, comprising:
 a core configured to divide the ML operation into a first plurality of sub-tasks and a second plurality of sub-tasks, wherein the first plurality of sub-tasks are executed by the core;   an inference engine comprising a plurality of processing tiles, wherein each processing tile of the plurality of processing tiles includes an on-chip memory (OCM) and one or more processing units,   wherein the OCM is configured to
 receive and maintain a stream of data for local access by the one or more processing units in the each processing tile; 
 maintain and output result of the ML operation performed by the one or more processing units in the each processing tile as an output data stream, 
   wherein the one or more processing units are configured to
 receive the second plurality of sub-tasks of the ML operation from the core; 
 perform the second plurality of sub-tasks of the ML operation on the stream of data in the OCM. 
   
     
     
         24 . The system of  claim 23 , wherein:
 the core and/or the inference engine are programmable via a set of programming instructions.   
     
     
         25 . The system of  claim 23 , wherein:
 the second plurality sub-tasks includes one or more of a dense and/or regular computation task and a sparse and/or irregular computation task of the ML operation.   
     
     
         26 . The system of  claim 23 , further comprising:
 a streaming engine configured to
 receive the second plurality of sub-tasks from the core; 
 transmit the second plurality of sub-tasks and the stream of data to the inference engine via a single load instruction. 
   
     
     
         27 . A system configured to perform a machine learning (ML) operation, comprising:
 a streaming engine configured to transmit a stream of data to an inference engine for the ML operation via a single load instruction;   said inference engine comprising a plurality of processing tiles, wherein each processing tile of the plurality of processing tiles includes an on-chip memory (OCM) and one or more processing units,   wherein the OCM is configured to
 receive and maintain the stream of data for local access by the one or more processing units in the each processing tile; 
 maintain and output result of the ML operation performed by the one or more processing units in the each processing tile as an output data stream, 
   wherein the one or more processing units are configured to perform one or more types of computation tasks of the ML operation on the stream of data in the OCM.   
     
     
         28 . A method to perform a machine learning (ML) operation, comprising:
 dividing the ML operation into a first plurality of sub-tasks and a second plurality of sub-tasks via a core;   performing the first plurality of sub-tasks and/or a second plurality of sub-tasks of the ML operation via an inference engine comprising a plurality of processing tiles, wherein each processing tile of the plurality of processing tiles includes an on-chip memory (OCM) and one or more processing units,   wherein the OCM is configured to
 receive and maintain a stream of data for local access by the one or more processing units in the each processing tile; 
 maintain and output result of the ML operation performed by the one or more processing units in the each processing tile as an output data stream, 
   wherein the one or more processing units are configured to
 receive the second plurality of sub-tasks of the ML operation from the core; 
 perform the second plurality of sub-tasks of the ML operation on the stream of data in the OCM. 
   
     
     
         29 . The method of  claim 28 , further comprising:
 programming the core and/or the inference engine via a set of programming instructions.   
     
     
         30 . The method of  claim 28 , further comprising:
 transmitting the stream of data to the inference engine via a single load instruction.   
     
     
         31 . A method to perform a machine learning (ML) operation, comprising:
 transmitting a stream of data to an inference engine for the ML operation via a single load instruction;   performing the ML operation via an inference engine comprising a plurality of processing tiles, wherein each processing tile of the plurality of processing tiles includes an on-chip memory (OCM) and one or more processing units,   wherein the OCM is configured to
 receive and maintain a stream of data for local access by the one or more processing units in the each processing tile; 
 maintain and output result of the ML operation performed by the one or more processing units in the each processing tile as an output data stream, 
   wherein the one or more processing units are configured to perform the second plurality of sub-tasks of the ML operation on the stream of data in the OCM.