Load balancing for memory channel controllers

Abstract

Methods, systems, and apparatus, including computer-readable media, are described for performing neural network computations using a system configured to implement a neural network on a hardware circuit. The system includes a process ID unit that receives requests to obtain data from a memory that includes memory locations that are each identified by an address. For each request, the process ID unit selects a channel controller to receive the request, provides the request to be processed by the selected channel controller, and obtains the data from memory in response to processing the request using the selected channel controller. The channel controller is one of multiple channel controllers that are configured to access any memory location of the memory. The system performs the neural network computations using the data obtained from memory and resources allocated from a shared memory of the hardware circuit.

What is claimed is: 
     
         1 . (canceled) 
     
     
         2 . A method for performing neural network computations using a system configured to implement a neural network on a hardware circuit, the method comprising:
 receiving a request to obtain data for performing neural network computations from a memory comprising one or more memory channels, each memory channel comprising one or more memory locations, wherein the data is distributed across the one or more memory locations of the one or more memory channels;   determining a channel controller to fetch at least a portion of the data associated with the request, wherein the channel controller is configured to access: i) each memory channel of the one or more memory channels and ii) each memory location of the one or more memory locations that is included in each memory channel of the one or more memory channels;   processing the request, using the channel controller, by accessing the one or more memory locations for a memory channel of the one or more memory channels to obtain the data associated with the request for performing neural network computations; and   obtaining at least the portion of the data, by the channel controller, from the memory in response to processing the request; and   performing the neural network computations using at least the portion of the data obtained from memory and resources allocated from a shared memory of the hardware circuit, wherein the resources allocated from the shared memory comprises one or more circular buffers each having a buffer size determined based at least on an observed latency of the memory access for the channel controller.   
     
     
         3 . The method of  claim 2 , wherein the channel controller comprises a circular buffer unit configured to manage each of the one or more circular buffers, wherein the circular buffer unit is configured to determine whether one of the one or more circular buffers has enough space to store a portion of the data associated with the request to be processed by a corresponding channel controller. 
     
     
         4 . The method of  claim 3 , wherein the one or more circular buffers each comprises a buffer offset, wherein the circular buffer unit is further configured to:
 in response to determining that a circular buffer of the one or more circular buffers does not have enough space to store the portion of the data, allocate the portion of the data to another circular buffer of the one or more circular buffers located from the circular buffer by at least the buffer offset.   
     
     
         5 . The method of  claim 2 , wherein the data comprises a respective vector of numbers mapped to, by an embedding neural network layer of the neural network, a respective vocabulary feature in a set of vocabulary features. 
     
     
         6 . The method of  claim 2 , wherein the channel controller is determined based on a dispatch algorithm, wherein the dispatch algorithm is configured to distribute respective addresses of memory locations to a plurality of channel controllers and periodically adjust a value of an increment parameter in the dispatch algorithm to bypass a selection of a particular channel controller. 
     
     
         7 . The method of  claim 6 , further comprising:
 receiving one or more requests to obtain different inputs from the memory, each request of the one or more requests specifying an address for a memory location that stores a corresponding input;   determining, based on the dispatch algorithm, an allocation of addresses corresponding to each of the one or more requests; and   distributing the one or more requests to the plurality of channel controllers based on the determined allocation of addresses.   
     
     
         8 . The method of  claim 3 , wherein the system comprises a shared on-chip interconnect that is configured to allow the channel controller to access memory locations allocated to any memory channel of the one or more memory channels in the memory, wherein the shared on-chip interconnect comprises an on-chip interconnect interface, a channel controller interface, and a crossbar device. 
     
     
         9 . The method of  claim 2 , wherein performing the neural network computations comprises:
 determining an allocation of shared resources in the shared memory; and   performing the neural network computations based on the determined allocation of shared resources, wherein the allocation of shared resources comprises allocating an amount of scratchpad memory and a vector processing unit for performing the neural network computations.   
     
     
         10 . A system configured to implement a neural network on a hardware circuit to perform neural network computations, the system comprising:
 one or more processing devices; and   one or more non-transitory machine-readable storage devices for storing instructions that are executable by the one or more processing devices to cause performance of operations comprising:   receiving a request to obtain data for performing neural network computations from a memory comprising one or more memory channels, each memory channel comprising one or more memory locations, wherein the data is distributed across the one or more memory locations of the one or more memory channels;   determining a channel controller to fetch at least a portion of the data associated with the request, wherein the channel controller is configured to access: i) each memory channel of the one or more memory channels and ii) each memory location of the one or more memory locations that is included in each memory channel of the one or more memory channels;   processing the request, using the channel controller, by accessing the one or more memory locations for a memory channel of the one or more memory channels to obtain the data associated with the request for performing neural network computations; and   obtaining at least the portion of the data, by the channel controller, from the memory in response to processing the request; and   performing the neural network computations using at least the portion of the data obtained from memory and resources allocated from a shared memory of the hardware circuit, wherein the resources allocated from the shared memory comprises one or more circular buffers each having a buffer size determined based at least on an observed latency of the memory access for the channel controller.   
     
     
         11 . The system of  claim 10 , wherein the channel controller comprises a circular buffer unit configured to manage each of the one or more circular buffers, wherein the circular buffer unit is configured to determine whether one of the one or more circular buffers has enough space to store a portion of the data associated with the request to be processed by a corresponding channel controller. 
     
     
         12 . The system of  claim 11 , wherein the one or more circular buffers each comprises a buffer offset, wherein the circular buffer unit is further configured to:
 in response to determining that a circular buffer of the one or more circular buffers does not have enough space to store the portion of the data, allocate the portion of the data to another circular buffer of the one or more circular buffers located from the circular buffer by at least the buffer offset.   
     
     
         13 . The system of  claim 10 , wherein the data comprises a respective vector of numbers mapped to, by an embedding neural network layer of the neural network, a respective vocabulary feature in a set of vocabulary features. 
     
     
         14 . The system of  claim 10 , wherein the channel controller is determined based on a dispatch algorithm, wherein the dispatch algorithm is configured to distribute respective addresses of memory locations to a plurality of channel controllers and periodically adjust a value of an increment parameter in the dispatch algorithm to bypass a selection of a particular channel controller. 
     
     
         15 . The system of  claim 10  further comprises a shared on-chip interconnect that is configured to allow the channel controller to access memory locations allocated to any memory channel of the one or more memory channels in the memory, wherein the shared on-chip interconnect comprises an on-chip interconnect interface, a channel controller interface, and a crossbar device. 
     
     
         16 . One or more non-transitory machine-readable storage devices for implementing a neural network implemented on a hardware circuit using a system to perform neural network computations and for storing instructions that are executable by one or more processing devices to cause performance of operations comprising:
 receiving a request to obtain data for performing neural network computations from a memory comprising one or more memory channels, each memory channel comprising one or more memory locations, wherein the data is distributed across the one or more memory locations of the one or more memory channels;   determining a channel controller to fetch at least a portion of the data associated with the request, wherein the channel controller is configured to access: i) each memory channel of the one or more memory channels and ii) each memory location of the one or more memory locations that is included in each memory channel of the one or more memory channels;   processing the request, using the channel controller, by accessing the one or more memory locations for a memory channel of the one or more memory channels to obtain the data associated with the request for performing neural network computations; and   obtaining at least the portion of the data, by the channel controller, from the memory in response to processing the request; and   performing the neural network computations using at least the portion of the data obtained from memory and resources allocated from a shared memory of the hardware circuit, wherein the resources allocated from the shared memory comprises one or more circular buffers each having a buffer size determined based at least on an observed latency of the memory access for the channel controller.   
     
     
         17 . The one or more non-transitory machine-readable storage devices of  claim 16 , wherein the channel controller comprises a circular buffer unit configured to manage each of the one or more circular buffers, wherein the circular buffer unit is configured to determine whether one of the one or more circular buffers has enough space to store a portion of the data associated with the request to be processed by a corresponding channel controller. 
     
     
         18 . The one or more non-transitory machine-readable storage devices of  claim 17 , wherein the one or more circular buffers each comprises a buffer offset, wherein the circular buffer unit is further configured to:
 in response to determining that a circular buffer of the one or more circular buffers does not have enough space to store the portion of the data, allocate the portion of the data to another circular buffer of the one or more circular buffers located from the circular buffer by at least the buffer offset.   
     
     
         19 . The one or more non-transitory machine-readable storage devices of  claim 16 , wherein the data comprises a respective vector of numbers mapped to, by an embedding neural network layer of the neural network, a respective vocabulary feature in a set of vocabulary features. 
     
     
         20 . The one or more non-transitory machine-readable storage devices of  claim 16 , wherein the channel controller is determined based on a dispatch algorithm, wherein the dispatch algorithm is configured to distribute respective addresses of memory locations to a plurality of channel controllers and periodically adjust a value of an increment parameter in the dispatch algorithm to bypass a selection of a particular channel controller. 
     
     
         21 . The one or more non-transitory machine-readable storage devices of  claim 16 , wherein the system comprises a shared on-chip interconnect that is configured to allow the channel controller to access memory locations allocated to any memory channel of the one or more memory channels in the memory, wherein the shared on-chip interconnect comprises an on-chip interconnect interface, a channel controller interface, and a crossbar device.