Methods and apparatus for distributed training of a neural network

Abstract

Methods, apparatus, systems and articles of manufacture for distributed training of a neural network are disclosed. An example apparatus includes a neural network trainer to select a plurality of training data items from a training data set based on a toggle rate of each item in the training data set. A neural network parameter memory is to store neural network training parameters. A neural network processor is to generate training data results from distributed training over multiple nodes of the neural network using the selected training data items and the neural network training parameters. The neural network trainer is to synchronize the training data results and to update the neural network training parameters.

What is claimed is: 
     
         1 . An apparatus for distributed training of neural networks, the apparatus comprising:
 a neural network trainer to select a plurality of training data items from a training data set based on a toggle rate of each item in the training data set;   a neural network parameter memory to store neural network training parameters; and   a neural network processor to implement a neural network to generate training data results from distributed training over a plurality of nodes of the neural network using the selected training data items and the neural network training parameters, the neural network trainer to synchronize the training data results and to update the neural network training parameters based on the synchronized training data results.   
     
     
         2 . The apparatus of  claim 1 , further including a power state controller to control an amount of power consumption of a central processing unit of the apparatus based on an average toggle rate of the training data items in the selected training data items. 
     
     
         3 . The apparatus of  claim 1 , further including:
 a toggle rate identifier to determine a toggle rate for each item of training data in the training data set;   a training data sorter to sort the items in the training data by their corresponding toggle rate; and   a training data grouper to allocate a first number of items of the sorted items to a first group, and to allocate a second number of items of the sorted items to a second group, the second number of items being sequentially located after the first number of items in the sorted training data, the selection of the plurality of training data items being performed among the first group and the second group.   
     
     
         4 . The apparatus of  claim 3 , wherein the training data grouper is further to shuffle the items allocated to the first group within the first group. 
     
     
         5 . The apparatus of  claim 1 , wherein the toggle rate associated with a training data item represents an amount of data variance within the training data item. 
     
     
         6 . The apparatus of  claim 1 , wherein the neural network is implemented as a deep neural network. 
     
     
         7 . The apparatus of  claim 1 , wherein the training data items include image data. 
     
     
         8 . A non-transitory computer readable medium comprising instructions which, when executed, cause a machine to at least:
 select a plurality of training items based on a toggle rate of training data items in a training data set;   perform neural network training using the selected plurality of training data items and stored training parameters to determine training results;   synchronize the training results with other nodes involved in distributed training; and   update stored training parameters based on the synchronized training results.   
     
     
         9 . The non-transitory computer readable medium of  claim 8 , wherein the instructions, when executed, cause the machine to at least:
 determine a toggle rate for each item of training data in the training data set;   sort the items in the training data by their corresponding toggle rate;   allocate a first number of items of the sorted items to a first group; and   allocate a second number of items of the sorted items to a second group, the second number of items being sequentially located after the first number of items in the sorted training data.   
     
     
         10 . The non-transitory computer readable medium of  claim 9 , further including shuffling the items allocated to the first group within the first group. 
     
     
         11 . The non-transitory computer readable medium of  claim 9 , wherein the toggle rate associated with a training data item represents an amount of data variance within the training item. 
     
     
         12 . The non-transitory computer readable medium of  claim 8 , wherein the instructions, when executed, cause the machine to at least:
 determine an average toggle rate of the items in the plurality of training items;   select a power state based on the average toggle rate; and   apply the selected power state to the machine.   
     
     
         13 . The non-transitory computer readable medium of  claim 8 , wherein the training data items include image data. 
     
     
         14 . A method for distributed training of neural networks, the method comprising:
 selecting, by executing an instruction with a processor of a node, a plurality of training items based on a toggle rate of training data items in a training data set;   performing, by executing an instruction with the processor of the node, neural network training using the selected plurality of training data items and stored training parameters to determine training results;   synchronizing the training results with other nodes involved in distributed training; and   updating stored training parameters based on the synchronized training results.   
     
     
         15 . The method of  claim 14 , wherein the selecting of the plurality of training items includes:
 determining a toggle rate for each item of training data in the training data set;   sorting the items in the training data by their corresponding toggle rate;   allocating a first number of items of the sorted items to a first group; and   allocating a second number of items of the sorted items to a second group, the second number of items being sequentially located after the first number of items in the sorted training data.   
     
     
         16 . The method of  claim 15 , further including shuffling the items allocated to the first group within the first group. 
     
     
         17 . The method of  claim 14 , wherein the toggle rate associated with a training data item represents an amount of data variance within the training item. 
     
     
         18 . The method of  claim 14 , further including:
 determining, by executing an instruction with the processor of the node, an average toggle rate of the items in the plurality of training items;   selecting, by executing an instruction with the processor of the node, a power state based on the average toggle rate; and   applying, by executing an instruction with the processor of the node, the selected power state to the node.