MALT: Distributed Data-Parallelism for Existing ML Applications

Abstract

Systems and methods are disclosed for parallel machine learning with a cluster of N parallel machine network nodes by determining k network nodes as a subset of the N network nodes to update learning parameters, wherein k is selected to disseminate the updates across all nodes directly or indirectly and to optimize predetermined goals including freshness, balanced communication and computation ratio in the cluster; sending learning unit updates to fewer nodes to reduce communication costs with learning convergence; and sending reduced learning updates and ensuring that the nodes send/receive learning updates in a uniform fashion.

What is claimed is: 
     
         1 . A method for machine learning, comprising:
 with a cluster of N parallel machine network nodes, determining k network nodes as a subset of the N network nodes to update learning parameters, wherein k is selected to disseminate the updates across all nodes directly or indirectly and to optimize predetermined goals including freshness, balanced communication and computation ratio in the cluster;   sending learning unit updates to fewer nodes to reduce communication costs with learning convergence; and   sending reduced learning updates and ensuring that the nodes send/receive learning updates in a uniform fashion.   
     
     
         2 . The method of  claim 1 , comprising training the parallel network nodes in a peer-peer fashion with per-sender receive queues. 
     
     
         3 . The method of  claim 1 , comprising performing uniform dissemination of network node updates with Halton series. 
     
     
         4 . The method of  claim 3 , wherein each learning node is selected in a uniform fashion to ensure that updates are distributed across all nodes. 
     
     
         5 . The method of  claim 3 , wherein each learning node propagates updates to k nodes, comprising picking k node IDs based on a uniform random sequence that generates successive points that uniformly cover an ID range. 
     
     
         6 . The method of  claim 5 , wherein the sequence comprises a Halton sequence. 
     
     
         7 . The method of  claim 6 , comprising maintain at each network node list of log(N) nodes and wherein the node list contains nodes in a Halton series to send updates. 
     
     
         8 . The method of  claim 1 , comprising marking each network node as 1 . . . N, wherein node 1 sends updates to a Halton series including N/2, N/4, 3N/4, N/8, 3N/8, 5N/8. 
     
     
         9 . The method of  claim 1 , comprising sending total updates in every iteration with a 2. The method of  claim 1 , comprising training the parallel network nodes in a peer-peer fashion with per-sender receive queues. 
     
     
         3 . The method of  claim 1 , comprising performing uniform dissemination of network node updates with Halton series. 
     
     
         4 . The method of  claim 3 , wherein each learning node is selected in a uniform fashion to ensure that updates are distributed across all nodes. 
     
     
         5 . The method of  claim 3 , wherein each learning node propagates updates to k nodes, comprising picking k node IDs based on a uniform random sequence that generates successive points that uniformly cover an ID range. 
     
     
         6 . The method of  claim 5 , wherein the sequence comprises a Halton sequence. 
     
     
         7 . The method of  claim 6 , comprising maintain at each network node list of log(N) nodes and wherein the node list contains nodes in a Halton series to send updates. 
     
     
         8 . The method of  claim 1 , comprising marking each network node as 1 . . . N, wherein node 1 sends updates to a Halton series including N/2, N/4, 3N/4, N/8, 3N/8, 5N/8. 
     
     
         9 . The method of  claim 1 , comprising sending total updates in every iteration with a complexity of O(N log(N)). 
     
     
         10 . The method of  claim 1 , comprising ensuring updates are sent uniformly across a range of nodes using any uniform random sequence other than Halton. 
     
     
         11 . The method of  claim 1 , comprising:
 picking a value for a number of outbound communication nodes; and   balancing computation with communication in parallel learning node.   
     
     
         12 . The method of  claim 1 , comprising balancing communication and computation costs in the cluster. 
     
     
         13 . The method of  claim 12 , comprising overlapping communication with computation in the cluster for maximum concurrency. 
     
     
         14 . The method of  claim 1 , comprising choosing ‘k’ nodes such that a time to communicate k updates is equal to a time to compute model updates for next iteration. 
     
     
         15 . A machine learning system, comprising:
 a plurality of parallel networked nodes each including
 a processor; 
 a network card coupling the processor to the other learning machine nodes; and 
 computer readable code executable by the processor for:
 determining k nodes as a subset of nodes to update model updates with 1<=k<=N, wherein k is chosen to disseminate the updates across all nodes directly or indirectly and to optimize predetermined goals of the system including freshness, balanced communication and computation ratio in the cluster; and 
 sending model updates to fewer nodes reduces communication costs but the overall model still converges; 
 sending reduced model updates and ensuring that models send/receive model updates in a uniform fashion. 
 
   
     
     
         16 . The system of  claim 15 , wherein the parallel network nodes communicate in a peer-peer fashion with per-sender receive queues. 
     
     
         17 . The system of  claim 15 , wherein the nodes perform uniform dissemination of network node updates with Halton series. 
     
     
         18 . The system of  claim 17 , wherein each learning node is selected in a uniform fashion to ensure that updates are distributed across all nodes. 
     
     
         19 . The system of  claim 17 , wherein each learning node propagates updates to k nodes, comprising picking k node IDs based on a uniform random sequence that generates successive points that uniformly cover an ID range and wherein each network node is marked as 1 . . . N, and node 1 sends updates to a Halton series including N/2, N/4, 3N/4, N/8, 3N/8, 5N/8. 
     
     
         20 . The system of  claim 15 , comprising a vector library to create unit parameters or gradients (updates to parameters) to be synchronized across network nodes and the vectors communicate over a DiSTributed One-sided Remote Memory (dstorm).