Intelligent Data Partitioning for Distributed Machine Learning Systems

Abstract

Techniques for implementing intelligent data partitioning for a distributed machine learning (ML) system are provided. In one set of embodiments, a computer system implementing a data partition module can receive a training data instance for a ML task and identify, using a clustering algorithm, a cluster to which the training data instance belongs, the cluster being one of a plurality of clusters determined via the clustering algorithm that partition a data space of the ML task. The computer system can then transmit the training data instance to a ML worker of the distributed ML system that is assigned to the cluster, where the ML worker is configured to build or update a ML model using the training data instance.

What is claimed is: 
     
         1 . A method comprising:
 receiving, by a computer system that is part of a distributed machine learning (ML) system comprising a plurality of ML workers, a query data instance for an ML task;   identifying, by the computer system, a data cluster in a plurality of data clusters to which the query data instance belongs, wherein each data cluster in the plurality of data clusters corresponds to a portion of a data space of the ML task, and wherein the identifying comprises determining that the query data instance is associated with the identified data cluster;   transmitting, by the computer system, the query data instance to an ML worker in the plurality of ML workers that is assigned to the identified data cluster; and   in response to the transmitting, receiving, by the computer system, a decision or classification result for the query data instance from the ML worker.   
     
     
         2 . The method of  claim 1  wherein each ML worker in the plurality of ML workers maintains an ML model that is trained on training data from a corresponding data cluster in the plurality of data clusters. 
     
     
         3 . The method of  claim 1  wherein the ML worker maintains an ML model that is trained on training data from the identified data cluster, and wherein the ML worker generates the decision or classification result using the ML model. 
     
     
         4 . The method of  claim 1  further comprising:
 forwarding the decision or classification result to an originator of the query data instance. 
 
     
     
         5 . The method of  claim 1  further comprising:
 identifying a quorum of data clusters to which the query data instance belongs, the quorum comprising:
 the identified data cluster; and 
 one or more other data clusters in the plurality of data clusters that are also associated with the query data instance; 
 
 transmitting the query data instance to one or more other ML workers that are assigned to the one or more other data clusters respectively; and 
 receiving, from the one or more other ML workers, one or more other decision or classification results for the query data instance. 
 
     
     
         6 . The method of  claim 5  further comprising:
 compiling the decision or classification result received from the ML worker and the one or more other decision or classification results received from the one or more other ML workers into a single final result. 
 
     
     
         7 . The method of  claim 5  wherein the quorum includes all data clusters in the plurality of data clusters whose centroids are within a threshold distance from the query data instance. 
     
     
         8 . A non-transitory computer readable storage medium having stored thereon program code executable by a computer system that is part of a distributed machine learning (ML) system comprising a plurality of ML workers, the program code embodying a method comprising:
 receiving a query data instance for an ML task;   identifying a data cluster in a plurality of data clusters to which the query data instance belongs, wherein each data cluster in the plurality of data clusters corresponds to a portion of a data space of the ML task, and wherein the identifying comprises determining that the query data instance is associated with the identified data cluster;   transmitting the query data instance to an ML worker in the plurality of ML workers that is assigned to the identified data cluster; and   in response to the transmitting, receiving a decision or classification result for the query data instance from the ML worker.   
     
     
         9 . The non-transitory computer readable storage medium of  claim 8  wherein each ML worker in the plurality of ML workers maintains an ML model that is trained on training data from a corresponding data cluster in the plurality of data clusters. 
     
     
         10 . The non-transitory computer readable storage medium of  claim 8  wherein the ML worker maintains an ML model that is trained on training data from the identified data cluster, and wherein the ML worker generates the decision or classification result using the ML model. 
     
     
         11 . The non-transitory computer readable storage medium of  claim 8  wherein the method further comprises:
 forwarding the decision or classification result to an originator of the query data instance. 
 
     
     
         12 . The non-transitory computer readable storage medium of  claim 8  wherein the method further comprises:
 identifying a quorum of data clusters to which the query data instance belongs, the quorum comprising:
 the identified data cluster; and 
 one or more other data clusters in the plurality of data clusters that are also associated with the query data instance; 
 
 transmitting the query data instance to one or more other ML workers that are assigned to the one or more other data clusters respectively; and 
 receiving, from the one or more other ML workers, one or more other decision or classification results for the query data instance. 
 
     
     
         13 . The non-transitory computer readable storage medium of  claim 12  wherein the method further comprises:
 compiling the decision or classification result received from the ML worker and the one or more other decision or classification results received from the one or more other ML workers into a single final result. 
 
     
     
         14 . The non-transitory computer readable storage medium of  claim 12  wherein the quorum includes all data clusters in the plurality of data clusters whose centroids are within a threshold distance from the query data instance. 
     
     
         15 . A computer system that is part of a distributed machine learning (ML) system comprising a plurality of ML workers, the computer system comprising:
 a processor; and   a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
 receive a query data instance for an ML task; 
 identify a data cluster in a plurality of data clusters to which the query data instance belongs, wherein each data cluster in the plurality of data clusters corresponds to a portion of a data space of the ML task, and wherein the identifying comprises determining that the query data instance is associated with the identified data cluster; 
 transmit the query data instance to an ML worker in the plurality of ML workers that is assigned to the identified data cluster; and 
 in response to the transmitting, receive a decision or classification result for the query data instance from the ML worker. 
   
     
     
         16 . The computer system of  claim 15  wherein each ML worker in the plurality of ML workers maintains an ML model that is trained on training data from a corresponding data cluster in the plurality of data clusters. 
     
     
         17 . The computer system of  claim 15  wherein the ML worker maintains an ML model that is trained on training data from the identified data cluster, and wherein the ML worker generates the decision or classification result using the ML model. 
     
     
         18 . The computer system of  claim 15  wherein the program code further causes the processor to:
 forward the decision or classification result to an originator of the query data instance. 
 
     
     
         19 . The computer system of  claim 15  wherein the program code further causes the processor to:
 identify a quorum of data clusters to which the query data instance belongs, the quorum comprising:
 the identified data cluster; and 
 one or more other data clusters in the plurality of data clusters that are also associated with the query data instance; 
 
 transmit the query data instance to one or more other ML workers that are assigned to the one or more other data clusters respectively; and 
 receive, from the one or more other ML workers, one or more other decision or classification results for the query data instance. 
 
     
     
         20 . The computer system of  claim 19  wherein the program code further causes the processor to:
 compile the decision or classification result received from the ML worker and the one or more other decision or classification results received from the one or more other ML workers into a single final result. 
 
     
     
         21 . The computer system of  claim 19  wherein the quorum includes all data clusters in the plurality of data clusters whose centroids are within a threshold distance from the query data instance.