Balancing data partitions among dynamic services in a cloud environment

Abstract

A method includes identifying, by a first instance of a service, a first number of data partitions of a data source to be processed by the service and a second number of instances of the service available to process the first number of data partitions. The method further includes separating the first number of data partitions into a first set of data partitions and a second set of data partitions in view of the second number of instances of the service, determining a target number of data partitions from the first set of data partitions to be claimed by each of the second number of instances of the service, and claiming, by the first instance of the service, the target number of data partitions from the first set of data partitions and up to one data partition from the second set of data partitions.

What is claimed is: 
     
         1 . A method, comprising:
 assigning a plurality of data partitions to one or more instances of a service in view of a number of data partitions in the plurality of data partitions; and   rebalancing the plurality of assigned data partitions in response to a change in the number of data partitions in the plurality of data partitions or a change in a number of the one or more instances of the service.   
     
     
         2 . The method of  claim 1 , wherein assigning the plurality of data partitions to the one or more instances of the service comprises:
 identifying the number of data partitions of the plurality of data partitions;   identifying the number of instances of the service in the one or more instances of the service; and   assigning a subset of the number of data partitions in the plurality of data partitions to each of the instances to balance the plurality of data partitions across the number of instances of the service.   
     
     
         3 . The method of  claim 2 , wherein assigning the plurality of data partitions to the one or more instances of the service further comprises:
 separating, by a processing device executing a first instance of the service, the number of data partitions in the plurality of data partitions into a first set of data partitions and a second set of data partitions in view of the number of instances of the service;   determining, by the first instance of the service, a target number of data partitions from the first set of data partitions to be assigned to each of the one or more instances of the service; and   assigning the target number of data partitions from the first set of data partitions and up to one data partition from the second set of data partitions to each of the one or more instances of the service.   
     
     
         4 . The method of  claim 3 , wherein rebalancing the plurality of assigned data partitions comprises:
 determining that at least one of the number of instances of the service is assigned more or fewer than the target number of data partitions from the first set of data partitions in view of an updated number of data partitions or an updated number of instances of the service; and   updating the assignment of data partitions such that each of the instances of the service is assigned the target number of data partitions from the first set of data partitions.   
     
     
         5 . The method of  claim 1 , wherein assigning the plurality of data partitions comprises: 
 claiming, by each instance of the service, a subset of the plurality of data partitions.   
     
     
         6 . The method of  claim 1 , wherein each data partition of the first number of data partitions is associated with a lease and wherein claiming a data partition comprises:
 claiming the lease associated with the data partition.   
     
     
         7 . The method of  claim 1 , wherein the data partitions of the plurality of data partitions are identified from a data source to be processed, and wherein the one or more instances of the service comprise instances of the service that are available to process data partitions from the data source to be processed. 
     
     
         8 . A system comprising:
 a memory; and   a processing device operatively coupled to the memory, the processing device to:
 group a plurality of data partitions into a first set of data partitions and a second set of data partitions in view of a first number of data partitions in the plurality of data partitions and a second number of instances of a service executing in a computing environment; 
 determine a quota of data partitions from the first set of data partitions to be processed by each of the one or more instances of the service; and 
 assign the quota of data partitions from the first set of data partitions and up to one data partition from the second set of data partitions to each of the one or more instances of the service to be processed. 
   
     
     
         9 . The system of  claim 8 , wherein the processing device is to:
 assign an ordinal value to each of the data partitions of the plurality of data partitions.   
     
     
         10 . The system of  claim 9 , wherein to group the plurality of data partitions into the first set of data partitions and the second set of data partitions, the processing device is to:
 determine a threshold number of data partitions corresponding to a maximum multiple of the second number of instances of the service that is less than the first number of data partitions; and   group the threshold number of data partitions into the first set of data partitions in view of the ordinal value assigned to each of the data partitions.   
     
     
         11 . The system of  claim 8 , further comprising:
 a data source comprising the plurality of data partitions; and   wherein the computing environment comprises a cluster of compute nodes executing the one or more instances of the service to process the plurality of data partitions of the data source.   
     
     
         12 . The system of  claim 11 , wherein to assign the quota of data partitions from the first set of data partitions, the processing device is to:
 search for an available lease associated with a data partition; and   assign the available lease to one of the one or more instances of the service.   
     
     
         13 . The system of  claim 12 , wherein each instance of the service is to begin consuming data from the data partition after the lease of the data partition is assigned to the instance. 
     
     
         14 . The system of  claim 8 , wherein the processing device is further to:
 determine that the second number of instances of the service executing in the computing environment has changed to a third number of instances; and   update the first set of data partitions and the second set of data partitions in view of the third number of instances.   
     
     
         15 . A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:
 determine a plurality of data partitions to be processed by a first number of service instances executing on a computing cluster;   split the plurality of data partitions into a first set of data partitions and a second set of data partitions;   acquire, by each service instance, data partitions from the first set of data partitions until each service instance of the first number of service instances has claimed a target number of data partitions from the first set of data partitions; and   acquire, by each service instance, up to one data partition from the second set of data partitions.   
     
     
         16 . The non-transitory computer-readable storage medium of  claim 15 , wherein the processing device is further to:
 determine the target number of data partitions of the first set of data partitions to be assigned to each service instance in view of the plurality of data partitions and the first number of service instances.   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 15 , wherein to claim a data partition, the processing device is to claim a lease associated with the data partition. 
     
     
         18 . The non-transitory computer-readable storage medium of  claim 15 , wherein the processing device is further to:
 determine that the first number of service instances has changed to a second number of service instances; and   update the first set of data partitions and the second set of data partitions in view of the second number of service instances.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 18 , wherein to split the plurality of data partitions into the first set of data partitions and the second set of data partitions, the processing device is to:
 calculate a threshold number of data partitions corresponding to a maximum multiple of the first number of service instances less that is less than a number of the plurality of data partitions; and   group the threshold number of data partitions into the first set of data partitions and all remaining data partitions into the second set of data partitions.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 19 , wherein the processing device is further to:
 calculate the target number of data partitions from the first set of data partitions to be assigned to each service instance by dividing the maximum multiple of the first number of service instances by the first number of service instances.