Reconfigurable distributed processing

Abstract

Distributed processing of a data collection includes receiving information for configuring a distributed processing system. A first configuration of components is formed including sources of data elements and workers configured to process data elements, distributed among computing resources. Each data element includes a partition value that identifies a subset of the workers according to a partition rule. Data elements are accepted from the sources for a first part of the data collection in a first processing epoch and the data elements are routed through the first configuration. After accepting a first part of the data collection, change of configuration is initiated to a second configuration. A succession of two or more transitions between configurations of components is performed to a succession of modified configurations, a last of which corresponds to the second configuration. Further data elements are accepted from sources of the second configuration in a second processing epoch.

What is claimed is: 
     
         1 . A method for distributed processing of a data collection, the method including:
 receiving, over an input device or port, information for configuring a distributed processing system, the configuring including forming a first configuration of components of the distributed processing system, the first configuration including a plurality of sources of data elements of the data collection and a plurality of workers configured to process data elements of the data collection, the sources and workers being distributed among a plurality of computing resources, wherein each data element includes a partition value that identifies a subset of the plurality of workers of the first configuration according to a partition rule of the first configuration; and   processing data in the distributed processing system during at least two processing epochs, the processing including:
 accepting data elements from the sources for a first part of the data collection in a first processing epoch and routing said data elements through the first configuration and completing processing of at least some of said data elements, wherein other of the data elements of the first part remain queued at components of the first configuration; 
 after accepting a first part of the data collection, initiating change of configuration of the distributed processing system from the first configuration to a second configuration; 
 after initiating the change of configuration, performing a succession of two or more transitions between configurations of components of the system to a succession of modified configurations of components, and after each transition causing transfer of data elements between components of the modified configuration, wherein a last of said modified configurations corresponds to the second configuration, thereby completing a transition from the first configuration to the second configuration; and 
 after completing a transition to the second configuration, accepting further data elements of the data collection from a plurality of sources of the second configuration in a second processing epoch.