Methods and systems for reconfiguration and repartitioning of a parallel distributed stream process

Abstract

A method of reconfiguring a stream process in a distributed system includes the initial step of managing a stream process including one or more operators. The one or more operators are communicatively associated with one or more stream targets. The one or more operators use a partition function to determine the routing of messages to the one or more stream targets. The method includes the steps of determining a safe state within the stream process, and configuring a configuration state of the one or more operators during the safe state.

1 . (canceled) 
     
     
         2 . A method of reconfiguring a stream process in a distributed system, the method comprising the steps of:
 managing the stream process including at least one operator, the at least one operator communicatively associated with at least one stream target, the at least one operator using a partition function to determine routing of messages to the at least one stream target,   wherein the at least one operator transmits batches partitioned according to the partition function based on a time derived from a size of a batch window spanned by the at least one operator.   
     
     
         3 . The method of  claim 2 , wherein the method further comprises:
 determining a safe state within the stream process; and   configuring a configuration state of the at least one operator during the safe state.   
     
     
         4 . The method of  claim 3 , wherein the configuration state is multiplexed with a data stream within the stream process. 
     
     
         5 . The method of  claim 3 , wherein the configuration state is stored for recovering of at least one operator in case of a failure. 
     
     
         6 . The method of  claim 3 , wherein the safe state is defined as a state between the processing of two batches having different identifiers from one another. 
     
     
         7 . The method of  claim 6 , wherein the values of the identifiers of the two batches are monotonically increasing or monotonically decreasing. 
     
     
         8 . The method of  claim 2 , wherein the at least one operator partitions a result of an operator function according to an old partition function and a new partition function until all batches partitioned according to the old partition function spanned by the batch window have been transmitted. 
     
     
         9 . The method of  claim 2 , further comprising the step of transmitting a combined data stream from a sending operator to at least one receiving operator partitioned according to an old partition function and a new partition function, wherein the at least one receiving operator filters the combined data stream according to the at least one sending operator's old partition function or new partition function. 
     
     
         10 . The method of  claim 2 , further comprising the steps of:
 instantiating at least one new operator; and   integrating the at least one new operator into the stream process by configuring an operator dataflow of the at least one operator of the stream process.   
     
     
         11 . The method of  claim 10 , wherein at least one of the at least one new operator is launched in an un-configured state and configured at runtime. 
     
     
         12 . The method of  claim 2 , wherein at least one of the at least one operator is removed from the stream process by configuration of an operator dataflow of the at least one operator of the stream process. 
     
     
         13 . The method of  claim 2 , wherein the at least one operator uses an acknowledgement protocol to one or more upstream operators to determine when the at least one operator can be safely shut down. 
     
     
         14 . The method of  claim 2 , wherein the at least one operator communicates with a controller process to shut down the at least one operator.