Streaming Application Upgrading Method, Master Node, and Stream Computing System

Abstract

A streaming application upgrading method includes obtaining an updated logical model of a streaming application, and determining a to-be-adjusted stream by comparing the updated logical model with an initial logical model; generating an upgrading instruction according to the to-be-adjusted stream; and delivering the generated upgrading instruction to a worker node, so that the worker node adjusts, according to an indication of the upgrading instruction, a stream between PEs distributed on the worker node.

What is claimed is: 
     
         1 . A method for upgrading an initial logical model of a streaming application deployed on a stream computing system and implemented by a master node of the stream computing system, wherein the method comprises:
 obtaining an updated logical model of the streaming application when the streaming application is updated, wherein the stream computing system comprises the master node and at least one worker node, wherein a plurality of process elements (PEs) are distributed on one or more worker nodes of the at least one worker node and are each configured to process data of the streaming application, and wherein the initial logical model denotes the PEs processing the data and a direction of a data stream between the PEs;   determining, by the master node, a to-be-adjusted data stream by comparing the initial logical model with the updated logical model;   generating an upgrading instruction according to the to-be-adjusted data stream; and   delivering the upgrading instruction to a first worker node of the at least one worker node,   wherein a PE related to the to-be-adjusted data stream is located at the first worker node, and   wherein the upgrading instruction instructs the first worker node to adjust a direction of a data stream between PEs distributed on the first worker node.   
     
     
         2 . The method of  claim 1 , wherein the PEs denoted by the initial logical model are the same as PEs denoted by the updated logical model. 
     
     
         3 . The method of  claim 1 , further comprising:
 comparing the initial logical model with the updated logical model to determine a to-be-adjusted PE, wherein the PEs denoted by the initial logical model are different from PEs denoted by the updated logical model;   generating a first upgrading instruction according to the to-be-adjusted data stream;   generating a second upgrading instruction according to the to-be-adjusted PE;   delivering the first upgrading instruction to the first worker node, wherein the first upgrading instruction instructs the first worker node to adjust the direction of the data stream between the PEs distributed on the first worker node; and   delivering the second upgrading instruction to a second worker node at which the to-be-adjusted PE is located, wherein the second upgrading instruction instructs the second worker node to adjust a quantity of PEs distributed on the second worker node.   
     
     
         4 . The method of  claim 1 , further comprising:
 determining, according to a dependency relationship between an input stream and an output stream of the PE related to the to-be-adjusted data stream, a target PE to perform data recovery and a checkpoint for the target PE performing data recovery;   delivering a data recovery instruction to a worker node at which the target PE is located, wherein the data recovery instruction instructs the target PE to recover data according to the checkpoint; and   triggering the target PE to input the data that has been recovered to a downstream PE of the target PE for processing after determining that the first worker node has completed adjustment of the direction of the data stream between the PEs distributed on the first worker node.   
     
     
         5 . The method of  claim 4 , wherein the to-be-adjusted data stream comprises a to-be-updated data stream and a to-be-deleted data stream, and wherein the method comprises:
 determining, according to status data of a PE related to the to-be-updated data stream and the to-be-deleted data stream, the checkpoint for performing data recovery; and   determining, according to a dependency relationship between an input stream and an output stream of the PE related to the to-be-updated data stream and the to-be-deleted data stream, the target PE to perform data recovery, wherein status data of each PE is backed up by the PE when triggered by an output event and indicates a status in which the PE processes data.   
     
     
         6 . The method of  claim 3 , wherein the to-be-adjusted PE comprises a to-be-added PE, wherein the method further comprises selecting the second worker node according to a load status of each worker node in the stream computing system, and wherein the second upgrading instruction instructs the second worker node to create the to-be-added PE. 
     
     
         7 . The method of  claim 3 , wherein the to-be-adjusted PE comprises a to-be-deleted PE, wherein the to-be-deleted PE is located at the second worker node, and wherein the second upgrading instruction instructs the second worker node to delete the to-be-deleted PE. 
     
     
         8 . The method of  claim 1 , further comprising:
 configuring the PEs according to the initial logical model of the streaming application; and   processing, by the PEs, the data of the streaming application.   
     
     
         9 . The method of  claim 1 , wherein the initial logical model is denoted using a directed acyclic graph (DAG). 
     
     
         10 . A master node for upgrading an initial logical model of a streaming application deployed on a stream computing system, wherein the master node comprises:
 an input device;   an output device;   a processor; and   a memory storing instructions which, when executed by the processor, cause the master node to:
 obtain an updated logical model of the streaming application using the input device when the streaming application is updated, wherein the stream computing system comprises the master node and at least one worker node, wherein a plurality of process elements (PEs) are distributed on one or more worker nodes of the at least one worker node and are each configured to process data of the streaming application, and wherein the initial logical model denotes the PEs processing the data and a direction of a data stream between the PEs, and 
 determine a to-be-adjusted data stream by comparing the initial logical model of the streaming application with the updated logical model; 
 generate an upgrading instruction according to the to-be-adjusted data stream; and 
 deliver the upgrading instruction to a first worker node of the at least one worker node using the output device, wherein a PE related to the to-be-adjusted data stream is located at the first worker node, and wherein the upgrading instruction instructs the first worker node to adjust a direction of a data stream between PEs distributed on the first worker node. 
   
     
     
         11 . The master node of  claim 10 , wherein the PEs denoted by the initial logical model of the streaming application are the same as PEs denoted by the updated logical model. 
     
     
         12 . The master node of  claim 10 , wherein the instructions, when executed by the processor, further cause the master node to:
 compare the initial logical model with the updated logical model to determine a to-be-adjusted PE, wherein the PEs denoted by the initial logical model are different from PEs denoted by the updated logical model;   generate a first upgrading instruction according to the to-be-adjusted data stream;   generate a second upgrading instruction according to the to-be-adjusted PE; and   deliver the first upgrading instruction to the first worker node using the output device, wherein the first upgrading instruction instructs the first worker node to adjust the direction of the data stream between the PEs distributed on the first worker node; and   deliver the second upgrading instruction to a second worker node using the output device, wherein the second worker node comprises a worker node at which the to-be-adjusted PE is located, and wherein the second upgrading instruction instructs the second worker node to adjust a quantity of PEs distributed on the second worker node.   
     
     
         13 . The master node of  claim 10 , wherein the instructions, when executed by the processor, further cause the master node to:
 determine, according to a dependency relationship between an input stream and an output stream of the PE related to the to-be-adjusted data stream, a target PE that needs to perform data recovery and a checkpoint for the target PE performing data recovery;   deliver a data recovery instruction to a worker node at which the target PE is located using the output device, wherein the data recovery instruction instructs the target PE to recover data according to the checkpoint; and   trigger the target PE to input the data that has been recovered to a downstream PE of the target PE for processing, wherein the first worker node completes adjustment of the direction of the data stream between the PEs distributed on the first worker node.   
     
     
         14 . The master node of  claim 13 , wherein the to-be-adjusted data stream comprises a to-be-updated data stream and a to-be-deleted data stream, wherein the instructions, when executed by the processor, further cause the master node to:
 determine, according to status data of a PE related to the to-be-updated data stream and the to-be-deleted data stream, the checkpoint for performing data recovery; and   determine, according to a dependency relationship between an input stream and an output stream of the PE related to the to-be-updated data stream and the to-be-deleted data stream, the target PE to perform data recovery, wherein status data of each PE is backed up by the PE when triggered by an output event, and indicates a status in which the PE processes data.   
     
     
         15 . The master node of  claim 12 , wherein the to-be-adjusted PE comprises a to-be-deleted PE, wherein the to-be-deleted PE is located at the second worker node, and wherein the second upgrading instruction instructs the second worker node to delete the to-be-deleted PE. 
     
     
         16 . The master node of  claim 12 , wherein the to-be-adjusted PE comprises a to-be-added PE, wherein the instructions, when executed by the processor, further cause the master node to select the second worker node according to a load status of each worker node in the stream computing system, and wherein the second upgrading instruction instructs the second worker node to create the to-be-added PE. 
     
     
         17 . A stream computing system, comprising:
 at least one worker node comprising a plurality of process elements (PEs), wherein the PEs are distributed on one or more worker nodes of the at least one worker node, and are configured to process data of a streaming application deployed in the stream computing system, wherein an initial logical model of the streaming application denotes the PEs processing the data and a direction of a data stream between the PEs; and   a master node coupled to the at least one worker node, wherein the master node is configured to:
 obtain an updated logical model of the streaming application when the streaming application is updated; 
 determine a to-be-adjusted data stream by comparing the initial logical model of the streaming application with the updated logical model; 
 generating an upgrading instruction according to the to-be-adjusted data stream; and 
 delivering the upgrading instruction to a first worker node of the at least one worker node, wherein a PE related to the to-be-adjusted data stream is located at the first worker node, and wherein the upgrading instruction instructs the first worker node to adjust a direction of a data stream between PEs distributed on the first worker node, and 
   wherein the first worker node is configured to:
 receive the upgrading instruction from the master node; and 
 adjust, according to an indication of the upgrading instruction, the direction of the data stream between the PEs distributed on the first worker node. 
   
     
     
         18 . The stream computing system of  claim 17 , wherein the master node is further configured to compare the initial logical model with the updated logical model to determine the to-be-adjusted data stream, wherein the PEs denoted by the initial logical model are the same as PEs denoted by the updated logical model. 
     
     
         19 . The stream computing system of  claim 17 , wherein the master node is further configured to:
 compare the initial logical model with the updated logical model to determine a to-be-adjusted PE and the to-be-adjusted data stream, wherein the PEs denoted by the initial logical model are different from PEs denoted by the updated logical model;   generate a first upgrading instruction according to the to-be-adjusted data stream; and   
       generate a second upgrading instruction according to the to-be-adjusted PE;
 deliver the first upgrading instruction to the first worker node; and 
 deliver the second upgrading instruction to a second worker node, wherein the to-be-adjusted PE is located at the second worker node, 
 wherein the first worker node is further configured to:
 receive the first upgrading instruction from the master node; and 
 adjust, according to an indication of the first upgrading instruction, the direction of the data stream between the PEs distributed on the first worker node, and 
 
 wherein the second worker node is configured to:
 receive the second upgrading instruction from the master node; and 
 adjust, according to an indication of the second upgrading instruction, a quantity of PEs distributed on the second worker node. 
 
 
     
     
         20 . The stream computing system of  claim 17 , wherein the master node is further configured to:
 determine, according to a dependency relationship between an input stream and an output stream of the PE related to the to-be-adjusted data stream, a target PE to perform data recovery and a checkpoint for the target PE performing data recovery;   deliver a data recovery instruction to a worker node at which the target PE is located, wherein the data recovery instruction instructs the target PE to recover data according to the checkpoint; and   trigger the target PE to input the data that has been recovered to a downstream PE of the target PE for processing after determining that the first worker node has completed adjustment of the direction of the data stream between the PEs distributed on the first worker node.