Stream Computing Method, Apparatus, and System

Abstract

A stream computing method performed by a manager node includes obtaining input channel description information, a structured query language (SQL) statement, and output channel description information, dynamically generating a data flow diagram according to the input channel description information, the SQL statement, and the output channel description information, and controlling, according to the data flow diagram, a worker node to execute a stream computing task.

1 . A stream computing method performed by a manager node of a stream computing system, wherein the stream computing method comprises:
 generating a first data flow diagram according to input channel description information, a structured query language (SQL) statement, and output channel description information, wherein the first data flow diagram comprises a plurality of logical nodes, wherein the input channel description information defines an input channel, wherein the input channel is a first logical channel for inputting an input data stream from a data production system to the first data flow diagram, wherein the output channel description information defines an output channel, and wherein the output channel is a logical channel for outputting an output data stream of the first data flow diagram to a data consumption system;   classifying the logical nodes in the first data flow diagram to obtain a plurality of logical node groups;   selecting, from a preset operator library, a common operator corresponding to each logical node group of the logical node groups;   generating a second data flow diagram according to the common operator, wherein each operator in the second data flow diagram implements functions of one or more logical nodes in a logical node group corresponding to the operator; and   controlling, according to the second data flow diagram, a worker node of the stream computing system to execute a stream computing task.   
     
     
         2 . The stream computing method of  claim 1 , wherein the first data flow diagram comprises a source logical node, an intermediate logical node, and a sink logical node coupled using first directed edges, and wherein the second data flow diagram comprises a source operator, an intermediate operator, and a sink operator coupled using second directed edges. 
     
     
         3 . The stream computing method of  claim 1 , wherein the SQL statement comprises a plurality of SQL rules, and wherein the stream computing method further comprises:
 generating a source logical node in the first data flow diagram according to the input channel description information, wherein the source logical node is for receiving the input data stream from the data production system;   generating intermediate logical nodes in the first data flow diagram according to select substatements in the SQL rules, wherein the intermediate logical nodes indicate computational logic for computing the input data stream, and wherein each intermediate logical node of the intermediate logical nodes corresponds to one SQL rule of the SQL rules;   generating a sink logical node in the first data flow diagram according to the output channel description information, wherein the sink logical node is for sending the output data stream to the data consumption system; and   generating first directed edges between the source logical node, the intermediate logical nodes, and the sink logical node according to an input substatement in each SQL rule of the SQL rules, an output substatement in each SQL rule of the SQL rules, or both the input substatement and the output substatement.   
     
     
         4 . The stream computing method of  claim 3 , wherein the second data flow diagram comprises a source operator, an intermediate operator, and a sink operator coupled using second directed edges, wherein the preset operator library comprises a common source operator, a common intermediate operator, and a common sink operator, and wherein the stream computing method further comprises:
 compiling the common source operator to obtain the source operator in the second data flow diagram;   selecting, from the preset operator library, at least one common intermediate operator for each logical node group of the logical node groups;   compiling the at least one common intermediate operator to obtain the intermediate operator in the second data flow diagram;   compiling, the common sink operator to obtain the sink operator in the second data flow diagram; and   generating the second directed edges according to the first directed edges.   
     
     
         5 . The stream computing method of  claim 4 , further comprising:
 receiving, from a client, modification information for modifying an SQL rule of the SQL rules; and   adding, modifying, or deleting, according to the modification information, an intermediate operator in the second data flow diagram.   
     
     
         6 . The stream computing method of  claim 4 , further comprising:
 receiving, from a client, modification information for modifying the input channel description information; and   adding, modifying, or deleting the source operator in the second data flow diagram according to the modification information.   
     
     
         7 . The stream computing method of  claim 4 , further comprising:
 receiving, from a client, modification information for modifying the output channel description information; and   adding, modifying, or deleting the sink operator in the second data flow diagram according to the modification information.   
     
     
         8 . The stream computing method of  claim 1 , wherein controlling the worker node to execute the stream computing task comprises:
 scheduling each operator in the second data flow diagram to at least one worker node of a plurality of working nodes in the stream computing system,   generating, according to an output data stream of each operator in the second data flow diagram, subscription publication information corresponding to the operator, wherein the subscription publication information indicates a manner of sending a respective output data stream corresponding to the operator;   configuring, for each operator in the second data flow diagram, the subscription publication information for the operator;   generating, for each operator in the second data flow diagram and according to an input data stream of the operator, input stream definition information corresponding to the operator, wherein the input stream definition information indicates a manner of receiving a respective input data stream corresponding to the operator; and   configuring, for each operator in the second data flow diagram, the input stream definition information for the operator.   
     
     
         9 . A manager node, comprising:
 a memory configured to store instructions; and   a processor coupled to the memory and configured to execute the instructions to cause the manager node to:
 generate a first data flow diagram according to input channel description information, a structured query language (SQL) statement, and output channel description information, wherein the first data flow diagram comprises a plurality of logical nodes, wherein the input channel description information defines an input channel, wherein the input channel is a logical channel for inputting an input data stream from a data production system to the first data flow diagram, the output channel description information defines an output channel, and wherein the output channel is a logical channel for outputting an output data stream of the first data flow diagram to a data consumption system; 
 classify the logical nodes in the first data flow diagram to obtain a plurality of logical node groups; 
   select, from a preset operator library, a common operator corresponding to each logical node group of the logical node groups; and   generate a second data flow diagram according to the common operator, wherein each operator in the second data flow diagram implements functions of one or more logical nodes in a logical node group corresponding to the operator; and   control, according to the second data flow diagram, a worker node of a stream computing system to execute a stream computing task.   
     
     
         10 . The manager node of  claim 9 , wherein the first data flow diagram comprises a source logical node, an intermediate logical node, and a sink logical node coupled using first directed edges, and wherein the second data flow diagram comprises a source operator, an intermediate operator, and a sink operator coupled using second directed edges. 
     
     
         11 . The manager node of  claim 9 , wherein the SQL statement comprises a plurality of SQL rules, and wherein the instructions further cause the processor to be manage node to:
 generate a source logical node in the first data flow diagram according to the input channel description information, wherein the source logical node is for receiving the input data stream from the data production system;   generating intermediate logical nodes in the first data flow diagram according to select substatements in each SQL rule of the SQL rules, wherein the intermediate logical nodes indicate computational logic for computing the input data stream, and wherein each intermediate logical node of the intermediate logical nodes corresponds to one SQL rule of the SQL rules;   generate a sink logical node in the first data flow diagram according to the output channel description information, wherein the sink logical node is for sending the output data stream to the data consumption system; and   generate first directed edges between the source logical node, the intermediate logical nodes, and the sink logical node according to an input substatement in each SQL rule of the SQL rules, an output substatement in each SQL rule of the SQL rules, or both the input substatement or the output substatement.   
     
     
         12 . The manager node of  claim 11 , wherein the second data flow diagram comprises a source operator, an intermediate operator, and a sink operator coupled using second directed edges, wherein the preset operator library comprises a common source operator, a common intermediate operator, and a common sink operator, and wherein the instructions further cause the manager node to:
 compile the common source operator to obtain the source operator in the second data flow diagram;   select, from the preset operator library, at least one common intermediate operator for each logical node group of the logical node groups;   compile the at least one common intermediate operator to obtain the intermediate operator in the second data flow diagram;   compile the common sink operator to obtain the sink operator in the second data flow diagram; and   generate the second directed edges according to the first directed edges.   
     
     
         13 . The manager node of  claim 12 , wherein the instructions further cause the manager node to:
 receive, from a client, modification information for modifying an SQL rule of the SQL rules; and   add, modify, or delete, according to the modification information, an intermediate operator in the second data flow diagram.   
     
     
         14 . The manager node of  claim 12 , wherein the instructions further cause the manager node to:
 receive, from a client, modification information for modifying the input channel description information; and   add, modify, or delete the source operator in the second data flow diagram according to the second modification information.   
     
     
         15 . The manager node of  claim 12 , wherein the instructions further cause the manager node to:
 receive, from a client, modification information for modifying the output channel description information; and   add, modify, or delete the sink operator in the second data flow diagram according to the modification information.   
     
     
         16 . The manager node of  claim 10 , wherein the instructions further cause the manager node to schedule each operator in the second data flow diagram to at least one worker node in the stream computing system. 
     
     
         17 . A computer program product comprising instructions that are stored on a computer-readable medium and that, when executed by a processor, cause a manager node to:
 generate a first data flow diagram according to input channel description information, a structured query language (SQL) statement, and output channel description information, wherein the first data flow diagram comprises a plurality of logical nodes, wherein the input channel description information defines an input channel, wherein the input channel is a logical channel for inputting an input data stream from a data production system to the first data flow diagram, wherein the output channel description information defines an output channel, and wherein the output channel is a logical channel for outputting an output data stream of the first data flow diagram to a data consumption system;   classify the logical nodes in the first data flow diagram to obtain a plurality of logical node groups;   select, from a preset operator library, a common operator corresponding to each logical node group of the logical node groups; and   generate a second data flow diagram according to the common operator, wherein each operator in the second data flow diagram implements functions of one or more logical nodes in a logical node group corresponding to the operator; and   control, according to the second data flow diagram, a worker node of a stream computing system to execute the stream computing task.   
     
     
         18 . The computer program product of  claim 17 , wherein the first data flow diagram comprises a source logical node, an intermediate logical node, and a sink logical node coupled using first directed edges, and wherein the second data flow diagram comprises a source operator, an intermediate operator, and a sink operator coupled using second directed edges. 
     
     
         19 . The computer program product of  claim 17 , wherein the SQL statement comprises a plurality of SQL rules, and wherein when executed by the processor, the instructions further cause the manager node to:
 generate a source logical node in the first data flow diagram according to the input channel description information, wherein the source logical node is for receiving the input data stream from the data production system;   generate intermediate logical nodes in the first data flow diagram according to select substatements in the SQL rules, wherein the intermediate logical nodes indicate computational logic for computing the input data stream, and wherein each intermediate logical node of the intermediate logical nodes corresponds to one SQL rule of the SQL rules;   generate a sink logical node in the first data flow diagram according to the output channel description information, wherein the sink logical node is for sending the output data stream to the data consumption system; and   generate first directed edges between the source logical node, the intermediate logical nodes, and the sink logical node according to an input substatement in each SQL rule of the SQL rules, an output substatement in each SQL rule of the SQL rules, or both the input substatement and the output substatement.   
     
     
         20 . The computer program product of  claim 19 , wherein the second data flow diagram comprises a source operator, an intermediate operator, and a sink operator coupled using second directed edges, wherein the preset operator library comprises a common source operator, a common intermediate operator, and a common sink operator, and wherein when executed by the processor, the instructions further cause the manager node to:
 compile the common source operator to obtain the source operator in the second data flow diagram;   select, from the preset operator library, at least one common intermediate operator for each logical node group of the logical node groups;   compile the at least one common intermediate operator to obtain the intermediate operator in the second data flow diagram;   compile the common sink operator to obtain the sink operator in the second data flow diagram; and   generate the second directed edges according to the first directed edges.