Systems and methods for implementing event-flow programs

Abstract

Described are systems and methods for processing data streams, e.g., to implement event flow programs in a manner that facilitate the co-existence of multiple independent event flow programs in a multi-tenant deployment. In various embodiments, an input data stream is partitioned into multiple primitive data streams that can be processed independently from each other, and portions of two or more primitive data streams are routed and combined to form a complex data stream, facilitating complex processing tasks that take the two or more primitive data streams as input.

1 . A system comprising:
 a partitioning module configured to receive a multi-dimensional input data stream partitionable along first and second dimensions, and to partition the input data stream into a plurality of primitive data streams differing in the first dimension, each of the primitive data streams including multiple sub-primitive data streams differing in the second dimension;   a plurality of physical communication channels forming a primitive bus connected to the partitioning module, each of the primitive data streams being carried on a single one of the physical communication channels of the primitive bus;   one or more physical communication channels forming a complex bus; and   a router connected between the physical communication channels forming the primitive bus and the one or more physical communication channels forming the complex bus, the router comprising a plurality of routing modules configured to filter the primitive data streams based on the second dimension and to route the filtered primitive data streams to the one or more physical communication channels forming the complex bus to create one or more complex data streams each comprising sub-primitive data streams from multiple primitive data streams and each being carried on a single one of the one or more physical communication channels forming the complex bus.   
     
     
         2 . The system of  claim 1 , wherein the complex bus comprises a plurality of physical communication channels carrying a plurality of complex data streams, and wherein the router is configured to partition an entirety of the primitive data streams along the second dimension to create the plurality of complex data streams. 
     
     
         3 . The system of  claim 2 , further comprising a plurality of primitive compute modules each connected to only one of the physical communication channels of the primitive bus and configured to process only one of the primitive data streams, and a plurality of complex compute modules each connected to only one of the physical communication channels of the complex bus and configured to process only one of the complex data streams. 
     
     
         4 . The system of  claim 3 , wherein the input data streams is an even data stream and wherein the plurality of primitive compute modules and the plurality of complex compute modules collectively implement one or more event-flow programs. 
     
     
         5 . The system of  claim 1 , wherein the multi-dimensional input data stream comprises a plurality of metrics for a plurality of monitored entities or geographic regions, the metrics corresponding to the first dimension and the monitored entities or geographic regions corresponding to the second dimension. 
     
     
         6 . The system of  claim 1 , wherein the input data stream comprises a stream of data packages including headers, and wherein the partitioning module partitions the input data stream based on a partitioning key specified in the headers. 
     
     
         7 . The system of  claim 1 , wherein the partitioning module comprises analytical functionality to partition the input data stream based on a partitioning key implicit in the data. 
     
     
         8 . The system of  claim 1 , further comprising one or more complex compute modules each connected to one of the one or more physical communication channels of the complex bus and configured to process only one of the one or more complex data streams. 
     
     
         9 . The system of  claim 8 , wherein the router is configured based on information about data needs of the one or more complex compute modules. 
     
     
         10 . The system of  claim 8 , wherein at least one of the one or more complex compute modules is further configured to output a result of the processing onto the primitive bus. 
     
     
         11 . The system of  claim 1 , wherein the one or more complex compute modules comprise at least one of a correlation module configured to correlate data of different primitive data streams, a calculation module configured to compute quantities based on input variables from multiple primitive data streams, or a complex rules module configured to evaluate a condition involving data from multiple primitive data streams. 
     
     
         12 . The system of  claim 1 , wherein the routing modules are configured to filter the primitive data streams to reduce an amount of data within each of the created one or more complex data streams to facilitate carrying the respective complex data stream on a single physical communication channel of the complex bus. 
     
     
         13 . A method comprising:
 receiving, at a partitioning module implemented by one or more computers, from a plurality of devices connected to the one or more computers, streamed data collectively constituting a multi-dimensional input data stream partitionable along first and second dimensions;   partitioning the input data stream, by the partitioning module, into a plurality of primitive data stream s differing in the first dimension, each of the primitive data streams including multiple sub-primitive data streams differing in the second dimension;   carrying the primitive data streams on a plurality of physical communication channels forming a primitive bus, each of the primitive data streams being carried on a single one of the physical communication channels of the primitive bus;   receiving the primitive data streams at a router connected to the plurality of physical communication channels of the primitive bus, the router implemented by one or more computers; and   filtering the primitive data streams, by the router, based on the second dimension and routing the filtered primitive data streams, by the router, to one or more physical communication channels forming a complex bus to create one or more complex data streams each comprising sub-primitive data streams from multiple primitive data streams and each being carried on a single one of the one or more physical communication channels forming the complex bus.   
     
     
         14 . The method of  claim 13 , wherein the complex bus comprises a plurality of physical communication channels carrying a plurality of complex data streams, and wherein filtering and routing the primitive data streams causes an entirety of the primitive data streams to be partitioned along the second dimension to create the plurality of complex data streams. 
     
     
         15 . The method of  claim 13 , further comprising processing the plurality of primitive data streams by a plurality of primitive compute modules each connected to only one of the physical communication channels of the primitive bus, and processing the one or more complex data streams by one or more complex compute modules each connected to only one of the physical communication channels of the complex bus. 
     
     
         16 . The method of  claim 15 , wherein processing the one or more complex data streams comprises at least one of correlating data of different primitive data streams, computing quantities based on input variables from multiple primitive data streams, or evaluating a condition involving data from multiple primitive data streams. 
     
     
         17 . The method of  claim 13 , wherein the multi-dimensional input data stream comprises a plurality of metrics for a plurality of monitored entities or geographic regions, the metrics corresponding to the first dimension and the monitored entities or geographic regions corresponding to the second dimension. 
     
     
         18 . The method of  claim 13 , wherein filtering the primitive data streams reduces an amount of data within each of the created one or more complex data streams to facilitate carrying the respective complex data stream on a single physical communication channel of the complex bus. 
     
     
         19 . One or more machine-readable media storing instructions for execution by one or more hardware processors, execution of the instructions causing the one or more hardware processors to perform operations comprising:
 partitioning a multi-dimensional input data stream partitionable along first and second dimensions into a plurality of primitive data streams differing in the first dimension, each of the primitive data streams including multiple sub-primitive data streams differing in the second dimension; and   filtering the primitive data streams based on the second dimension and routing the filtered primitive data streams to form one or more complex data streams each comprising sub-primitive data streams from multiple primitive data streams.   
     
     
         20 . The one or more machine-readable media of  claim 19 , wherein the filtering and routing causes an entirety of the primitive data streams to be partitioned along the second dimension to create the plurality of complex data streams.