Systems and methods for performing data processing operations using variable level parallelism

Abstract

Techniques for determining processing layouts to nodes of a dataflow graph. The techniques include: obtaining information specifying a dataflow graph, the dataflow graph comprising a plurality of nodes and a plurality of edges connecting the plurality nodes, the plurality of edges representing flows of data among nodes in the plurality of nodes, the plurality of nodes comprising: a first set of one or more nodes; and a second set of one or more nodes disjoint from the first set of nodes; obtaining a first set of one or more processing layouts for the first set of nodes; and determining a processing layout for each node in the second set of nodes based on the first set of processing layouts and one or more layout determination rules, the one or more layout determination rules including at least one rule for selecting among processing layouts having different degrees of parallelism.

What is claimed is: 
     
         1 . A data processing system, comprising:
 at least one computer hardware processor; and   at least one non-transitory computer readable storage medium storing processor executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
 obtaining information specifying a dataflow graph, the dataflow graph comprising a plurality of nodes, the plurality of nodes comprising:
 a first set of one or more nodes, each node in the first set of nodes representing a respective input dataset in a set of one or more input datasets; 
 a second set of one or more nodes, each node in the second set of nodes representing a respective output dataset in a set of one or more output datasets; and 
 a third set of one or more nodes, each node in the third set of nodes representing at least one respective data processing operation; 
 
 obtaining a first set of one or more processing layouts for the set of input datasets; 
 obtaining a second set of one or more processing layouts for the set of output datasets; and 
 determining a processing layout for each node in the third set of nodes based on the first set of processing layouts, the second set of processing layouts, and one or more layout determination rules including at least one rule for selecting among processing layouts having different degrees of parallelism. 
   
     
     
         2 - 29 . (canceled)