Maintaining throughput of a stream processing framework while increasing processing load

Abstract

The technology disclosed relates to maintaining throughput of a stream processing framework while increasing processing load. In particular, it relates to defining a container over at least one worker node that has a plurality workers, with one worker utilizing a whole core within a worker node, and queuing data from one or more incoming near real-time (NRT) data streams in multiple pipelines that run in the container and have connections to at least one common resource external to the container. It further relates to concurrently executing the pipelines at a number of workers as batches, and limiting simultaneous connections to the common resource to the number of workers by providing a shared connection to a set of batches running on a same worker regardless of the pipelines to which the batches in the set belong.

What is claimed is: 
     
         1 . A method of maintaining throughput of a stream processing framework while increasing processing load, the method including:
 defining a container over at least one worker node that has a plurality of workers, with one worker utilizing a whole processor core within a worker node;   queuing data from one or more incoming near real-time (NRT) data streams into multiple pipelines that run in the container and have connections to at least one common resource external to the container;   concurrently executing the pipelines at a number of workers as batches; and   limiting simultaneous connections between the common resource and the workers by providing a shared connection used to process a set of batches running on a same worker regardless of the pipelines to which the batches in the set belong.