Streaming Microservices for Stream Processing Applications

Abstract

The technology disclosed relates to a system that provides exactly-once processing of stream data. The system includes a queue manager which receives a stream of data. The system establishes aggregation intermediation checkpoints during processing of the received data. To do this, the system partitions delivery of the data stream at offsets, saves partition demarcation offsets at the end of processing windows, and saves intermediate aggregation results to a distributed file system with a window identifier (abbreviated ID) that correlates the offsets and the aggregation results. At each checkpoint, the intermediate aggregation results can be initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme.

What is claimed is: 
     
         1 . A computer-implemented method of connecting stream processing microservices in a pipeline, the method including:
 accepting input from an external client to a pipeline comprised of services selected by the external client, wherein the input is accepted in a first named queue of a queue manager;   performing a first selected service using the input from the first named queue and sending output of the first selected service to a second named queue;   performing a second selected service using the input from the second named queue and sending output of the second selected service to a third named queue; and   returning output to the external client from the third named queue.   
     
     
         2 . The computer-implemented method of  claim 1 , further including using one or more write-ahead logs (abbreviated WALs) to store offsets at each processing window during processing of the input. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein at least one of the first and second selected services is an aggregation service. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein at least one of the first and second selected services is a deduplication service. 
     
     
         5 . The computer-implemented method of  claim 1 , further including reading fault tolerantly from the first named queue to the first selected service by idempotently reading the input using at least one write-ahead log (abbreviated WAL). 
     
     
         6 . The computer-implemented method of  claim 1 , further including writing fault tolerantly from the second selected service to the third named queue using at least one WAL. 
     
     
         7 . A system including one or more processors coupled to memory, the memory loaded with computer instructions to connect stream processing microservices in a pipeline, the instructions, when executed on the processors, implement actions comprising:
 accepting input from an external client to a pipeline comprised of services selected by the external client, wherein the input is accepted in a first named queue of a queue manager;   performing a first selected service using the input from the first named queue and sending output of the first selected service to a second named queue;   performing a second selected service using the input from the second named queue and sending output of the second selected service to a third named queue; and   returning output to the external client from the third named queue.   
     
     
         8 . The system of  claim 7 , further implementing actions comprising using one or more write-ahead logs (abbreviated WALs) to store offsets at each processing window during processing of the input. 
     
     
         9 . The system of  claim 7 , wherein at least one of the first and second selected services is an aggregation service. 
     
     
         10 . The system of  claim 7 , wherein at least one of the first and second selected services is a deduplication service. 
     
     
         11 . The system of  claim 7 , further implementing actions comprising reading fault tolerantly from the first named queue to the first selected service by idempotently reading using at least one write-ahead log (abbreviated WAL) the input. 
     
     
         12 . The system of  claim 7 , further implementing actions comprising writing fault tolerantly from the second selected service to a third named queue using at least one WAL. 
     
     
         13 . A non-transitory computer readable storage medium impressed with computer program instructions to connect stream processing microservices in a pipeline, the instructions, when executed on a processor, implement a method comprising:
 accepting input from an external client to a pipeline comprised of services selected by the external client, wherein the input is accepted in a first named queue of a queue manager;   performing a first selected service using the input from the first named queue and sending output of the first selected service to a second named queue;   performing a second selected service using the input from the second named queue and sending output of the second selected service to a third named queue; and   returning output to the external client from the third named queue.   
     
     
         14 . The non-transitory computer readable storage medium of  claim 13 , implementing the method further comprising using one or more write-ahead logs (abbreviated WALs) to store offsets at each processing window during processing of the input. 
     
     
         15 . The non-transitory computer readable storage medium of  claim 13 , wherein at least one of the first and second selected services is an aggregation service. 
     
     
         16 . The non-transitory computer readable storage medium of  claim 13 , wherein at least one of the first and second selected services is a deduplication service. 
     
     
         17 . The non-transitory computer readable storage medium of  claim 13 , implementing the method further comprising reading fault tolerantly from the first named queue to the first selected service by idempotently reading the input using at least one write-ahead log (abbreviated WAL). 
     
     
         18 . The non-transitory computer readable storage medium of  claim 13 , implementing the method further comprising writing fault tolerantly from the second selected service to the third named queue using at least one WAL.