Processing Data Feeds

Abstract

Exemplary embodiments allow performance of stream computations on real-time data streams using one or more map operations and/or one or more update operations. A map operation is a stream computation in which stream events in one or more real-time data streams are processed in a real-time manner to generate zero, one or more new stream events. An update operation is a stream computation in which stream events in one or more real-time data streams are processed in a real-time manner to create or update one or more static “slate” data structures that are stored in a durable manner.

1 . A computer-implemented method comprising:
 receiving, at a worker process, a first stream event in a real-time data stream;   processing, at the worker process, the first stream using a map operation to generate output data;   transforming the output data, at the worker process, to generate a second stream event associated with at least one slate that records a set of related stream events; and   transmitting the second stream event in an intermediate data stream.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the map operation is a stream computation in which stream events are processed to generate the output data. 
     
     
         3 . The computer-implemented method of  claim 2 , wherein the stream computation determines at least one of a popular topic in the first stream, post time information, and a user's current interest. 
     
     
         4 . The computer-implemented method of  claim 2 , wherein the stream computation includes ranking computations to determine a user's influence on other users. 
     
     
         5 . The computer-implemented method of  claim 2 , wherein the stream computation includes analyzing data published on a web site. 
     
     
         6 . The computer-implemented method of  claim 2 , wherein the stream computation includes grouping web page view events into visits and aggregating statistics based on visits. 
     
     
         7 . The computer-implemented method of  claim 2 , wherein the stream computation includes determining links on a web page that maximize a click-through rate. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the map operation is subscribed to receive a plurality of stream events in the real-time data stream. 
     
     
         9 . The computer-implemented method of  claim 1 , further comprising: determining a computational load at a plurality of worker nodes on a network; and scheduling the worker process on a particular worker node with the lowest computation load. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein the first stream event is a unit of data transmitted in the real-time data stream, and wherein the second stream event is a unit of data transmitted in the intermediate data stream in a real-time manner. 
     
     
         11 . The computer-implemented method of  claim 1 , wherein the at least one slate records the set of related stream events in a persistent storage device. 
     
     
         12 . A computer-implemented method comprising:
 receiving, at a worker process, a first stream event in a real-time data stream;   processing, at the worker process, the first stream using an update operation to generate updated data;   transforming the updated data, at the worker process, to generate a second stream event associated with a set of related stream events; and   transmitting the second stream event in an intermediate data stream.   
     
     
         13 . The computer-implemented method of  claim 12 , wherein the set of related stream event is associated with a slate that records the set of related stream events in a persistent storage device. 
     
     
         14 . The computer-implemented method of  claim 12 , wherein the update operation is a stream computation in which stream events are processed to generate the output data. 
     
     
         15 . The computer-implemented method of  claim 14 , wherein the stream computation determines at least one of a popular topic in the first stream, post time information, and a user's current interest. 
     
     
         16 . The computer-implemented method of  claim 14 , wherein the stream computation includes ranking computations to determine a user's influence on other users. 
     
     
         17 . The computer-implemented method of  claim 14 , wherein the stream computation includes analyzing data published on a web site. 
     
     
         18 . The computer-implemented method of  claim 14 , wherein the stream computation includes grouping web page view events into visits and aggregating statistics based on visits. 
     
     
         19 . The computer-implemented method of  claim 14 , wherein the stream computation includes determining links on a web page that maximize a click-through rate. 
     
     
         20 . An apparatus comprising:
 a memory; and   one or more processors coupled to the memory, the one or more processors configured to:
 receive a first stream event in a real-time data stream; 
 process the first stream using a map operation to generate output data; 
 transforming the output data to generate a second stream event associated with at least one slate that records a set of related stream events; and 
 transmitting the second stream event in an intermediate data stream.