Unified data processing across streaming and indexed data sets

Abstract

Systems and methods are described for unified processing of indexed and streaming data. A system enables users to query indexed data or to apply such queries to streaming data. In some instances, a user may specify a query in the same manner as if the query were intended to be run against indexed data, but may specify criteria that includes not-yet-indexed data (e.g., a future time frame). The system may convert the query into a processing graph to be applied to not-yet-indexed data, thus increasing the efficiency of the system.

What is claimed: 
     
         1 . A method comprising:
 obtaining a query for data indexed at a data processing system, the query specifying criteria for identifying search results from data items previously indexed by an indexing subsystem of the data processing system;   converting the query into a processing graph, the processing graph specifying a set of nodes and interconnections between individual nodes within the set, wherein the individual nodes designate a manipulation of data items within the processing graph and wherein the interconnections designate a routing of data items through the processing graph, and wherein the set of nodes and interconnections logically represent the query; and   applying the processing graph to a streaming data processing subsystem of the data processing system to cause the streaming data processing subsystem to process data items within a stream of not-yet-indexed messages obtained at the data processing system according to the processing graph.   
     
     
         2 . The method of  claim 1 , wherein the streaming data processing subsystem operates to retrieve messages from a source publish-subscribe messaging system including the stream of not-yet-indexed messages and to publish results of processing the data items within the stream of not-yet-indexed messages according to processing graph to a destination publish-subscribe messaging system. 
     
     
         3 . The method of  claim 1 , wherein the query is designated by a user as a recurring query, and wherein converting the query into the processing graph implements the recurring query without requiring that the query be run against data items previously indexed by the indexing subsystem. 
     
     
         4 . The method of  claim 1  further comprising displaying search results including processed data items that have resulted from processing the data items within the stream of not-yet-indexed messages according to the processing graph. 
     
     
         5 . The method of  claim 1  further comprising:
 executing the query against data items previously indexed by the indexing subsystem to identify historical results; and 
 displaying search results including the historical results identified from the data items previously indexed by the indexing subsystem and processed data items that have resulted from processing the data items within the stream of not-yet-indexed messages according to the processing graph. 
 
     
     
         6 . The method of  claim 1  further comprising:
 executing the query against data items previously indexed by the indexing subsystem to identify historical results; 
 displaying search results including the historical results identified from the data items previously indexed by the indexing subsystem and processed data items that have resulted from processing the data items within the stream of not-yet-indexed messages according to the processing graph; and 
 subsequent to displaying the search results, updating the search results to display new processed data items that have resulted from processing additional data items within the stream of not-yet-indexed messages according to the processing graph. 
 
     
     
         7 . The method of  claim 1 , wherein the stream of not-yet-indexed messages obtained at the data processing system comprises a message queue including results of processing a stream of source data by the streaming data processing subsystem according to a second processing graph. 
     
     
         8 . The method of  claim 1 , wherein the stream of not-yet-indexed messages include data items representing raw machine data. 
     
     
         9 . The method of  claim 1 , wherein the processing graph is represented as directed acyclic graph (DAG). 
     
     
         10 . The method of  claim 1 , wherein the stream of not-yet-indexed messages obtained at the data processing system comprises multiple message queues, and wherein causing the streaming data processing subsystem to process data items within the stream of not-yet-indexed messages obtained at the data processing system according to the processing graph comprises utilizing multiple processing partitions of the data processing system to process data items from individual messages queues of the multiple message queues at least partly in parallel. 
     
     
         11 . The method of  claim 1 , wherein causing the streaming data processing subsystem to process data items within the stream of not-yet-indexed messages obtained at the data processing system according to the processing graph comprises dividing the stream of not-yet-indexed messages into a plurality of message sets, each message set of the plurality of message sets representing messages obtained given time window, and processing each message set of the plurality of item sets as a distinct group. 
     
     
         12 . The method of  claim 1 , wherein stream of not-yet-indexed messages corresponds to a topic on a publish/subscribe messaging system, and wherein the query specifies the topic. 
     
     
         13 . The method of  claim 1 , wherein the processing graph includes multiple branches, and wherein causing the streaming data processing subsystem to process data items within the stream of not-yet-indexed messages obtained at the data processing system according to the processing graph comprises causing the data processing subsystem to process the data items according to the multiple branches at least partly in parallel. 
     
     
         14 . The method of  claim 1 , wherein the processing graph includes multiple phases, and wherein causing the streaming data processing subsystem to process data items within the stream of not-yet-indexed messages obtained at the data processing system according to the processing graph comprises causing the data processing subsystem to implement the multiple phases at least partly in parallel. 
     
     
         15 . A system comprising:
 a data store including computer-executable instructions; and   a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the processor to:
 obtain a query for data indexed at a data processing system, the query specifying criteria for identifying search results from data items previously indexed by an indexing subsystem of the data processing system; 
 convert the query into a processing graph, the processing graph specifying a set of nodes and interconnections between individual nodes within the set, wherein the individual nodes designate a manipulation of data items within the processing graph and wherein the interconnections designate a routing of data items through the processing graph, and wherein the set of nodes and interconnections logically represent the query; and 
 apply the processing graph to a streaming data processing subsystem of the data processing system to cause the streaming data processing subsystem to process data items within a stream of not-yet-indexed messages obtained at the data processing system according to the processing graph. 
   
     
     
         16 . The system of  claim 15 , wherein the streaming data processing subsystem operates to retrieve messages from a source publish-subscribe messaging system including the stream of not-yet-indexed messages and to publish results of processing the data items within the stream of not-yet-indexed messages according to processing graph to a destination publish-subscribe messaging system. 
     
     
         17 . The system of  claim 15 , wherein the query is designated by a user as a recurring query, and wherein converting the query into the processing graph implements the recurring query without requiring that the query be run against data items previously indexed by the indexing subsystem. 
     
     
         18 . One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
 obtain a query for data indexed at a data processing system, the query specifying criteria for identifying search results from data items previously indexed by an indexing subsystem of the data processing system;   convert the query into a processing graph, the processing graph specifying a set of nodes and interconnections between individual nodes within the set, wherein the individual nodes designate a manipulation of data items within the processing graph and wherein the interconnections designate a routing of data items through the processing graph, and wherein the set of nodes and interconnections logically represent the query; and   apply the processing graph to a streaming data processing subsystem of the data processing system to cause the streaming data processing subsystem to process data items within a stream of not-yet-indexed messages obtained at the data processing system according to the processing graph.   
     
     
         19 . The one or more non-transitory computer-readable media of  claim 18 , wherein the streaming data processing subsystem operates to retrieve messages from a source publish-subscribe messaging system including the stream of not-yet-indexed messages and to publish results of processing the data items within the stream of not-yet-indexed messages according to processing graph to a destination publish-subscribe messaging system. 
     
     
         20 . The one or more non-transitory computer-readable media of  claim 18 , wherein the query is designated by a user as a recurring query, and wherein converting the query into the processing graph implements the recurring query without requiring that the query be run against data items previously indexed by the indexing subsystem.