Real-time dashboards, alerts and analytics for a log intelligence system

Abstract

This disclosure describes how data supporting real-time reporting services can be cached during a log intake process. In particular, instead of caching all the log data being generated by an operational system, only the log data relevant to existing queries associated with the real-time reporting services are cached. In some embodiments, only particular metrics contained within the log data are stored for rapid access by the real-time reporting services.

What is claimed is: 
     
         1 . A computer implemented method for displaying metrics associated with log data, the computer implemented method comprising:
 at a data plane:
 receiving a stream of log data generated by an operational system; 
 extracting a subset of log data from the received stream of log data, the extracting performed in accordance with a set of rules based on predefined queries of a real-time reporting service; 
 storing the extracted subset of log data; and 
 transmitting one or more metrics included in the extracted subset of log data to a real-time reporting service. 
   
     
     
         2 . The computer implemented method of  claim 1 , further comprising:
 providing the one or more metrics to a user of the real-time reporting service.   
     
     
         3 . The computer implemented method of  claim 1 , wherein the data plane is a first data plane, and the receiving further comprises receiving the stream of log data from a second data plane different from the first data plane. 
     
     
         4 . The computer implemented method of  claim 3 , wherein the first data plane is a constraint plane, and the second data plane is an ingestion plane configured to receive the stream of log data from the operational system and to forward the stream of log data to the first data plane. 
     
     
         5 . The computer implemented method of  claim 1 , wherein the storing further comprises, of the received stream of log data, storing only the extracted subset of log data and discarding a remainder of the log data besides the extracted subset of log data. 
     
     
         6 . The computer implemented method of  claim 1 , wherein the subset of log data includes only logs of the received log data that include metrics specified by the set of rules. 
     
     
         7 . The computer implemented method of  claim 1 , wherein the storing further comprises storing the entire extracted subset of log data at a single shard. 
     
     
         8 . The computer implemented method of  claim 1 , wherein the real-time reporting service comprises a dashboard service. 
     
     
         9 . The computer implemented method of  claim 8 , further comprising initiating a graphical display of the one or more metrics, the graphical display illustrating a number of occurrences of an event type over a predefined period of time. 
     
     
         10 . The computer implemented method of  claim 1 , wherein the subset of log data includes only metric data. 
     
     
         11 . The computer implemented method of  claim 1 , further comprising updating the set of rules in response to an update to one or more of the queries of the real-time reporting service. 
     
     
         12 . The computer implemented method of  claim 11 , wherein the data plane is a first data plane, the method further comprising requesting historical data from a second data plane different from the first data plane when metrics specified by the one or more queries are not stored at the first data plane. 
     
     
         13 . The computer implemented method of  claim 12 , further comprising, at the first data plane, receiving the historical data from the second data plane. 
     
     
         14 . The computer implemented method of  claim 13 , further comprising extracting, from the historical data, one or more metrics specified by the queries. 
     
     
         15 . A non-transitory computer-readable storage medium storing instructions configured to be executed by one or more processors of a computing device cause the computing device to carry out steps that include:
 at a data plane:
 receiving a stream of log data generated by an operational system; 
 extracting a subset of log data from the received stream of log data, the extracting performed in accordance with a set of rules based on predefined queries of a real-time reporting service; 
 storing the extracted subset of log data; and 
 transmitting one or more metrics included in the extracted subset of log data to a real-time reporting service. 
   
     
     
         16 . The non-transitory computer-readable storage medium of  claim 15 , wherein the data plane is a first data plane, and the receiving further comprises receiving the stream of log data from a second data plane different from the first data plane. 
     
     
         17 . The non-transitory computer-readable storage medium of  claim 16 , wherein the first data plane is a constraint plane, and the second data plane is an ingestion plane configured to receive the stream of log data from the operational system and to forward the stream of log data to the first data plane. 
     
     
         18 . A computer system, comprising:
 one or more processors; and   memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
 at a data plane:
 receiving a stream of log data generated by an operational system; 
 extracting a subset of log data from the received stream of log data, the extracting performed in accordance with a set of rules based on predefined queries of a real-time reporting service; 
 storing the extracted subset of log data; and 
 transmitting one or more metrics included in the extracted subset of log data to a real-time reporting service. 
 
   
     
     
         19 . The computer system of  claim 18 , wherein the data plane is a first data plane, and the receiving further comprises receiving the stream of log data from a second data plane different from the first data plane. 
     
     
         20 . The computer system of  claim 19 , wherein the first data plane is a constraint plane, and the second data plane is an ingestion plane configured to receive the stream of log data from the operational system and to forward the stream of log data to the first data plane.