Machine learning output sampling for a data intake and query system

Abstract

Systems and methods are described for providing a user interface through which a user can program operation of a data processing pipeline by specifying a graph of nodes that transform data and interconnections that designate routing of data between individual nodes within the graph. In response to a user request, a preview mode can be activated that causes the data processing pipeline to retrieve data from at least one source specified by the graph, transform the data according to the nodes of the graph, sample the transformed data, and display the sampling of the transformed data to at least one node without writing the transformed data to at least one destination specified by the graph.

What is claimed is: 
     
         1 . A method, comprising:
 providing a user interface depicting a graph representing a data processing pipeline;   receiving, via the user interface, a request to activate a preview mode in association with a machine learning model;   obtaining first data generated by a component of the data processing pipeline interconnected with the machine learning model;   applying the first data as an input to the machine learning model to generate output data; and   causing the user interface to display a preview of a portion of the output data that comprises a sampling of one or more different types of data output by the machine learning model.   
     
     
         2 . The method of  claim 1 , wherein causing the user interface to display a preview further comprises causing the user interface to display the preview without writing the output data to at least one destination specified by the graph. 
     
     
         3 . The method of  claim 1 , further comprising retrieving input data from at least one source specified by the graph in response to the request to activate the preview mode. 
     
     
         4 . The method of  claim 1 , wherein the first data comprises live data streamed from a source specified by the graph. 
     
     
         5 . The method of  claim 1 , further comprising:
 retrieving input data from at least one source specified by the graph in response to the request to activate the preview mode; and   causing the input data to be transformed according to the component of the data processing pipeline to generate the first data.   
     
     
         6 . The method of  claim 1 , further comprising transmitting an abstract syntax tree (AST) of the data processing pipeline to an intake system, wherein the intake system produces an augmented AST by causing a function of the graph that writes to an external database to drop received data instead of writing the received data to the external database and by adding a preview node to the graph in association with the machine learning model. 
     
     
         7 . The method of  claim 1 , further comprising transmitting an abstract syntax tree (AST) of the data processing pipeline to an intake system, wherein the intake system produces an augmented AST by causing a function of the graph that writes to an external database to drop received data instead of writing the received data to the external database and by adding a preview node to the graph in association with the machine learning model, and wherein the intake system runs a job using the augmented AST that results in the first data being transmitted to the preview node. 
     
     
         8 . The method of  claim 1 , further comprising transmitting an abstract syntax tree (AST) of the data processing pipeline to an intake system, wherein the intake system produces an augmented AST by causing a function of the graph that writes to an external database to drop received data instead of writing the received data to the external database and by adding a preview node to the graph in association with the machine learning model, wherein the intake system runs a job using the augmented AST that results in the first data being transmitted to the preview node, and wherein applying the first data as an input to the machine learning model to generate output data further comprises applying, by the preview node, the first data as an input to the machine learning model to generate output data. 
     
     
         9 . The method of  claim 1 , wherein the first data comprises a stream of data items generated by the component of the data processing pipeline in sequence, and wherein applying the first data as an input to the machine learning model further comprises applying, in sequence, each of the data items of the stream of data items as an input to the machine learning model to generate the output data. 
     
     
         10 . The method of  claim 1 , wherein applying the first data as an input to the machine learning model to generate output data further comprises applying the first data as the input to the machine learning model for a first period of time. 
     
     
         11 . The method of  claim 1 , wherein applying the first data as an input to the machine learning model to generate output data further comprises applying the first data as the input to the machine learning model for a first period of time, and wherein the first data corresponds to a second period of time. 
     
     
         12 . The method of  claim 1 , wherein applying the first data as an input to the machine learning model to generate output data further comprises applying the first data as the input to the machine learning model for a first period of time, and wherein the first data corresponds to a second period of time greater than the first period of time. 
     
     
         13 . The method of  claim 1 , wherein the first data comprises a stream of data items generated by the component of the data processing pipeline in sequence, wherein applying the first data as an input to the machine learning model to generate output data further comprises:
 for each data item of the stream of data items in sequence, applying the respective data item as an input to the machine learning model to generate a portion of the output data; and   determining, a first period of time after an initial portion of the output data is generated, that no portion of the output data corresponds to a first type of label.   
     
     
         14 . The method of  claim 1 , wherein the first data comprises a stream of data items generated by the component of the data processing pipeline in sequence, wherein applying the first data as an input to the machine learning model to generate output data further comprises:
 for each data item of the stream of data items in sequence, applying the respective data item as an input to the machine learning model to generate a portion of the output data;   determining, a first period of time after an initial portion of the output data is generated, that no portion of the output data corresponds to a first type of label; and   stopping application of the stream of data items as an input to the machine learning model.   
     
     
         15 . The method of  claim 1 , wherein the sampling of the one or more different types of data output by the machine learning model comprises a sampling of one or more different label types found in the output data. 
     
     
         16 . A system, comprising:
 one or more data stores including computer-executable instructions; and   one or more processors configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:
 provide a user interface depicting a graph representing a data processing pipeline; 
 receive, via the user interface, a request to activate a preview mode in association with a machine learning model; 
 obtain first data generated by a component of the data processing pipeline interconnected with the machine learning model; 
 apply the first data as an input to the machine learning model to generate output data; and 
 cause the user interface to display a preview of a portion of the output data that comprises a sampling of one or more different types of data output by the machine learning model. 
   
     
     
         17 . The system of  claim 16 , wherein execution of the computer-executable instructions further causes the system to cause the user interface to display the preview without writing the output data to at least one destination specified by the graph. 
     
     
         18 . The system of  claim 16 , wherein execution of the computer-executable instructions further causes the system to transmit an abstract syntax tree (AST) of the data processing pipeline to an intake system, wherein the intake system produces an augmented AST by causing a function of the graph that writes to an external database to drop received data instead of writing the received data to the external database and by adding a preview node to the graph in association with the machine learning model, wherein the intake system runs a job using the augmented AST that results in the first data being transmitted to the preview node, and wherein execution of the computer-executable instructions further causes the system to apply, by the preview node, the first data as an input to the machine learning model to generate output data. 
     
     
         19 . A non-transitory computer-readable medium comprising instructions executable by a computing system that, when executed, cause the computing system to:
 provide a user interface depicting a graph representing a data processing pipeline;   receive, via the user interface, a request to activate a preview mode in association with a machine learning model;   obtain first data generated by a component of the data processing pipeline interconnected with the machine learning model;   apply the first data as an input to the machine learning model to generate output data; and   cause the user interface to display a preview of a portion of the output data that comprises a sampling of one or more different types of data output by the machine learning model.   
     
     
         20 . The non-transitory computer-readable medium of  claim 19 , wherein the instructions, when executed, further cause the computing system to cause the user interface to display the preview without writing the output data to at least one destination specified by the graph.