Data ingestion by distributed-computing systems

Abstract

Techniques for data ingestion by a distributed-computing system are provided. In one embodiment, data received from one or more data sources is processed at a management node of the distributed-computing system. The data is generated by one or more components of an information processing system external to the distributed-computing system. The data is stored at the management plane. The management plane selects, from a plurality of data cores, a data core to ingest the data. The plurality of data cores are stored across one or more data plane containers of a data plane of the distributed-computing system. The management plane processes the data to generate one or more event records corresponding to the data. The one or more event records are sent to the selected data core, which causes the data plane to store the one or more event records in the selected data core.

What is claimed is: 
     
         1 . A method for data ingestion by a distributed-computing system implementing a management plane and a data plane, the method comprising:
 at the management plane:
 receiving data generated by one or more components of an information processing system; 
 storing the received data in a memory of the management plane; 
 upon the storing, providing to the information processing system a confirmation that the received data has been accepted; 
 storing a copy of the received data in each management node of a plurality of management nodes of the management plane; 
 processing the received data to generate one or more event records corresponding to one or more log events that occurred at the information processing system; and 
 sending the one or more event records to a data core of the data plane. 
   
     
     
         2 . The method of  claim 1 , wherein receiving the data comprises receiving log data representing one or more log events that occurred at the information processing system. 
     
     
         3 . The method of  claim 1 , further comprising:
 receiving, at the management plane, an ingestion request with the data, wherein the data is a payload of the ingestion request.   
     
     
         4 . The method of  claim 3 , wherein the data core is selected from among a plurality of data cores using information included in the received ingestion request. 
     
     
         5 . The method of  claim 1 , wherein the sending causes the data plane to store the one or more event records in the data core. 
     
     
         6 . The method of  claim 5 , further comprising:
 at the management plane:
 detecting a failure to store the one or more event records in the data core, wherein the data core is a first data core; 
 in response to detecting the failure, selecting a second data core to ingest the data; and 
 sending the one or more event records to the selected second data core, including causing the data plane to store the one or more event records in the selected second data core. 
   
     
     
         7 . The method of  claim 1 , further comprising:
 storing a back-up copy of the one or more event records in a back-up storage system external to the distributed-computing system; and   after storing the one or more event records in the data core and storing the back-up copy in the back-up storage system, removing the data from the management plane.   
     
     
         8 . The method of  claim 1 , further comprising selecting the data core from among a plurality of data cores, the selecting being performed using metadata obtained from the data plane. 
     
     
         9 . The method of  claim 8 , wherein, for each respective data core of the plurality of data cores, the metadata includes information regarding available storage capacity of the respective data core, current availability of the respective data core for ingesting data, or ingestion processing capacity of a data plane container storing the respective data core. 
     
     
         10 . The method of  claim 1 , further comprising selecting the data core, wherein the selected data core is stored in a first data plane container of one or more data plane containers, and wherein selecting the data core further comprises:
 determining, by the management plane, a current data ingestion rate of the first data plane container relative to one or more other data plane containers of the data plane, wherein the selected data core is selected based on the determined current data ingestion rate of the first data plane container relative to the one or more other data plane containers.   
     
     
         11 . The method of  claim 1 , wherein processing the data further comprises:
 decompressing the data; and   parsing the decompressed data to generate the one or more event records.   
     
     
         12 . The method of  claim 11 , wherein:
 parsing the decompressed data further comprises extracting, from the decompressed data, one or more fields for the one or more event records; and   sending the one or more event records to the data core further comprises causing the data plane to store the one or more fields in an index of the selected data core, the one or more fields stored in association with the one or more event records.   
     
     
         13 . The method of  claim 1 , wherein the providing is performed prior to completion of ingestion of the received data by a data platform. 
     
     
         14 . A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a distributed-computing system, the distributed-computing system implementing a management plane and a data plane, the one or more programs including instructions for:
 at the management plane:
 receiving data generated by one or more components of an information processing system; 
 storing the received data in a memory of the management plane; 
 upon the storing, providing to the information processing system a confirmation that the received data has been accepted; 
 storing a copy of the received data in each management node of a plurality of management nodes of the management plane; 
 processing the received data to generate one or more event records corresponding to one or more log events that occurred at the information processing system; and 
 sending the one or more event records to a data core of the data plane. 
   
     
     
         15 . The non-transitory computer-readable storage medium of  claim 14 , wherein the one or more programs further include instructions for:
 receiving, at the management plane, an ingestion request with the data, wherein the data is a payload of the ingestion request.   
     
     
         16 . The non-transitory computer-readable storage medium of  claim 14 , wherein the providing is performed prior to completion of ingestion of the received data by a data platform. 
     
     
         17 . The non-transitory computer-readable storage medium of  claim 14 , wherein the sending causes the data plane to store the one or more event records in the data core. 
     
     
         18 . A distributed-computing system for data ingestion, wherein the system implements a management plane and a data plane, the system comprising:
 one or more processors; and   memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:   at the management plane:
 receiving data generated by one or more components of an information processing system; 
 storing the received data in a memory of the management plane; 
 upon the storing, providing to the information processing system a confirmation that the received data has been accepted; 
 storing a copy of the received data in each management node of a plurality of management nodes of the management plane; 
 processing the received data to generate one or more event records corresponding to one or more log events that occurred at the information processing system; and 
   sending the one or more event records to a data core of the data plane.   
     
     
         19 . The system of  claim 18 , wherein the one or more programs further include instructions for:
 receiving, at the management plane, an ingestion request with the data, wherein the data is a payload of the ingestion request.   
     
     
         20 . The system of  claim 18 , wherein the providing is performed prior to completion of ingestion of the received data by a data platform.