Techniques for Data Log Processing, Retention, and Storage

Abstract

Techniques are disclosed relating to retaining a log entry in response to detection of a respective triggering event occurring within a computer network. This triggering event may result in a set of processes being performed. A computer system may determine a trace signature for the log entry. This trace signature may track information related to the set of processes. The computer system may compute, using the trace signature, a log retention value for the log entry. This log retention value may be computed using weight factors for ones of the set of processes. The computer system may retain the log entry within a log file according to a retention period that corresponds to the log retention value.

1 . (canceled) 
     
     
         2 . A method, comprising:
 receiving, by a computer system, a log entry to store within a log file;   using, by the computer system, a natural language programming (NLP) pipeline to categorize information included in the log entry;   using information about usage of currently retained log entries, determining, by the computer system, one or more patterns associated with log entries;   generating, by the computer system, a log retention value based on mapping log retention estimates to the categorized information and the one or more patterns; and   setting, by the computer system using the log retention value, a retention period for storing the log entry in the log file.   
     
     
         3 . The method of  claim 2 , further comprising:
 determining a trace signature for the log entry, the trace signature including information related to a triggering event and subsequent processes that are performed in response to the triggering event; and   using the information from the trace signature as input to the NLP pipeline.   
     
     
         4 . The method of  claim 3 , wherein information included in the trace signature further includes:
 indications of what triggered the subsequent processes; and   data used as inputs to the subsequent processes.   
     
     
         5 . The method of  claim 2 , wherein using the NLP pipeline includes:
 determining meanings of textual content included in the log entry; and   mapping the meanings to corresponding ones of a plurality of topics.   
     
     
         6 . The method of  claim 2 , wherein the determining of the one or more patterns associated with log entries includes:
 analyzing the information about usage of a subset of the currently retained log entries, the subset including currently retained log entries that are similar to the log entry; and   determining a particular usage pattern for the subset of currently retained log entries.   
     
     
         7 . The method of  claim 6 , further comprising:
 determining a set of retention maps using usage patterns from a plurality of subsets of retained log entries; and   updating the set of retention maps using the particular usage pattern.   
     
     
         8 . The method of  claim 7 , wherein generating the log retention value includes:
 identifying, using the categorized information from the log entry, a particular retention map of the set of retention maps; and   using the particular retention map to generate the log retention value.   
     
     
         9 . The method of  claim 2 , wherein generating the log retention value includes:
 determining an overall log retention value for the log entry; and   determining a respective semantic log retention value for one or more portions of the categorized information.   
     
     
         10 . The method of  claim 9 , further comprising using the overall log retention value and one or more of the semantic retention values to set the retention period for storing the log entry in the log file. 
     
     
         11 . A computer-readable, non-transient medium including instructions that when executed by a computer system within a computer network, cause the computer system to perform operations including:
 receiving a log entry to store within a log file;   using information included in the log entry, mapping the log entry to a subset of a plurality of topics;   selecting, using the information included in the log entry, a particular retention map from a set of retention maps;   generating a log retention value using the subset of topics and the particular retention map; and   setting, using the log retention value, a retention period for storing the log entry in the log file.   
     
     
         12 . The computer-readable medium of  claim 11 , wherein mapping the log entry to the subset of topics includes:
 using a natural language programming (NLP) pipeline to determine meanings of textual content included in the log entry; and   mapping the meanings to corresponding ones of the subset of topics.   
     
     
         13 . The computer-readable medium of  claim 11 , further comprising:
 determining a trace signature for the log entry, the trace signature including information related to a triggering event and subsequent processes that are performed in response to the triggering event; and   using the information from the trace signature to select the particular retention map.   
     
     
         14 . The computer-readable medium of  claim 11 , further comprising:
 analyzing the information about usage of a subset of currently retained log entries, the subset including currently retained log entries that are similar to the log entry; and   determining a particular usage pattern for the subset of currently retained log entries.   
     
     
         15 . The computer-readable medium of  claim 14 , further comprising updating the set of retention maps using the particular usage pattern. 
     
     
         16 . The computer-readable medium of  claim 11 , wherein generating the log retention value includes:
 determining a respective semantic log retention value for ones of the subset of topics; and   selecting a particular one of the semantic log retention values as the log retention value.   
     
     
         17 . A system comprising:
 a processor circuit; and   a memory circuit including instructions that when executed by processor circuit, cause the system to perform operations including:
 receiving a log entry to store within a log file; 
 using a natural language programming (NLP) pipeline to categorize information included in the log entry; 
 selecting, using the information included in the log entry, a particular retention map from a set of retention maps; 
 generating a log retention value using the categorized information and the particular retention map; and 
 setting, using the log retention value, a retention period for storing the log entry in the log file. 
   
     
     
         18 . The system of  claim 17 , wherein the operations further include:
 determining a trace signature for the log entry, the trace signature including information related to a triggering event and subsequent processes that are performed in response to the triggering event; and   using the information from the trace signature to select the particular retention map.   
     
     
         19 . The system of  claim 17 , wherein using the NLP pipeline includes:
 determining meanings of textual content in the log entry; and   mapping the meanings to corresponding ones of a plurality of topics.   
     
     
         20 . The system of  claim 17 , wherein the operations further include:
 using information about usage of currently retained log entries, determining one or more patterns associated with log entries; and   using the one or more patterns to update the set of retention maps.   
     
     
         21 . The system of  claim 17 , wherein generating the log retention value includes:
 determining one or more semantic log retention values using the categorized information; and   selecting a particular one of the semantic log retention values as the log retention value.