Probability-distribution-based log-file analysis

Abstract

The current document is directed to systems, and methods incorporated within the systems, that carry out probability-distribution-based analysis of log-file entries. A monitoring subsystem within a distributed computer system uses probability-distribution-based analysis of log-file entries to detect changes in the state of the distributed computer system. A log-file-analysis subsystem within a distributed computer system uses probability-distribution-based analysis of log-file entries to identify subsets of log-file entries that predict anomalies and impending problems in the distributed computer system. In many implementations, a numerical comparison of probability distributions of log-file-entry types is used to detect state changes in the distributed computer system.

1 . A log-file analysis system comprising:
 one or more processors;   one or more memories; and   computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the log-file analysis system to
 generate, for a first set of log-file entries, each of which is associated with an event type, a first probability distribution of all or a subset of the event types in the first set of log-file entries, 
 generate, for a second set of log-file entries, each of which is associated with an event type, a second probability distribution of all or a subset of the event types in the second set of log-file entries, 
 generate a numeric divergence metric from the first and second probability distributions, and 
 when the numeric divergence metric is greater than a threshold value, generate one of a displayed indication, alarm, event, or other physical-state transformation to indicate that the first and second sets of log-file entries exhibit more than a threshold level of difference. 
   
     
     
         2 . The log-file analysis system of  claim 1  wherein the log-file analysis system generates a probability distribution of all or a subset of the event types in a set of log-file entries by:
 determining and storing the count of log-file entries for each event type or for each event type within the subset of event types; 
 determining the total number N of log-file entries in the set of log-file entries; and 
 determining and storing, in memory, the probability for each event type or each event type within the set of event types as the count of log-file entries for the event type divided by N. 
 
     
     
         3 . The log-file analysis system of  claim 1  wherein numeric divergence metric is the Jensen-Shannon numeric divergence metric. 
     
     
         4 . A log-file analysis subsystem within a computer system having one or more processors, one or more memories, and computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the log-file analysis system to monitor a state of a computer system by repeatedly:
 generating, for one or more log files, each having multiple entries that are each associated with an event type, and for a time interval, a probability distribution of one or more event types in the one or more log files that represents a state of a monitored computer system;   storing the generated probability distribution in association with an indication of the time interval;   computing a numeric divergence metric from the two most recently generated and stored probability distributions; and   when numeric divergence metric is greater than a threshold value, raising an alarm to indicate a significant system-state change.   
     
     
         5 . The log-file analysis system of  claim 4  wherein the log-file analysis system generates a probability distribution of all or a subset of the event types in one or more log files for a time interval by:
 selecting a set of log-file entries represents events from the one or more log files; 
 determining and storing the count of log-file entries for each event type or each event type within the subset of event types; 
 determining a total number N of log-file entries in the selected set of log-file entries; and 
 determining and storing, in memory, a probability for each event type or each event type within the subset of event types as the count of log-file entries for the event type divided by N. 
 
     
     
         6 . The log-file analysis subsystem of  claim 4  wherein the numeric divergence metric is computed after generating and storing each probability distribution following generation and storing of an initial set of probability distributions. 
     
     
         7 . The log-file analysis subsystem of  claim 4  wherein the numeric divergence metric is the Jensen-Shannon numeric divergence metric. 
     
     
         8 . The log-file analysis subsystem of  claim 4  monitoring the state of the computer system by the log-file analysis system further includes:
 using the stored probability distributions collected over a first time interval spanning multiple shorter, secondary time intervals to generate a typical probability distribution for each of a set of time intervals selected from the multiple shorter, secondary time intervals; and 
 at subsequent secondary time intervals,
 generating a probability distribution for the event types of log entries selected from the most recently completed secondary time interval, 
 computing a Jensen-Shannon numeric divergence metric for the probability distribution generated from the most recently completed secondary time interval and the typical probability distribution for the most recently completed secondary time interval, and 
 when the Jensen-Shannon numeric divergence metric is greater than a threshold value, raising an alarm to indicate a system-state change. 
 
 
     
     
         9 . The log-file analysis subsystem of  claim 4  wherein the numeric divergence metric is the Jensen-Shannon divergence metric. 
     
     
         10 . The log-file analysis subsystem of  claim 4  wherein monitoring the state of the computer system by the log-file analysis system further includes:
 for each of a number of different subsets of the event types for which the log-file analysis subsystem has generated and stored probability distributions for different time intervals,
 computing a Jensen-Shannon numeric divergence metric for the probability distributions for different pairs of time intervals, and 
 computing a measure of the variance of the Jensen-Shannon divergence metrics computed for the probability distributions for different pairs of the time intervals; and 
 
 selecting, as a basis for a monitoring fingerprint, a subset of the event types having the greatest computed variance. 
 
     
     
         11 . The log-file analysis subsystem of  claim 4  wherein the numeric divergence metric is the Jensen-Shannon divergence metric. 
     
     
         12 . A method that monitors a state of a distributed computer system that includes multiple, network interconnected discrete computer systems, each having one or more processors, one or more memories, and one or more data-storage devices, one or more of the discrete computer systems, the method comprising:
 repeatedly
 generating, for one or more log files, each having multiple entries that are each associated with an event type, a probability distribution of all or a subset of the event types in the one or more log files for a time interval to represent the state of a monitored computer system for the time interval, 
 storing the generated probability distribution in association with an indication of the time interval in one or more of one or more memories and/or data-storage devices, 
 computing a numeric divergence metric from the two most recently generated and stored probability distributions; and 
 when numeric divergence metric is greater than a threshold value, raising an alarm to indicate a system-state change. 
   
     
     
         13 . The method of  claim 12  wherein generating a probability distribution of all or a subset of the event types in one or more log files for a time interval further includes:
 selecting a set of log-file entries represents events from the one or more log files; 
 determining and storing a count of log-file entries for each event type or each event type within the subset of event types; 
 determining a total number N of log-file entries in the selected set of log-file entries; and 
 determining and storing, in memory, the probability for each event type or each event type within the subset of event types as the count of log-file entries for the event type divided by N. 
 
     
     
         14 . The method of  claim 12  wherein the numeric divergence metric is generated after generating and storing each probability distribution following generation and storing of an initial set of probability distributions. 
     
     
         15 . The method of  claim 14  wherein the numeric divergence metric is the Jensen-Shannon numeric divergence metric. 
     
     
         16 . The method of  claim 12  further including:
 using the stored probability distributions collected over a first time interval spanning multiple shorter, secondary time intervals to generate a typical probability distribution for each of a set of time intervals selected from the multiple shorter, secondary time intervals; and 
 at subsequent secondary time intervals,
 generating a probability distribution for the event types of log entries selected from the most recently completed secondary time interval, 
 computing a numeric divergence metric for the probability distribution generated from the most recently completed secondary time interval and the typical probability distribution for the most recently completed secondary time interval, and 
 when the numeric divergence metric is greater than a threshold value, raising an alarm to indicate a system-state change. 
 
 
     
     
         17 . The method of  claim 16  wherein the numeric divergence metric is the Jensen-Shannon numeric divergence metric. 
     
     
         18 . The method of  claim 12  further including:
 for each of a number of different subsets of the event types for which the log-file analysis subsystem has generated and stored probability distributions for different time intervals,
 computing a numeric divergence metric for the probability distributions for different pairs of time intervals, and 
 computing a measure of the variance of the numeric divergence metrics computed for the probability distributions for different pairs of the time intervals; and 
 
 selecting, as a basis for a monitoring fingerprint, a subset of the event types having the greatest computed variance. 
 
     
     
         19 . The method of  claim 18  wherein the numeric divergence metric is the Jensen-Shannon numeric divergence metric. 
     
     
         20 . A computer-readable device that stores a set of computer instructions that, when executed on one or more processors of a computer system that additionally includes one or more memories, controls the computer system to:
 generate, for a first set of log-file entries, each of which is associated with an event type, a first probability distribution of all or a subset of the event types in the first set of log-file entries,   generate, for a second set of log-file entries, each of which is associated with an event type, a second probability distribution of all or a subset of the event types in the second set of log-file entries,   generate a numeric divergence metric from the first and second probability distributions, and   when the numeric divergence metric is greater than a threshold value, generate one of a displayed indication, alarm, event, or other physical-state transformation to indicate that the first and second sets of log-file entries exhibit more than a threshold level of difference.