Anomaly classification, analytics and resolution based on annotated event logs

Abstract

Operational event loggings and operational alarm productions within a running multiserver data processing system are automatically and repeatedly sampled and co-associated with one another so as to build annotated logs that can be used by post-process analytics for filling in mappings thereof into an anomalies versus parameters mapping space and for keeping track of unusual changes in the mappings or their rates where the unusual changes can be indicative of emerging new problems of significance within the system.

1 .- 20 . (canceled) 
     
     
         21 . A machine-implemented method comprising:
 (a) automatically first determining whether newly emerging first non-routine anomalies are developing within a first section of a data processing system having at least first and second sections, each of the first and second sections including a respective behaviors logging subsystem configured to automatically log monitored behaviors within the respective section and a respective section alarming subsystem configured to automatically generate alarms for alarm worthy events within the respective section, the first determining including automatically repeatedly building a first annotated log for the first section, the first annotated log providing logical co-associations between logged behaviors produced by the respective section alarming subsystem of the first section and contemporaneously generated alarms generated by the respective section alarming subsystem of the first section;   (b) automatically second determining whether newly emerging second non-routine anomalies are developing within the second section of the data processing system, the second determining including automatically repeatedly building a second annotated log for the second section, the second annotated log providing logical co-associations between logged behaviors produced by the respective section alarming subsystem of the second section and contemporaneously generated alarms generated by the respective section alarming subsystem of the second section;   (c) automatically third determining from the first and second determinings whether the first and second non-routine anomalies develop within a same specified time frame; and   (d) in response to the third determining indicating development within the same specified time frame of the first and second non-routine anomalies, automatically identifying the first and second sections as locations in the data processing system where cotemporaneous and multi-sectional non-routine anomalies are emerging and automatically generating an alarm indicating that the emerging multi-sectional non-routine anomalies constitute a more widespread problem than just anomalous behaviors in the first and second sections individually.   
     
     
         22 . The method of  claim 21  wherein:
 each of the first and second sections further comprises respective locally intercoupled resources including one or more local data processing units and one or more local data storage units; and 
 each of the first and second automatic determinings of whether newly emerging non-routine anomalies are developing comprises for each of the respective sections, using the respective annotated log to automatically repeatedly map into a respective anomalies versus parameters mapping space, sample point indicators indicative of respective coordinates in the mapping space corresponding to plural parameters associated with each generating and non-generating of alarms by the respective section alarming subsystem and corresponding to the temporally co-associated, recently logged behaviors of the respective section. 
 
     
     
         23 . The method of  claim 22  wherein:
 at least one of the parameters in a respective anomalies versus parameters mapping space of a respective section represents a performance metric of at least one of the local data processing units and/or the local data storage units of the respective section. 
 
     
     
         24 . The method of  claim 22  wherein:
 the automatically repeated mapping of the sample point indicators into a respective anomalies versus parameters mapping space includes mapping as alarmed sample points (ASP's) entries in the respective annotated log for which both an alarm was generated by the respective section alarming subsystem and one or more cotemporaneous events were logged by the respective behaviors logging subsystem. 
 
     
     
         25 . The method of  claim 24  wherein:
 the automatically repeated mapping of the sample point indicators into a respective anomalies versus parameters mapping space includes mapping as non-alarmed sample points (NASP's) entries in the respective annotated log for which an alarm was not generated by the respective section alarming subsystem and one or more cotemporaneous events were logged by the respective behaviors logging subsystem. 
 
     
     
         26 . The method of  claim 25  wherein:
 the automatic first and second determining of whether respective newly emerging non-routine anomalies are developing within the respective section of the data processing system respectively include classifying regions in the respective anomalies versus parameters mapping space populated by NASP's as regions in which ASP's do not routinely occur. 
 
     
     
         27 . The method of  claim 26  wherein:
 the automatic first and second determining of whether respective newly emerging non-routine anomalies are developing within the respective section of the data processing system respectively include identifying as newly emerging non-routine anomalies those ASP's that map into a region previously classified as one in which ASP's do not routinely occur. 
 
     
     
         28 . The method of  claim 21  and further comprising:
 using at least one the respective annotated logs of the respective first and second sections for creating at least one of respective behavior mimicking models of the first and second section alarming subsystems, the created at least one of the respective behavior mimicking models having accessible internal logic structures configured to mimic output behaviors of the corresponding at least one of the first and second section alarming subsystems; 
 for a specified time period during the running of the at least one of the first and second sections, comparing alarms generated by the created at least one of respective behavior mimicking models with alarms generated by the respective at least one of the first and second section alarming sub systems; 
 in response to detection of difference by said comparing step, modifying the respective internal logic structures of the corresponding at least one of the respective behavior mimicking models so as to reduce difference in subsequent time periods; and 
 automatically repeating said comparing and modifying steps for the subsequent time periods. 
 
     
     
         29 . The method of  claim 28  wherein:
 the modifying step includes changing a subset of input parameters that the at least one of the respective behavior mimicking models uses as its input parameters; and 
 the changing of the subset of input parameters is responsive to automatically repeated updates made to the corresponding at least one of the respective annotated logs of the respective first and second sections. 
 
     
     
         30 . The method of  claim 22  wherein:
 the respective anomalies versus parameters spaces of the first and second sections respectively defined in a database storing corresponding first and second data representing alarmed sample points (ASP's) of the first and second sections as points within corresponding first and second multi-parameter coordinate spaces where each respective alarmed sample point (ASP) of the first and second sections respectively correlates to one or more of the temporally corresponding generatings of alarms by the corresponding one of the first and second section alarming subsystems; and 
 at least one of parameter axes of the anomalies versus parameters second mapping space corresponds to one of the parameter axes of the anomalies versus parameters first mapping space such that co-emergence within said same specified time frame of respective newly emerging non-routine anomalies of the first and second sections can be cross-correlated to one another as mapped along each of the corresponding parameter axes of the first and second mapping spaces. 
 
     
     
         31 . The method of  claim 30  wherein:
 the anomalies versus parameters mapping spaces are respectively further defined in the database by stored second data representing non-alarmed sample points (NASP's) of the first and second sections as points within the respective first and second multi-parameters coordinate spaces where each non-alarmed sample point (NASP) correlates to an event logging time when there are no temporally corresponding generatings of alarms by the respective one of the first and second section alarming subsystems. 
 
     
     
         32 . The method of  claim 22  wherein:
 in addition to its respective one or more local data processing units and its respective one or more local data storage units, at least one of the first and second sections includes a corresponding data input/output communicating unit; 
 the recently logged behaviors of the respective generated log of the at least one of the first and second sections includes a data processing rate of at least one of the respective local data processing units of the respective section, a data access rate of at least one of the respective local data storage units of the respective section and a data communicating rate of the first data input/output communicating unit of the respective section. 
 
     
     
         33 . The method of  claim 21  and further comprising:
 automatically repeatedly searching for cross correlations between event parameters and non-routine alarm occurrences in the respective annotated logs of the first and second sections. 
 
     
     
         34 . The method of  claim 33  and further comprising:
 building a knowledge database based on found cross correlations between event parameters and non-routine alarm occurrences in the respective annotated logs of first and second sections. 
 
     
     
         35 . The method of  claim 28  wherein the data processing system has a hierarchical structure composed of plural parent sections and respective sections within the parent sections, the first and second sections belonging to a first parent section, the method further comprising:
 running a third section of a second parent section within the data processing system where the running third section includes as its respective section alarming subsystem, a third section alarming subsystem and includes as its respective section behaviors logging subsystem, a third section behaviors logging subsystem, the third section alarming subsystem being configured to generate alarms for non-catastrophic alarm-worthy events detected within the third section, the third section behaviors logging subsystem being configured to generate a log of monitored behaviors within the third section; 
 logically co-associating recently logged behaviors of the generated log produced by the third section behaviors logging subsystem with substantially cotemporaneous alarms generated by the third section alarming subsystem; 
 building a third annotated log comprised of the logically co-associated logged behaviors and the substantially cotemporaneous alarms of the third section; 
 using the third annotated log of the respective third section to create a corresponding third behavior mimicking model of the third section alarming subsystem, the created third behavior mimicking model having accessible internal logic structures configured to mimic output behaviors of the third section alarming subsystem; 
 for the specified time frame and during the running of the third section, comparing alarms generated by the created third behavior mimicking model with alarms generated by the corresponding third section alarming subsystem; 
 in response to detection of differences by said comparing step for the third section, modifying the respective internal logic structures of the corresponding third behavior mimicking model so as to reduce future differences; 
 automatically repeating said comparing and modifying steps for subsequent time frames for the third behavior mimicking model; and 
 building a knowledge database over said subsequent time frames where the over-time built knowledge database provides insights as to operations of the third section alarming subsystem of the second parent section based on access to the accessible internal logic structures of the corresponding third behavior mimicking model. 
 
     
     
         36 . The method of  claim 21  and further comprising:
 merging the annotated logs of the sections that represent hierarchical children of a first parent section of the data processing system to thereby form a first parent annotated log; 
 merging the annotated logs of the sections that represent hierarchical children of a second parent section of the data processing system to thereby form a second parent annotated log; 
 automatically repeatedly searching for cross correlations between event parameters and non-routine alarm occurrences in the respective first and second parent annotated logs of the first and second parent sections. 
 
     
     
         37 . The method of  claim 36  and further comprising:
 building a knowledge database based on found cross correlations between event parameters and non-routine alarm occurrences in the respective first and second parent annotated logs of the first and second parent sections. 
 
     
     
         38 . A machine-implemented method of developing behavior mimicking and internals-accessible models of respective pre-configured alarming subsystems of a respective sections of a data processing system having a hierarchical structure composed of plural parent sections and respective sections within the parent sections, wherein each section of a respective parent section comprises locally intercoupled resources including one or more local data processing units and one or more local data storage units, at least a respective one of the sections further comprising a respective section behaviors logging subsystem configured to automatically log monitored behaviors within the respective section and a respective section alarming subsystem configured to automatically generate alarms for alarm worthy events within the respective section, the respective alarming subsystem not necessarily having internals that are easily accessible for determining why the respective alarming subsystem did or did not generate an alarm for a given event within the respective section, the method comprising:
 running a first section of a respective first parent section within the data processing system where the running first section includes a first section alarming subsystem as its respective section alarming subsystem and includes a first section behaviors logging subsystem as its respective section behaviors logging subsystem, the first section alarming subsystem being configured to generate alarms for non-catastrophic alarm-worthy events detected within the first section, the first section behaviors logging subsystem being configured to generate a log of monitored behaviors within the first section;   logically co-associating recently logged behaviors of the generated log produced by the first section behaviors logging subsystem with substantially cotemporaneous alarms generated by the first section alarming subsystem;   building a first annotated log comprised of the logically co-associated logged behaviors and the substantially cotemporaneous alarms of the first section;   using the first annotated log of the respective first section to create a corresponding first behavior mimicking model of the first section alarming subsystem, the created first behavior mimicking model having accessible internal logic structures configured to mimic output behaviors of the first section alarming subsystem, the accessible internal logic structures being configured to allow for determining why the first behavior mimicking model did or did not generate an alarm for a given event within the first section;   for a specified time frame during the running of the first section, comparing alarms generated by the created first behavior mimicking model with alarms generated by the corresponding first section alarming subsystem;   in response to detection of a difference by said comparing step, modifying the respective internal logic structures of the corresponding first behavior mimicking model so as to reduce future differences;   automatically repeating said comparing and modifying steps for subsequent time frames; and   building a knowledge database over said subsequent time frames where the over-time built knowledge database provides insights as to operations of the first section alarming subsystem based on access to the accessible internal logic structures of the corresponding first behavior mimicking model.   
     
     
         39 . The method of  claim 38  and further comprising:
 concurrently running a second section of the first parent section within the data processing system where the running second section includes a second section alarming subsystem as its respective section alarming subsystem, and includes a second section behaviors logging subsystem as its respective section behaviors logging subsystem, the second section alarming subsystem being configured to generate alarms for non-catastrophic alarm-worthy events detected within the second section, the second section behaviors logging subsystem being configured to generate a log of monitored behaviors within the second section; 
 logically co-associating recently logged behaviors of the generated log produced by the second section behaviors logging subsystem with substantially cotemporaneous alarms generated by the second section alarming subsystem; 
 building a second annotated log comprised of the logically co-associated logged behaviors and the substantially cotemporaneous alarms of the second section; 
 using the second annotated log of the respective second section to create a corresponding second behavior mimicking model of the second section alarming subsystem, the created second behavior mimicking model having accessible internal logic structures configured to mimic output behaviors of the second section alarming subsystem; 
 for a specified time frame during the running of the second section, respectively comparing alarms generated by the created second behavior mimicking model with alarms generated by the corresponding second section alarming subsystem; 
 in response to detection of a corresponding difference by said respective comparing step, modifying the respective internal logic structures of the corresponding second behavior mimicking model so as to reduce future differences; 
 automatically repeating said respective comparing and modifying steps for subsequent time frames for the second behavior mimicking model; and 
 building a knowledge database over said subsequent time frames where the over-time built knowledge database provides insights as to operations of the second section alarming subsystem based on access to the accessible internal logic structures of the corresponding second behavior mimicking model. 
 
     
     
         40 . A data processing system configured to deal with emerging non-routine anomalies within one or more of plural sections of the data processing system, the emerging non-routine anomalies developing in one or the other of localized portions of the data processing system or on a more widespread basis and not being catastrophic failures, the data processing system being subdivided into a plurality of parent sections with each parent section comprising respective plural sections, each section having locally intercoupled resources including one or more local data processing units and one or more local data storage units, wherein at least one respective section of a respective two or more of the plural parent sections each respectively includes a respective section behaviors logging subsystem configured to automatically log monitored behaviors within the respective section and to generate a respective local log and each of the at least one respective sections respectively includes a respective section alarming subsystem configured to automatically generate alarms for alarm worthy events within the respective section, the data processing system further comprising:
 an annotated logs storing database storing one or more respective annotated logs that respectively indicate correlations for respective ones of the system sections between recently logged behaviors of the respective system sections as recently recorded in the respective local logs of the respective sections and temporally correlated generatings and non-generatings of alarms by the respective section alarming subsystems of the respective system sections;   an annotated logs builder, coupled to the database and configured to automatically repeatedly for respective ones of the sections, add to the respective stored and annotated logs of the respective sections additional samples of temporal correlations between recently logged behaviors logged in the respective local logs and temporally corresponding generatings and non-generatings of alarms by the respective section alarming subsystems of the respective sections; and   a post-process analytics portion of the data processing system that is operatively coupled to respective ones of the annotated logs stored in the database for the respective sections and is configured to automatically repeatedly map into respective anomalies versus parameters mapping spaces of respective ones of the system sections, sample point indicators indicative of respective coordinates in the respective mapping space corresponding to plural parameters associated with each generating and non-generating of alarms by the respective section alarming subsystem of the respective sections and corresponding to temporal correlated, recently logged behaviors of the respective local log produced by the section behaviors logging subsystem of that respective section;   wherein the post-process analytics portion is configured to flag out abnormal changes over time in the automatically repeatedly made mappings of the sample point indicators into the respective anomalies versus parameters mapping spaces, where the flagged out abnormal changes include those representing emerging non-routine anomalies that are not catastrophic failures; and   wherein the post-process analytics portion is configured to flag out concurrent development for two or more respective sections within one or within plural ones of the parent sections of respective newly emerging non-routine anomalies where such concurrent development for the two or more respective sections is indicative of emergence of non-routine anomalies on a more widespread basis than just separately in individualized ones of the sections.