Methods and systems for intelligent sampling of normal and erroneous application traces

Abstract

Computer-implemented methods and systems described herein perform intelligent sampling of application traces generated by an application. Computer-implemented methods and systems determine different sampling rates based on frequency of occurrence of normal traces and erroneous traces of the application. The sampling rates for low frequency normal and erroneous traces are larger than the sampling rates for high frequency normal and erroneous traces. The relatively larger sampling rates for low frequency trace ensures that low frequency traces are sampled in sufficient numbers and are not passed over during sampling of the application traces. The sampled normal and erroneous traces are stored in a data storage device.

1 . A method stored in one or more data storage devices and executed using one or more processors of a computer system for sampling a set of traces of an application executed in a distributed computing system, the method comprising:
 retrieving a set of trace data associated with the application from a data storage device;   determining sampling rates for sampling normal traces in the set and for sampling erroneous traces in the set, wherein the different sampling rates are inversely proportional to the frequency of occurrence of the normal traces and erroneous traces;   sampling the traces using the sampling rates to obtain sampled normal traces and sampled erroneous traces, wherein less frequently occurring normal traces are sampled at higher sampling rates than more frequently occurring normal traces and less frequently occurring erroneous traces are sampled at higher sampling rates than more frequently occurring erroneous traces; and   storing the sampled traces in a data storage device.   
     
     
         2 . The method of  claim 1  wherein determining the sampling rates comprises:
 sorting the traces according to trace type to obtain one or more groups of traces, each group of traces having a different associated trace type; and 
 for each group of traces.
 partitioning the group of traces into normal traces and erroneous traces, 
 determining a frequency of occurrence of normal traces in the group, 
 determining a frequency of occurrence of erroneous traces in the group, 
 constructing a normal trace histogram of the normal traces, 
 constructing an erroneous trace histogram of the erroneous traces, 
 determining a frequency of occurrence of normal traces in each bin of the normal trace histogram, 
 determining a frequency of occurrence of normal traces in each bin of the erroneous trace histogram, 
 determining a normal hybrid sampling rate for each bin of the normal histogram based on the frequency of occurrence of normal traces in each bin, the frequency of occurrence of the normal traces, and 
 determining an erroneous hybrid sampling rate for each bin of the normal histogram based on the frequency of occurrence of normal traces in each bin, the frequency of occurrence of the normal traces. 
 
 
     
     
         3 . The method of  claim 1  wherein determining the sampling rates comprises:
 sorting the traces according to trace type to obtain one or more groups of traces, each group of traces having a different associated trace type; 
 receiving a sampling level via a graphical user interface; 
 determining a trace-type sampling parameter based on the user-selected sampling level; and 
 for each group of traces,
 partitioning the group of traces into normal traces and erroneous traces; 
 determining a frequency of occurrence of normal traces in the group of traces, 
 determining a normal trace-type sampling rate based on the frequency of occurrence of normal traces, 
 determining a frequency of occurrence of erroneous traces in the group of traces, 
 determining an erroneous trace-type sampling rate based on the frequency of occurrence of erroneous traces. 
 
 
     
     
         4 . The method of  claim 1  wherein determining the sampling rates comprises:
 constructing a histogram of traces based on the durations, each bin of the histogram corresponding to a time interval and containing traces with durations in the time interval; 
 determining a frequency of occurrence of normal traces in each bin of the histogram; 
 for each bin of the histogram,
 determining a duration-sampling rate based on the frequency of occurrence of traces in the bin and the duration-sampling parameter: 
 
 determining a frequency of occurrence of traces in a lower bin; 
 computing a short duration sampling rate from the frequency of occurrence of traces in the lower bin; 
 determining a frequency of occurrence of traces in an upper bin; and 
 computing a long duration sampling rate from the frequency of occurrence of traces in the upper bin. 
 
     
     
         5 . The method of  claim 1  wherein determining the sampling rates comprises:
 partitioning the set of trace data into normal traces and erroneous traces: 
 determining a frequency of occurrence of the normal traces: 
 determining a normal trace sampling rate based on the frequency of occurrence of the normal traces; 
 determining a frequency of occurrence of the erroneous traces; and 
 determining a erroneous trace sampling rate based on the frequency of occurrence of the erroneous traces. 
 
     
     
         6 . The method of  claim 1  wherein determining the sampling rates comprises:
 partitioning the set of trace data into normal traces and erroneous traces; 
 determining a frequency of occurrence of the normal traces; 
 determining a frequency of occurrence of the erroneous traces; and 
 determining a normal trace sampling rate based on the frequency of occurrence of the normal traces, frequency of occurrence of the erroneous traces, an overall sampling rate, and erroneous trace sampling rate. 
 
     
     
         7 . The method of  claim 1  wherein sampling the traces using the sampling rates comprises sampling normal traces with a normal trace sampling rate, wherein the normal trace sampling rate is inversely proportional to a frequency of occurrence of the normal traces. 
     
     
         8 . The method of  claim 1  wherein sampling the traces using the sampling rates comprises sampling erroneous traces with an erroneous trace sampling rate, wherein the erroneous trace sampling rate is inversely proportional to a frequency of occurrence of the erroneous traces. 
     
     
         9 . The method of  claim 1  further comprises:
 performing troubleshooting on the sampled erroneous traces to identify a performance problem with the application; and 
 executing remedial measures to correct the performance problem. 
 
     
     
         10 . A computer system for sampling application traces of an application executed in a distributed computer system, the system comprising:
 one or more processors:   one or more data storage devices: and   machine-readable instructions stored in the one or more data storage devices that when executed using the one or more processors controls the system to perform operations comprising:
 retrieving a set of trace data associated with the application from a data storage device; 
 determining sampling rates for sampling normal traces in the set and for sampling erroneous traces in the set, wherein the different sampling rates are inversely proportional to the frequency of occurrence of the normal traces and erroneous traces: 
 sampling the traces using the sampling rates to obtain sampled normal traces and sampled erroneous traces, wherein less frequently occurring normal traces are sampled at higher sampling rates than more frequently occurring normal traces and less frequently occurring erroneous traces are sampled at higher sampling rates than more frequently occurring erroneous traces; and 
 storing the sampled traces in a data storage device. 
   
     
     
         11 . The computer system of  claim 10  wherein determining the sampling rates comprises:
 sorting the traces according to trace type to obtain one or more groups of traces, each group of traces having a different associated trace type; and 
 for each group of traces,
 partitioning the group of traces into normal traces and erroneous traces, 
 determining a frequency of occurrence of normal traces in the group, 
 determining a frequency of occurrence of erroneous traces in the group 
 constructing a normal trace histogram of the normal traces, 
 constructing an erroneous trace histogram of the erroneous traces, 
 determining a frequency of occurrence of normal traces in each bin of the normal trace histogram, 
 determining a frequency of occurrence of normal traces in each bin of the erroneous trace histogram, 
 determining a normal hybrid sampling rate for each bin of the normal histogram based on the frequency of occurrence of normal traces in each bin, the frequency of occurrence of the normal traces, and 
 determining an erroneous hybrid sampling rate for each bin of the normal histogram based on the frequent of occurrence of normal traces in each bin, the frequency of occurrence of the normal traces. 
 
 
     
     
         12 . The computer system of  claim 10  wherein determining the sampling rates comprises:
 sorting the traces according to trace type to obtain one or more groups of traces, each group of traces having a different associated trace type; 
 receiving a sampling level via a graphical user interface; 
 determining a trace-type sampling parameter based on the user-selected sampling level; and 
 for each group of traces,
 partitioning the group of traces into normal traces and erroneous traces; 
 determining a frequency of occurrence of normal traces in the group of traces, 
 determining a normal trace-type sampling rate based on the frequency of occurrence of normal traces, 
 determining a frequency of occurrence of erroneous traces in the group of traces, 
 determining an erroneous trace-type sampling rate based on the frequency of occurrence of erroneous traces. 
 
 
     
     
         13 . The computer system of  claim 10  wherein determining the sampling rates comprises:
 constructing a histogram of traces based on the durations, each bin of the histogram corresponding to a time interval and containing traces with durations in the time interval; 
 determining a frequency of occurrence of normal traces in each bin of the histogram: 
 for each bin of the histogram,
 determining a duration-sampling rate based on the frequency of occurrence of traces in the bin and the duration-sampling parameter; 
 
 determining a frequency of occurrence of traces in a lower bin; 
 computing a short duration sampling rate from the frequency of occurrence of traces in the lower bin; 
 determining a frequency of occurrence of traces in an upper bin; and 
 computing a long duration sampling rate from the frequency of occurrence of traces in the upper bin. 
 
     
     
         14 . The computer system of  claim 10  wherein determining the sampling rates comprises:
 partitioning the set of trace data into normal traces and erroneous traces; 
 determining a frequency of occurrence of the normal traces; 
 determining a normal trace sampling rate based on the frequency of occurrence of the normal traces; 
 determining a frequency of occurrence of the erroneous traces; and 
 determining a erroneous trace sampling rate based on the frequency of occurrence of the erroneous traces. 
 
     
     
         15 . The computer system of  claim 10  wherein determining the sampling rates comprises:
 partitioning the set of trace data into normal traces and erroneous traces; 
 determining a frequency of occurrence of the normal traces; 
 determining a frequency of occurrence of the erroneous traces; and 
 determining a normal trace sampling rate based on the frequency of occurrence of the normal traces, frequency of occurrence of the erroneous traces, an overall sampling rate, and erroneous trace sampling rate. 
 
     
     
         16 . The computer system of  claim 10  wherein sampling the traces using the sampling rates comprises sampling normal traces with a normal trace sampling rate, wherein the normal trace sampling rate is inversely proportional to a frequency of occurrence of the normal traces. 
     
     
         17 . The computer system of  claim 10  wherein sampling the traces using the sampling rates comprises sampling erroneous traces with an erroneous trace sampling rate, wherein the erroneous trace sampling rate is inversely proportional to a frequency of occurrence of the erroneous traces. 
     
     
         18 . The computer system of  claim 10  further comprises:
 performing troubleshooting on the sampled erroneous traces to identify a performance problem with the application; and 
 executing remedial measures to correct the performance problem. 
 
     
     
         19 . A non-transitory computer-readable medium encoded with machine-readable instructions that when executed by one or more processors of a computer system perform operations comprising:
 retrieving a set of trace data associated with the application from a data storage device;   determining sampling rates for sampling normal traces in the set and for sampling erroneous traces in the set, wherein the different sampling rates are inversely proportional to the frequency of occurrence of the normal traces and erroneous traces:   sampling the traces using the sampling rates to obtain sampled normal traces and sampled erroneous traces, wherein less frequently occurring normal traces are sampled at higher sampling rates than more frequently occurring normal traces and less frequently occurring erroneous traces are sampled at higher sampling rates than more frequently occurring erroneous traces; and   storing the sampled traces in a data storage device.   
     
     
         20 . The medium of  claim 19  wherein determining the sampling rates comprises:
 sorting the traces according to trace type to obtain one or more groups of traces, each group of traces having a different associated trace type; and 
 for each group of traces,
 partitioning the group of traces into normal traces and erroneous traces, 
 determining a frequency of occurrence of normal traces in the group, 
 determining a frequency of occurrence of erroneous traces in the group, 
 constructing a normal trace histogram of the normal traces, 
 constructing an erroneous trace histogram of the erroneous traces, 
 determining a frequency of occurrence of normal traces in each bin of the normal trace histogram, 
 determining a frequency of occurrence of normal traces in each bin of the erroneous trace histogram, 
 determining a normal hybrid sampling rate for each bin of the normal histogram based on the frequency of occurrence of normal traces in each bin, the frequency of occurrence of the normal traces, and 
 determining an erroneous hybrid sampling rate for each bin of the normal histogram based on the frequency of occurrence of normal traces in each bin, the frequency of occurrence of the normal traces. 
 
 
     
     
         21 . The medium of  claim 19  wherein determining the sampling rates comprises:
 sorting the traces according to trace type to obtain one or more groups of traces, each group of traces having a different associated trace type; 
 receiving a sampling level via a graphical user interface; 
 determining a trace-type sampling parameter based on the user-selected sampling level; and 
 for each group of traces,
 partitioning the group of traces into normal traces and erroneous traces; 
 determining a frequency of occurrence of normal traces in the group of traces, 
 determining a normal trace-type sampling rate based on the frequency of occurrence of normal traces, 
 determining a frequency of occurrence of erroneous traces in the group of traces, 
 determining an erroneous trace-type sampling rate based on the frequency of occurrence of erroneous traces. 
 
 
     
     
         22 . The medium of  claim 19  wherein determining the sampling rates comprises:
 constructing a histogram of traces based on the durations, each bin of the histogram corresponding to a time interval and containing traces with durations in the time interval; 
 determining a frequency of occurrence of normal traces in each bin of the histogram; 
 for each bin of the histogram,
 determining a duration-sampling rate based on the frequency of occurrence of traces in the bin and the duration-sampling parameter; 
 
 determining a frequency of occurrence of traces in a lower bin; 
 computing a short duration sampling rate from the frequency of occurrence of traces in the lower bin; 
 determining a frequency of occurrence of traces in an upper bin; and 
 computing a long duration sampling rate from the frequency of occurrence of traces in the upper bin. 
 
     
     
         23 . The medium of  claim 19  wherein determining the sampling rates comprises:
 partitioning the set of trace data into normal traces and erroneous traces; 
 determining a frequency of occurrence of the normal traces; 
 determining a normal trace sampling rate based on the frequency of occurrence of the normal traces; 
 determining a frequency of occurrence of the erroneous traces; and 
 determining a erroneous trace sampling rate based on the frequency of occurrence of the erroneous traces. 
 
     
     
         24 . The medium of  claim 19  wherein determining the sampling rates comprises:
 partitioning the set of trace data into normal traces and erroneous traces; 
 determining a frequency of occurrence of the normal traces; 
 determining a frequency of occurrence of the erroneous traces; and 
 determining a normal trace sampling rate based on the frequency of occurrence of the normal traces, frequency of occurrence of the erroneous traces, an overall sampling rate, and erroneous trace sampling rate. 
 
     
     
         25 . The medium of  claim 19  wherein sampling the traces using the sampling rates comprises sampling normal traces with a normal trace sampling rate, wherein the normal trace sampling rate is inversely proportional to a frequency of occurrence of the normal traces. 
     
     
         26 . The medium of  claim 19  wherein sampling the traces using the sampling rates comprises sampling erroneous traces with an erroneous trace sampling rate, wherein the erroneous trace sampling rate is inversely proportional to a frequency of occurrence of the erroneous traces. 
     
     
         27 . The medium of  claim 19  further comprises:
 performing troubleshooting on the sampled erroneous traces to identify a performance problem with the application; and 
 executing remedial measures to correct the performance problem.