Methods and systems that efficiently store metric data

Abstract

The current document is directed to methods and systems that collect metric data within computing facilities, including large data centers and cloud-computing facilities. In a described implementation, lower and higher metric-data-value thresholds are used to partition collected metric data into outlying metric data and inlying metric data. The inlying metric data is quantized to compress the inlying metric data and adjacent data points having the same quantized metric-data values are eliminated, to further compress the inlying metric data. The resulting compressed data includes original metric-data representations for outlier data points and compressed metric-data representations for inlier data points, providing accurate restored metric-data values for significant data points when compressed metric data is decompressed.

1 . A metric-data collection-and-storage subsystem within a distributed computer system, the metric-data collection-and-storage subsystem comprising:
 one or more processors:   one or more memories;   one or more data-storage devices;   one or more virtual machines instantiated by computer instructions stored in one or more of the one or more memories and executed by one or more of the one or more processors that together collect and store metric data by
 receiving a sequence of metric data points, 
 determining lower metric-data-value threshold and a higher metric-data-value threshold that partition the metric data points of the sequence of metric data points into lower-threshold-outlier metric data points, higher-threshold-outlier metric data points, and inlier metric data points, 
 determining a number n of quantization intervals; 
 storing lower-threshold-outlier metric data points in uncompressed form; 
 storing higher-threshold-outlier metric data points in uncompressed form; and 
 for each of the remaining inlier metric data points,
 determining a quantization interval for the inlier metric data point, and 
 when the determined quantization interval is not a quantization interval determined for a preceding inlier metric data point,
 storing the inlier metric data point in compressed form. 
 
 
   
     
     
         2 . The metric-data collection-and-storage subsystem of  claim 1  wherein each received metric data point is representable as a timestamp/metric-data-value pair. 
     
     
         3 . The metric-data collection-and-storage subsystem of  claim 2  wherein the n quantization intervals are defined by n+1 quantization threshold metric-data values that include the lower metric-data-value threshold as the first quantization threshold metric-data value and the higher metric-data-value threshold as the n th  quantization threshold metric-data value. 
     
     
         4 . The metric-data collection-and-storage subsystem of  claim 3  further including storing a representative metric-data value for each quantization interval, an indication of the number of lower-threshold-outlier metric data points, an indication of the number of higher-threshold-outlier metric data points, an indication of the number of stored inlier metric data points, and the number n. 
     
     
         5 . The metric-data collection-and-storage subsystem of  claim 4  wherein the quantization threshold metric-data values are equally spaced apart in the range of metric-data values defined by the lower metric-data-value threshold and the higher metric-data-value threshold. 
     
     
         6 . The metric-data collection-and-storage subsystem of  claim 4  wherein the representative metric-data value for a quantization interval is determined from the metric-data values of a number of metric data points within the quantization interval. 
     
     
         7 . The metric-data collection-and-storage subsystem of  claim 6  wherein the representative metric-data value for a quantization interval is determined as:
 an arithmetic mean of the metric-data values of a number of metric data points within the quantization interval; and 
 an arithmetic average of the metric-data values of a number of metric data points within the quantization interval. 
 
     
     
         8 . The metric-data collection-and-storage subsystem of  claim 4  wherein the compressed form of an inlier metric data point comprises a timestamp and an indication of a quantization interval to which the metric data point belongs. 
     
     
         9 . A physical data-storage device that stores a sequence of computer instructions that, when executed by one or more processors within one or more computer systems that each includes one or more processors, one or more memories, and one or more data-storage de ices control the one or more computer systems to:
 collect and store a time sequence of metric data by
 partitioning metric data points of the metric data into lower-threshold-outlier metric data points, higher-threshold-outlier metric data points, and inlier metric data points, 
 storing lower-threshold-outlier metric data points and higher-threshold-outlier metric data points in uncompressed form, and 
 storing the inlier metric data point in compressed form. 
   
     
     
         10 . The physical data-storage device of  claim 9  wherein each received metric data point is representable as a timestamp/metric-data-value pair. 
     
     
         11 . The physical data-storage device of  claim 10  wherein storing the inlier metric data point in compressed form further comprises:
 for each inlier metric data point,
 determining a quantization interval for the inlier metric data point, and 
 when the determined quantization interval is not a quantization interval determined for a preceding inlier metric data point,
 storing the inlier metric data point in compressed form. 
 
 
 
     
     
         12 . The physical data-storage device of  claim 11  wherein the compressed form of an inlier metric data point comprises a timestamp and an indication of a quantization interval to which the metric data point belongs. 
     
     
         13 . The physical data-storage device of  claim 11   wherein n quantization intervals are defined by n+1 quantization threshold metric-data values that include a lower metric-data-value threshold as the first quantization threshold metric-data value and a higher metric-data-value threshold as the n th  quantization threshold metric-data value;   wherein the lower-threshold-outlier metric data points have metric-data values less than a lowest-valued quantization threshold metric-data value; and   wherein the higher-threshold-outlier metric data points have metric-data values greater than a highest-valued quantization threshold metric-data value.   
     
     
         14 . The physical data-storage device of  claim 13  further including storing a representative metric-data value for each quantization interval, an indication of the number of lower-threshold-outlier metric data points, an indication of the number of higher-threshold-outlier metric data points, an indication of the number of stored inlier metric data points, and the number n. 
     
     
         15 . The physical data-storage device of  claim 9  wherein execution of the computer instructions additionally control the one or more computer systems to decompress stored metric data by:
 while unextracted metric data points remain in the container,
 determining a next time in a time sequence of metric data points generated by decompressing the stored metric data, 
 when an unextracted outlier metric data point corresponds to the next time,
 extracting the outlier metric data point and appending the extracted metric data point to the time sequence of metric data points, 
 
 when an unextracted compressed inlier metric data point corresponds to the next time,
 extracting the compressed inlier metric data point, decompressing the compressed inlier metric data point, and appending the decompressing the inlier metric data point to the time sequence of metric data points, and 
 
 when no unextracted inlier metric data point corresponds to the next time,
 appending a copy of the most recently decompressed inlier metric data point to the time sequence of metric data points.