Measuring and improving index quality in a distrubuted data system

Abstract

Embodiments described herein are directed to measuring and improving an index quality of a distributed data system. For example, various quality metrics are determined on a per partition basis of the distributed data system. Each of the quality metrics are indicative of a quality of a particular property of a partition. The quality metrics are aggregated to generate an overall index quality score, which provides a measure of the performance of the index. The index quality score is utilized to automatically detect an inefficiency of the index and automatically determine that certain index maintenance actions should be automatically performed to improve the performance of the index. Each quality metric may also be individually analyzed to determine which database property is affecting the performance of the index the most.

What is claimed is: 
     
         1 . A method, comprising:
 receiving, from a first compute node via a network, a first quality metric for a first partition of a database table maintained by a distributed database system, the first quality metric indicative of a quality of a first property of the first partition;   receiving, from a second compute node via the network, a second quality metric for a second partition of the database table, the second quality metric indicative of a quality of a second property of the second partition;   generating a global quality metric indicative of the performance of an index of the distributed database system based at least on the first quality metric and the second quality metric;   detecting an inefficiency with respect to the index based on the global quality metric meeting a condition with respect to a predetermined threshold;   determining a first unit of work required to improve the first clustering quality metric;   determining a second unit of work required to improve the second clustering quality metric; and   performing an action to alter a manner in which data is stored by the database table based on the first unit of work and the second unit of work.   
     
     
         2 . The method of  claim 1 , wherein the first quality metric comprises a number of rows of the first partition, the second quality metric comprises a number of rows of the second partition, and said generating the global quality metric comprises:
 determining, based on the first quality metric and the second quality metric, whether a distribution of the rows of the first partition and the rows of the second partition meets a second condition with respect to a second predetermined threshold;   responsive to determining that the distribution meets the second condition, maintaining a partitioning key for the first partition and the second partition; and   responsive to determining that the distribution does not meet the second condition, determining a different partitioning key for the first partition and the second partition.   
     
     
         3 . The method of  claim 2 , wherein said performing the action to alter the manner in which data is stored by the database table comprises partitioning the database table in accordance with the different partitioning key. 
     
     
         4 . The method of  claim 1 , wherein the first quality metric comprises a first data clustering quality metric based on a maximum number of first data files of the first partition that include a row having a particular clustering key of a plurality of first clustering keys, and the second quality metric comprises a second data clustering quality metric based on a maximum number of second data files of the second partition that include a row having a particular clustering key of a plurality of second clustering keys, and said generating the global quality metric comprises:
 designating a greatest data clustering quality metric among the first data clustering quality metric and the second data clustering quality metric as a global data clustering quality metric. 
 
     
     
         5 . The method of  claim 4 , wherein said performing the action to alter the manner in which data is stored by the database table comprises one of:
 re-clustering first data files each storing rows having different first clustering keys such that the rows of each re-clustered first data file have a single clustering key of the first clustering keys; or   re-clustering second data files each storing rows having different second clustering keys such that the rows of each re-clustered second data files have a single clustering key of the second clustering keys.   
     
     
         6 . The method of  claim 1 , wherein the first quality metric comprises a first data packing quality metric based on an average number of rows stored by first data files of the first partition and a predetermined ideal row count, and the second quality metric comprises a second data packing quality metric based on an average number of rows stored by second data files of the second partition and the predetermined ideal row count, and said generating the global quality metric comprises:
 generating a global data packing quality metric based on the first data packing metric and the second data packing metric.   
     
     
         7 . The method of  claim 6 , wherein said performing the action to alter the manner in which data is stored by the database table comprises performing at least one of:
 merging first data files storing fewer rows than the predetermined ideal row count; or   merging second data files storing fewer rows than the predetermined ideal row count.   
     
     
         8 . The method of  claim 1 , wherein the first quality metric comprises a first compaction quality metric based on a number of uncompressed rows maintained by first data files of the first partition, and the second quality metric comprises a second compaction quality metric based on a number of uncompressed rows maintained by second data files of the second partition, and said generating the global quality metric comprises:
 generating a global compaction quality metric based on a combination of the first compaction quality metric, the second compaction quality metric, and a predetermined compression threshold.   
     
     
         9 . The method of  claim 8 , wherein said performing the action to alter the manner in which data is stored by the database table comprises performing at least one of:
 compressing the uncompressed rows of the first data files; or   compressing the uncompressed rows of the second data files.   
     
     
         10 . The method of  claim 1 , wherein the first quality metric comprises a first fragmentation quality metric based on a number of rows marked for deletion in first data files of the first partition, and the second quality metric comprises a second fragmentation quality metric based on a number of rows marked for deletion in second data files of the second partition, and said generating the global quality metric comprises:
 generating a global fragmentation quality metric based on a combination of the first fragmentation quality metric, the second fragmentation quality metric, and a total number of rows in the first partition and the second partition.   
     
     
         11 . The method of  claim 10 , wherein said performing the action to alter the manner in which data is stored by the database table comprises performing at least one of:
 merging active rows from the first data files into a first new data file, and merging rows marked for deletion from the first data files into a second new data file; or   merging active rows from the second data files into a third new data file, and merging rows marked for deletion from the second data files into a fourth new data file.   
     
     
         12 . A system, comprising:
 a processor; and   a memory storing program code structured to cause the processor to:
 receive, from a first compute node via a network, a first quality metric for a first partition of a database table maintained by a distributed database system, the first quality metric indicative of a quality of a first property of the first partition; 
 receive, from a second compute node via the network, a second quality metric for a second partition of the database table, the second quality metric indicative of a quality of a second property of the second partition; 
 generate a global quality metric indicative of the performance of an index of the distributed database system based at least on the first quality metric and the second quality metric; 
 detect an inefficiency with respect to the index based on the global quality metric meeting a condition with respect to a predetermined threshold; 
 determine a first unit of work required to improve the first clustering quality metric; 
 determine a second unit of work required to improve the second clustering quality metric; and 
 perform an action to alter a manner in which data is stored by the database table based on the first unit of work and the second unit of work. 
   
     
     
         13 . The system of  claim 12 , wherein the first quality metric comprises a number of rows of the first partition, the second quality metric comprises a number of rows of the second partition, and, to generate the global quality metric, the program code is structured to cause the processor to:
 determine, based on the first quality metric and the second quality metric, whether a distribution of the rows of the first partition and the rows of the second partition meets a second condition with respect to a second predetermined threshold;   responsive to determining that the distribution meets the second condition, maintaining a partitioning key for the first partition and the second partition; and   responsive to determining that the distribution does not meet the second condition, determining a different partitioning key for the first partition and the second partition.   
     
     
         14 . The system of  claim 13 , wherein, to perform an action to alter a manner in which data is stored by the database table, the program code is structured to cause the processor to partition the database table in accordance with the different partitioning key. 
     
     
         15 . The system of  claim 12 , wherein the first quality metric comprises a first data clustering quality metric based on a maximum number of first data files of the first partition that include a row having a particular clustering key of a plurality of first clustering keys, the second quality metric comprises a second data clustering quality metric based on a maximum number of second data files of the second partition that include a row having a particular clustering key of a plurality of second clustering keys, and, to generate the global quality metric, the program code is structured to cause the processor to:
 designate a greatest data clustering quality metric among the first data clustering quality metric and the second data clustering quality metric as a global data clustering quality metric.   
     
     
         16 . The system of  claim 15 , wherein, to perform an action to alter a manner in which data is stored by the database table, the program code is structured to cause the processor to perform one of:
 re-cluster first data files each storing rows having different first clustering keys such that the rows of each re-clustered first data file have a single clustering key of the first clustering keys; or   re-cluster second data files each storing rows having different second clustering keys such that the rows of each re-clustered second data files have a single clustering key of the second clustering keys.   
     
     
         17 . The system of  claim 12 , wherein the first quality metric comprises a first data packing quality metric based on an average number of rows stored by first data files of the first partition and a predetermined ideal row count, the second quality metric comprises a second data packing quality metric based on an average number of rows stored by second data files of the second partition and the predetermined ideal row count, and, to generate the global quality metric, the program code is structured to cause the processor to:
 generate a global data packing quality metric based on the first data packing metric and the second data packing metric.   
     
     
         18 . The system of  claim 17 , wherein, to perform an action to alter a manner in which data is stored by the database table, the program code is structured to cause the processor to perform at least one of:
 merge first data files storing fewer rows than the predetermined ideal row count; or   merge second data files storing fewer rows than the predetermined ideal row count.   
     
     
         19 . The system of  claim 12 , wherein the first quality metric comprises a first compaction quality metric based on a number of uncompressed rows maintained by first data files of the first partition, the second quality metric comprises a second compaction quality metric based on a number of uncompressed rows maintained by second data files of the second partition, and, to generate the global quality metric, the program code is structured to cause the processor to:
 generate a global compaction quality metric based on a combination of the first compaction quality metric, the second compaction quality metric, and a predetermined compression threshold.   
     
     
         20 . A computer-readable storage medium comprising program instructions that, when executed by a processor, cause the processor to:
 receive, from a first compute node via a network, a first quality metric for a first partition of a database table maintained by a distributed database system, the first quality metric indicative of a quality of a first property of the first partition;   receive, from a second compute node via the network, a second quality metric for a second partition of the database table, the second quality metric indicative of a quality of a second property of the second partition;   generate a global quality metric indicative of the performance of an index of the distributed database system based at least on the first quality metric and the second quality metric;   detect an inefficiency with respect to the index based on the global quality metric meeting a condition with respect to a predetermined threshold;   determine a first unit of work required to improve the first clustering quality metric;   determine a second unit of work required to improve the second clustering quality metric; and   perform an action to alter a manner in which data is stored by the database table based on the first unit of work and the second unit of work.