Increasing clustering of partitions of a table

Abstract

The subject technology creates one or more partitions based on changes to a table, at least one of the one or more partitions overlapping with respect to values of one or more attributes with at least one of another partition and a previous partition, the creating of the one or more partitions resulting in a decrease in a degree of clustering of the table. The subject technology determines, after creating the one or more partitions, that the degree of clustering of the table is below a clustering threshold, the clustering threshold corresponding to a clustering ratio, the clustering ratio determined by at least a proportion of rows in a layout of the table that satisfy an ordering criteria based at least in part a particular attribute of the one or more attributes. The subject technology increases the degree of clustering of the table by reclustering partitions of the table.

What is claimed is: 
     
       1. A method comprising:
 creating one or more partitions based on changes to a table, at least one of the one or more partitions overlapping with respect to values of one or more attributes with at least one of another partition and a previous partition, the creating of the one or more partitions resulting in a decrease in a degree of clustering of the table; 
 determining, after creating the one or more partitions, that the degree of clustering of the table is below a clustering threshold, the clustering threshold corresponding to a clustering ratio, the clustering ratio determined by at least a proportion of rows in a layout of the table that satisfy an ordering criteria based at least in part on a particular attribute of the one or more attributes, the determining that the degree of clustering of the table is below the clustering threshold comprising:
 determining an amount of new data added to the table, 
 determining an amount of time that a first query should take to execute, the first query comprising a test query utilized for testing the degree of clustering of the table, and 
 determining that an execution time of the first query exceeds a threshold query execution time corresponding to the amount of time that the first query should take to execute; and 
 
 in response to the execution time of the first query exceeding the threshold query execution time, increasing the degree of clustering of the table by reclustering one or more partitions of the table. 
 
     
     
       2. The method of  claim 1 , further comprising:
 storing table data for a table in a plurality of partitions, each partition comprising a portion of table data for the table, the partitions being at least partially clustered based on one or more attributes in the table. 
 
     
     
       3. The method of  claim 1 , further comprising:
 determining the degree of clustering of the table based on one or more of a number of partitions that overlap other partitions of the table, a degree of overlap of one or more partitions with other partitions of the table, each individual depth of the partitions, or a distribution of depth of the partitions. 
 
     
     
       4. The method of  claim 1 , wherein determining that the degree of clustering of the table is below the clustering threshold comprises determining one or more of an amount, frequency, or type of data manipulation language (DML) statements performed on the table, and further comprising:
 creating one or more intermediate partitions for the new data added to the table; and 
 merging the one or more intermediate partitions into a new partition. 
 
     
     
       5. The method of  claim 1 , wherein determining that the degree of clustering of the table is below the clustering threshold is based on pruning effectiveness during query compilation, and filter selectivity during query execution. 
     
     
       6. The method of  claim 1 , wherein reclustering comprises reclustering based on one or more of a reclustering resource budget, a number of partitions, data size, and available computing resources. 
     
     
       7. The method of  claim 1 , wherein reclustering comprises merging two or more partitions. 
     
     
       8. The method of  claim 2 , wherein the table being ideally clustered comprises, for each partition of the partitions of the table, the each partition comprises no overlaps with one or more other partitions in ranges of values corresponding to the one or more attributes. 
     
     
       9. The method of  claim 2 , wherein the table being ideally clustered comprises, for each partition of the partitions of the table, all rows of the each partition comprise the same value for an attribute of the one or more attributes. 
     
     
       10. A system, the system comprising:
 one or more processors; and 
 a memory device storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: 
 creating one or more partitions based on changes to a table, at least one of the one or more partitions overlapping with respect to values of one or more attributes with at least one of another partition and a previous partition, the creating of the one or more partitions resulting in a decrease in a degree of clustering of the table; 
 determining, after creating the one or more partitions, that the degree of clustering of the table is below a clustering threshold, the clustering threshold corresponding to a clustering ratio, the clustering ratio determined by at least a proportion of rows in a layout of the table that satisfy an ordering criteria based at least in part on a particular attribute of the one or more attributes, the determining that the degree of clustering of the table is below the clustering threshold comprising:
 determining an amount of new data added to the table, 
 determining an amount of time that a first query should take to execute, the first query comprising a test query utilized for testing the degree of clustering of the table, and 
 determining that an execution time the first query exceeds a threshold query execution time corresponding to the amount of time that the first query should take to execute; and 
 
 in response to the execution time of the first query exceeding the threshold query execution time, increasing the degree of clustering of the table by reclustering one or more partitions of the table. 
 
     
     
       11. The system of  claim 10 , wherein the operations further comprise:
 storing table data for a table in a plurality of partitions, each partition comprising a portion of table data for the table, the partitions being at least partially clustered based on one or more attributes in the table. 
 
     
     
       12. The system of  claim 10 , wherein the operations further comprise:
 determining the degree of clustering of the table based on one or more of a number of partitions that overlap other partitions of the table, a degree of overlap of one or more partitions with other partitions of the table, each individual depth of the partitions, or a distribution of depth of the partitions. 
 
     
     
       13. The system of  claim 10 , wherein determining that the degree of clustering of the table is below the clustering threshold comprises determining one or more of an amount, frequency, or type of data manipulation language (DML) statements performed on the table, and further comprising:
 creating one or more intermediate partitions for the new data added to the table; and 
 merging the one or more intermediate partitions into a new partition. 
 
     
     
       14. The system of  claim 10 , wherein determining that the degree of clustering of the table is below the clustering threshold is based on pruning effectiveness during query compilation, and filter selectivity during query execution. 
     
     
       15. The system of  claim 10 , wherein reclustering comprises reclustering based on one or more of a reclustering resource budget, a number of partitions, data size, and available computing resources. 
     
     
       16. The system of  claim 10 , wherein reclustering comprises merging two or more partitions. 
     
     
       17. The system of  claim 11 , wherein the table being ideally clustered comprises, for each partition of the partitions of the table, the each partition comprises no overlaps with one or more other partitions in ranges of values corresponding to the one or more attributes. 
     
     
       18. The system of  claim 11 , wherein the table being ideally clustered comprises, for each partition of the partitions of the table, all rows of the each partition comprise the same value for an attribute of the one or more attributes. 
     
     
       19. A non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
 creating one or more partitions based on changes to a table, at least one of the one or more partitions overlapping with respect to values of one or more attributes with at least one of another partition and a previous partition, the creating of the one or more partitions resulting in a decrease in a degree of clustering of the table; 
 determining, after creating the one or more partitions, that the degree of clustering of the table is below a clustering threshold, the clustering threshold corresponding to a clustering ratio, the clustering ratio determined by at least a proportion of rows in a layout of the table that satisfy an ordering criteria based at least in part on a particular attribute of the one or more attributes, the determining that the degree of clustering of the table is below the clustering threshold comprising:
 determining an amount of new data added to the table, 
 determining an amount of time that a first query should take to execute, the first query comprising a test query utilized for testing the degree of clustering of the table, and 
 determining that an execution time of the first query exceeds a threshold query execution time; and 
 
 in response to the execution time of the first query exceeding the threshold query execution time, increasing the degree of clustering of the table by reclustering one or more partitions of the table. 
 
     
     
       20. The non-transitory computer readable storage media of  claim 19 , wherein the operations further comprise:
 storing table data for a table in a plurality of partitions, each partition comprising a portion of table data for the table, the partitions being at least partially clustered based on one or more attributes in the table. 
 
     
     
       21. The non-transitory computer readable storage media of  claim 19 , wherein the operations further comprise:
 determining the degree of clustering of the table based on one or more of a number of partitions that overlap other partitions of the table, a degree of overlap of one or more partitions with other partitions of the table, each individual depth of the partitions, or a distribution of depth of the partitions. 
 
     
     
       22. The non-transitory computer readable storage media of  claim 19 , wherein determining that the degree of clustering of the table is below the clustering threshold comprises determining one or more of an amount, frequency, or type of data manipulation language (DML) statements performed on the table, and further comprising:
 creating one or more intermediate partitions for the new data added to the table; and 
 merging the one or more intermediate partitions into a new partition. 
 
     
     
       23. The non-transitory computer readable storage media of  claim 19 , wherein determining that the degree of clustering of the table is below the clustering threshold is based on pruning effectiveness during query compilation, and filter selectivity during query execution. 
     
     
       24. The non-transitory computer readable storage media of  claim 19 , wherein reclustering comprises reclustering based on one or more of a reclustering resource budget, a number of partitions, data size, and available computing resources. 
     
     
       25. The non-transitory computer readable storage media of  claim 19 , wherein reclustering comprises merging two or more partitions. 
     
     
       26. The non-transitory computer readable storage media of  claim 20 , wherein the table being ideally clustered comprises, for each partition of the partitions of the table, the each partition comprises no overlaps with one or more other partitions in ranges of values corresponding to the one or more attributes. 
     
     
       27. The non-transitory computer readable storage media of  claim 20 , wherein the table being ideally clustered comprises, for each partition of the partitions of the table, all rows of the each partition comprise the same value for an attribute of the one or more attributes.