Efficient export scheduling using index and schema to determine data distribution

Abstract

A method includes determining a data record partition size based on resources used for transferring data from a higher storage tier that comprises first memory to one or more lower storage tiers that comprise second memory, determining a distribution mapping of the first memory, identifying all sets of contiguously stored data records on the higher storage tier that are suitable for export and greater in size than the data record partition size, sending a list of sets of contiguously stored data records to an exporter, transmitting from the higher storage tier to the one or more lower storage tiers, by the exporter, each set of contiguously stored data records, and transmitting from the higher storage tier to the one or more lower storage tiers, all remaining data records that are suitable for export and are not included on the list of logically sorted sets of contiguously stored data records.

What is claimed is: 
     
         1 . A method, comprising:
 determining a data record partition size based on resources used for transferring data from a higher storage tier that comprises first memory to one or more lower storage tiers that comprise second memory;   determining a distribution mapping of the first memory, the distribution mapping indicating a relative distribution of storage locations for all of the data records that are stored to the higher storage tier;   identifying all sets of contiguously stored data records on the higher storage tier that are suitable for export and greater in size than the data record partition size;   sending a list of logically sorted sets of contiguously stored data records to an exporter;   transmitting from the higher storage tier to the one or more lower storage tiers, by the exporter, each set of contiguously stored data records from the list of logically sorted sets of contiguously stored data records; and   transmitting from the higher storage tier to the one or more lower storage tiers, by the exporter using dynamic object sizing, all remaining data records that are suitable for export and are not included on the list of logically sorted sets of contiguously stored data records, wherein empty regions in the first memory of the higher storage tier are skipped during transmission of the remaining data records that are suitable for export.   
     
     
         2 . The method as recited in  claim 1 , further comprising determining a number of processor threads to use for transmission of a first set of contiguously stored data records that minimizes use of the resources. 
     
     
         3 . The method as recited in  claim 1 , further comprising determining a number of processor threads to use for transmission of a first set of contiguously stored data records that minimizes use of the resources based on a size of the first set of contiguously stored data records. 
     
     
         4 . The method as recited in  claim 1 , further comprising determining an object size and a number of processor threads to use for transmission of a first data record or a first set of data records of the remaining data records that minimizes use of the resources. 
     
     
         5 . The method as recited in  claim 1 , further comprising determining an object size and a number of processor threads to use for transmission of a first data record or a first set of data records of the remaining data records that minimizes use of the resources based on a size of the first data record or the first set of data records. 
     
     
         6 . The method as recited in  claim 1 , wherein the resources used for transferring data from the higher storage tier comprise: processor capacity, storage bandwidth, and network bandwidth. 
     
     
         7 . The method as recited in  claim 1 , wherein the determining which data records stored to the higher storage tier are suitable for export is based on detection of a triggering event. 
     
     
         8 . A system, comprising:
 a higher storage tier comprising first memory;   one or more lower storage tiers comprising second memory;   a processing circuit; and   logic integrated with the processing circuit, executable by the processing circuit, or integrated with and executable by the processing circuit, the logic being configured to cause the processing circuit to:
 determine a data record partition size based on resources used for transferring data from the higher storage tier; 
 determine a distribution mapping of the first memory, the distribution mapping indicating a relative distribution of storage locations for all of the data records that are stored to the higher storage tier; 
 identify all sets of contiguously stored data records on the higher storage tier that are suitable for export and greater in size than the data record partition size; 
 send a list of logically sorted sets of contiguously stored data records to an exporter; 
 transmit from the higher storage tier to the one or more lower storage tiers, by the exporter, each set of contiguously stored data records from the list of logically sorted sets of contiguously stored data records; and 
 transmit from the higher storage tier to the one or more lower storage tiers, by the exporter using dynamic object sizing, all remaining data records that are suitable for export and are not included on the list of logically sorted sets of contiguously stored data records, wherein empty regions in the first memory of the higher storage tier are skipped during transmission of the remaining data records that are suitable for export. 
   
     
     
         9 . The system as recited in  claim 8 , wherein the logic is further configured to cause the processing circuit to:
 determine a number of processor threads to use for transmission of a first set of contiguously stored data records that minimizes use of the resources.   
     
     
         10 . The system as recited in  claim 8 , wherein the logic is further configured to cause the processing circuit to:
 determine a number of processor threads to use for transmission of a first set of contiguously stored data records that minimizes use of the resources based on a size of the first set of contiguously stored data records.   
     
     
         11 . The system as recited in  claim 8 , wherein the logic is further configured to cause the processing circuit to:
 determine an object size and a number of processor threads to use for transmission of a first data record or a first set of data records of the remaining data records that minimizes use of the resources.   
     
     
         12 . The system as recited in  claim 8 , wherein the logic is further configured to cause the processing circuit to:
 determine an object size and a number of processor threads to use for transmission of a first data record or a first set of data records of the remaining data records that minimizes use of the resources based on a size of the first data record or the first set of data records.   
     
     
         13 . The system as recited in  claim 8 , wherein the resources used for transferring data from the higher storage tier comprise: processor capacity, storage bandwidth, and network bandwidth. 
     
     
         14 . The system as recited in  claim 8 , wherein the logic configured to cause the processing circuit to determine which data records stored to the higher storage tier are suitable for export is based on detection of a triggering event. 
     
     
         15 . A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the embodied program instructions executable by a processing circuit to cause the processing circuit to:
 determine, by the processing circuit, a data record partition size based on resources used for transferring data from a higher storage tier that comprises first memory to one or more lower storage tiers that comprise second memory;   determine, by the processing circuit, a distribution mapping of the first memory, the distribution mapping indicating a relative distribution of storage locations for all of the data records that are stored to the higher storage tier;   identify, by the processing circuit, all sets of contiguously stored data records on the higher storage tier that are suitable for export and greater in size than the data record partition size;   send, by the processing circuit, a list of logically sorted sets of contiguously stored data records to an exporter;   transmit, by the processing circuit, from the higher storage tier to the one or more lower storage tiers, by the exporter, each set of contiguously stored data records from the list of logically sorted sets of contiguously stored data records; and   transmit, by the processing circuit, from the higher storage tier to the one or more lower storage tiers, by the exporter using dynamic object sizing, all remaining data records that are suitable for export and are not included on the list of logically sorted sets of contiguously stored data records, wherein empty regions in the first memory of the higher storage tier are skipped during transmission of the remaining data records that are suitable for export.   
     
     
         16 . The computer program product as recited in  claim 15 , wherein the program instructions are further executable by the processing circuit to cause the processing circuit to determine a number of processor threads to use for transmission of a first set of contiguously stored data records that minimizes use of the resources. 
     
     
         17 . The computer program product as recited in  claim 15 , wherein the program instructions are further executable by the processing circuit to cause the processing circuit to determine a number of processor threads to use for transmission of a first set of contiguously stored data records that minimizes use of the resources based on a size of the first set of contiguously stored data records. 
     
     
         18 . The computer program product as recited in  claim 15 , wherein the program instructions are further executable by the processing circuit to cause the processing circuit to determine an object size and a number of processor threads to use for transmission of a first data record or a first set of data records of the remaining data records that minimizes use of the resources. 
     
     
         19 . The computer program product as recited in  claim 15 , wherein the program instructions are further executable by the processing circuit to cause the processing circuit to determine an object size and a number of processor threads to use for transmission of a first data record or a first set of data records of the remaining data records that minimizes use of the resources based on a size of the first data record or the first set of data records. 
     
     
         20 . The computer program product as recited in  claim 15 , wherein the resources used for transferring data from the higher storage tier comprise: processor capacity, storage bandwidth, and network bandwidth.