Data ingest optimization

Abstract

Methods and systems for optimizing the retrieval of data from multiple sources are described. A slot map including slots for the storage of data elements can be obtained. The data elements associated with the slots can be prioritized by weighting values with costs of retrieving the data elements from respective data sources. Each value can be associated with a different data element and can indicate a respective degree of importance of the associated data element. Further, the systems and methods can direct the retrieval of data elements from the respective data sources in an order in accordance with the priority of the data elements to optimize the quality of data obtainable within a critical time constraint. In addition, the retrieved data elements can be stored in corresponding slots on a storage medium.

1 . A method for optimizing the retrieval of data from multiple sources comprising:
 obtaining a slot map including slots for the storage of data elements;   prioritizing the data elements associated with the slots by weighting values, each of which is associated with a different data element and indicates a respective degree of importance of the associated data element, with costs of retrieving the data elements from respective data sources; and   directing a retrieval of the data elements from the respective data sources in an order in accordance with the priority of the data elements to optimize the quality of data obtainable within a critical time constraint; and   storing the retrieved data elements in corresponding slots on a storage medium.   
     
     
         2 . The method of  claim 1 , wherein the data elements provide material for analysis of a subject and wherein each value indicates a respective degree of importance of a corresponding data element in the analysis. 
     
     
         3 . The method of  claim 1 , wherein the prioritizing further comprises weighting the values with probabilities of retrieving valid data elements from respective data sources at particular times. 
     
     
         4 . The method of  claim 1 , wherein each value is based upon an expectation of success of retrieving the data element associated with the value from a corresponding data source. 
     
     
         5 . The method of  claim 1 , wherein each value is based upon an expected resource expenditure of retrieving the data element associated with the value from a corresponding data source. 
     
     
         6 . The method of  claim 1 , wherein the retrieval is constrained by at least one of a resource budget or a hard-stop end time. 
     
     
         7 . The method of  claim 1 , wherein the retrieval comprises adding additional slots to the slot map and repeating the prioritizing for the additional slots. 
     
     
         8 . The method of  claim 1 , wherein the method further comprises outputting a priority queue of the data elements. 
     
     
         9 . The method of  claim 1 , wherein the directing further comprises populating the slot map with retrieved data elements and the method further comprises outputting the slot map. 
     
     
         10 . A computer readable storage medium comprising a computer readable program code, wherein the computer readable program code when executed on a computer causes the computer to:
 obtain a slot map including slots for the storage of data elements;   prioritize the data elements associated with the slots by weighting values, each of which is associated with a different data element and indicates a respective degree of importance of the associated data element, with costs of retrieving the data elements from respective data sources; and   direct a retrieval of the data elements from the respective data sources in an order in accordance with the priority of the data elements to optimize the quality of data obtainable for the analysis within a critical time constraint.   
     
     
         11 . A method for prioritizing data from multiple sources for retrieval purposes comprising:
 receiving an indication of available data elements, an indication of available data sources capable of providing the respective data elements and quality tags for the data elements indicating a respective degree of importance of the data elements;   prioritizing the data elements by weighting the quality tags with costs of retrieving the data elements from respective data sources to generate a priority queue;   storing the priority queue on a storage medium; and   outputting the priority queue, which indicates the prioritized data elements that are retrievable from respective data sources within a critical time constraint.   
     
     
         12 . The method of  claim 11 , wherein the data elements provide material for analysis of a subject and wherein each quality tag indicates a respective degree of importance of a corresponding data element in the analysis. 
     
     
         13 . The method of  claim 11 , wherein the prioritizing further comprises weighting the quality tags with probabilities of retrieving valid data elements from respective data sources at particular times. 
     
     
         14 . The method of  claim 11 , wherein the prioritizing further comprises weighting each quality tag with an expectation of success of retrieving the data element associated with the quality tag from a corresponding data source. 
     
     
         15 . The method of  claim 11 , further comprising weighting each quality tag with an expected resource expenditure of retrieving the data element associated with the quality tag from a corresponding data source. 
     
     
         16 . The method of  claim 11 , wherein the prioritizing comprises selecting data elements for inclusion in the priority queue based upon at least one of a resource budget or a hard-stop end time. 
     
     
         17 . A system for optimizing the retrieval of data from multiple sources comprising:
 a slot map generator configured to generate a slot map including slots for the storage of data elements;   a priority module configured to prioritize data elements associated with the slots by weighting values, each of which is associated with a different data element and indicates a respective degree of importance of the associated data element, with probabilities of retrieving data elements from respective data sources; and   a processor configured to direct a retrieval of the data elements from the respective data sources in an order in accordance with the priority of the data elements to optimize the quality of data obtainable within a critical resource constraint.   
     
     
         18 . The system of  claim 17 , wherein the data elements provide material for analysis of a subject and wherein each value indicates a respective degree of importance of a corresponding data element in the analysis. 
     
     
         19 . The system of  claim 17 , wherein the priority module is further configured to weight the values with costs of retrieving the data elements from respective data sources. 
     
     
         20 . The system of  claim 17 , wherein the priority module is further configured to base each value upon an expectation of success of retrieving the data element associated with the value from a corresponding data source. 
     
     
         21 . The system of  claim 17 , wherein the priority module is further configured to base each value upon an expected resource expenditure of retrieving the data element associated with the value from a corresponding data source. 
     
     
         22 . The system of  claim 17 , wherein the critical resource constraint is at least one of a resource budget or a critical time constraint. 
     
     
         23 . The system of  claim 17 , wherein the processor is further configured to add additional slots to the slot map and to repeat the prioritizing for the additional slots. 
     
     
         24 . The system of  claim 17 , wherein the processor is further configured to output a priority queue of the data elements. 
     
     
         25 . The system of  claim 17 , wherein the processor is further configured to populate the slot map with retrieved data elements and to output the slot map.