Data profiling

Abstract

Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.

What is claimed is: 
     
         1 . A method for processing data including:
 profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data; and   processing the data from the data source, including accessing the stored profile information and processing the data according to the accessed profile information.   
     
     
         2 . The method of  claim 1  wherein processing the data from the data source further includes reading the data from the data source. 
     
     
         3 . The method of  claim 1  wherein profiling the data is performed without maintaining a copy of the data outside the data source. 
     
     
         4 . The method of  claim 3  wherein the data includes variable record structure records with at least one of a conditional field and a variable number of fields. 
     
     
         5 . The method of  claim 4  wherein computing summary data characterizing the data while reading the data includes interpreting the variable record structure records while computing summary data characterizing the data. 
     
     
         6 . The method of  claim 1  wherein the data source includes a data storage system. 
     
     
         7 . The method of  claim 6  wherein the data storage system includes a database system. 
     
     
         8 . The method of  claim 1  wherein computing the summary data includes counting a number of occurrences for each of a set of distinct values for a field. 
     
     
         9 . The method of  claim 8  wherein storing profile information includes storing statistics for the field based on the counted number of occurrences for said field. 
     
     
         10 . The method of  claim 1  further including maintaining a metadata store that contains metadata related to the data source 
     
     
         11 . The method of  claim 10  wherein storing the profile information includes updating the metadata related to the data source. 
     
     
         12 . The method of  claim 10  wherein profiling the data and processing the data each make use of metadata for the data source 
     
     
         13 . The method of  claim 1  wherein profiling data from the data source further includes determining a format specification based on the profile information. 
     
     
         14 . The method of  claim 1  wherein profiling data from the data source further includes determining a validation specification based on the profile information. 
     
     
         15 . The method of  claim 14  wherein processing the data includes identifying invalid records in the data based on the validation specification. 
     
     
         16 . The method of  claim 1  wherein profiling data from the data source further includes specifying data transformation instructions based on the profile information. 
     
     
         17 . The method of  claim 16  wherein processing the data includes applying the transformation instructions to the data. 
     
     
         18 . The method of  claim 1  wherein processing the data includes importing the data into a data storage subsystem. 
     
     
         19 . The method of  claim 18  wherein processing the data includes validating the data prior to importing the data into a data storage subsystem. 
     
     
         20 . The method of  claim 19  wherein validating the data includes comparing characteristics of the data to reference characteristics for said data. 
     
     
         21 . The method of  claim 20  wherein the reference characteristics include statistical properties of the data. 
     
     
         22 . The method of  claim 1  wherein profiling the data includes profiling said data in parallel, including partitioning the data into parts and processing the parts using separate ones of a first set of parallel components. 
     
     
         23 . The method of  claim 22  wherein profiling the data in parallel further includes computing the summary data for different fields of the data using separate ones of a second set of parallel components. 
     
     
         24 . The method of  claim 23  wherein profiling the data in parallel further includes repartitioning outputs of the first set of parallel components to form inputs for the second set of parallel components. 
     
     
         25 . The method of  claim 22  wherein profiling the data in parallel includes reading the data from a parallel data source, each part of the parallel data source being processed by a different one of the first set of parallel components. 
     
     
         26 . A method for processing data including:
 profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data; wherein   profiling the data includes profiling said data in parallel, including partitioning the data into parts and processing the parts using separate ones of a first set of parallel components.   
     
     
         27 . Software stored on a computer-readable medium including instructions for causing a computer system to:
 profile data from a data source by reading the data from the data source, compute summary data characterizing the data while reading the data, and store profile information that is based on the summary data; and   process the data from the data source by accessing the stored profile information and process the data according to the accessed profile information.   
     
     
         28 . A data processing system including:
 a profiling module configured to read data from a data source, to compute summary data characterizing the data while reading the data, and to store profile information that is based on the summary data; and   a processing module configured to access the stored profile information and to process the data from the data source according to the accessed profile information.   
     
     
         29 . A data processing system including:
 means for profiling data from a data source, including means for reading the data from the data source, means for computing summary data characterizing the data while reading the data, and means for storing profile information that is based on the summary data; and   means for processing the data from the data source, including means for accessing the stored profile information and means for processing the data according to the accessed profile information.