System and method for integrating systems to implement data quality processing

Abstract

System and method for integrating systems to implement data quality processing. A business rule creation module is configured to create a business rule associated with a business term. A data quality specification module is configured to create a data quality specification based on the business rule. The data quality specification comprises (1) an identity of a column of a table stored in a database comprising data to be tested; (2) a test to perform on the data to be tested; and (3) reference data required to perform the test on the data. A validation module is configured to receive the data quality specification; retrieve data associated with the column from the database; and test the retrieved data in accordance with the test using the reference data. A result publication module is configured to return a result of the test to the data quality specification module.

1 . (canceled) 
     
     
         2 . A computer-implemented method, comprising:
 receiving a data quality rule, wherein the data quality rule corresponds to a data quality specification generated using a business rule and stored metadata, and wherein the data quality rule is used to determine whether data in a production environment conforms to the data quality specification;   executing a test to validate the data quality rule, wherein the test includes applying reference data to the data quality rule to obtain a set of results, and wherein the test and the reference data are specified in the data quality specification;   continually updating the data quality rule based on a determination as to whether the data quality rule is ready for production, wherein the determination is generated based on the set of results;   generating a code configured to automatically execute the data quality rule, wherein the code is generated as a result of the data quality rule being ready for production; and   promoting the code to the production environment, wherein when the code is promoted to the production environment, the data quality rule is automatically executed to process the data in the production environment to determine whether the data conforms to the data quality specification.   
     
     
         3 . The computer-implemented method of  claim 2 , wherein the data quality specification includes a threshold, and wherein the threshold corresponds to one or more metrics for generating the determination. 
     
     
         4 . The computer-implemented method of  claim 2 , wherein the business rule includes a policy implemented by the business rule. 
     
     
         5 . The computer-implemented method of  claim 2 , wherein the data quality specification includes an execution frequency for automatically executing the data quality rule. 
     
     
         6 . The computer-implemented method of  claim 2 , wherein the data quality specification includes an effective date of the data quality rule. 
     
     
         7 . The computer-implemented method of  claim 2 , wherein the data quality specification includes a set of indications of database columns that include the reference data. 
     
     
         8 . The computer-implemented method of  claim 2 , further comprising publishing the set of results to a record associated with the data quality rule, wherein the record is accessible via a link. 
     
     
         9 . A system, comprising:
 one or more processors; and   memory storing thereon instructions that, as a result of being executed by the one or more processors, cause the system to:
 receive a data quality rule, wherein the data quality rule corresponds to a data quality specification generated using a business rule and stored metadata, and wherein the data quality rule is used to determine whether data in a production environment conforms to the data quality specification; 
 execute a test to validate the data quality rule, wherein the test includes applying reference data to the data quality rule to obtain a set of results, and wherein the test and the reference data are specified in the data quality specification; 
 continually update the data quality rule based on a determination as to whether the data quality rule is ready for production, wherein the determination is generated based on the set of results; 
 generate a code configured to automatically execute the data quality rule, wherein the code is generated as a result of the data quality rule being ready for production; and 
 promote the code to the production environment, wherein when the code is promoted to the production environment, the data quality rule is automatically executed to process the data in the production environment to determine whether the data conforms to the data quality specification. 
   
     
     
         10 . The system of  claim 9 , wherein the data quality specification includes a threshold, and wherein the threshold corresponds to one or more metrics for generating the determination. 
     
     
         11 . The system of  claim 9 , wherein the business rule includes a policy implemented by the business rule. 
     
     
         12 . The system of  claim 9 , wherein the data quality specification includes an execution frequency for automatically executing the data quality rule. 
     
     
         13 . The system of  claim 9 , wherein the data quality specification includes an effective date of the data quality rule. 
     
     
         14 . The system of  claim 9 , wherein the data quality specification includes a set of indications of database columns that include the reference data. 
     
     
         15 . The system of  claim 9 , wherein the instructions further cause the system to publish the set of results to a record associated with the data quality rule, wherein the record is accessible via a link. 
     
     
         16 . A non-transitory, computer-readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:
 receive a data quality rule, wherein the data quality rule corresponds to a data quality specification generated using a business rule and stored metadata, and wherein the data quality rule is used to determine whether data in a production environment conforms to the data quality specification;   execute a test to validate the data quality rule, wherein the test includes applying reference data to the data quality rule to obtain a set of results, and wherein the test and the reference data are specified in the data quality specification;   continually update the data quality rule based on a determination as to whether the data quality rule is ready for production, wherein the determination is generated based on the set of results;   generate a code configured to automatically execute the data quality rule, wherein the code is generated as a result of the data quality rule being ready for production; and   promote the code to the production environment, wherein when the code is promoted to the production environment, the data quality rule is automatically executed to process the data in the production environment to determine whether the data conforms to the data quality specification.   
     
     
         17 . The non-transitory, computer-readable storage medium of  claim 16 , wherein the data quality specification includes a threshold, wherein the threshold corresponds to one or more metrics for generating the determination. 
     
     
         18 . The non-transitory, computer-readable storage medium of  claim 16 , wherein the business rule includes a policy implemented by the business rule. 
     
     
         19 . The non-transitory, computer-readable storage medium of  claim 16 , wherein the data quality specification includes an execution frequency for automatically executing the data quality rule. 
     
     
         20 . The non-transitory, computer-readable storage medium of  claim 16 , wherein the data quality specification includes an effective date of the data quality rule. 
     
     
         21 . The non-transitory, computer-readable storage medium of  claim 16 , wherein the data quality specification includes a set of indications of database columns that include the reference data. 
     
     
         22 . The non-transitory, computer-readable storage medium of  claim 16 , wherein the executable instructions further cause the computer system to publish the set of results to a record associated with the data quality rule, wherein the record is accessible via a link.