Process and method for data assurance management by applying data assurance metrics

Abstract

The present invention relates generally to methods, software and systems for measuring and valuing the quality of information and data, where such measurements and values are made and processed by implementing objectively defined, measurable, comparable and repeatable dimensions using software and complex computers. The embodiments include processes, systems and method for identifying optimal scores of the data dimension. The invention further includes processes, systems and method for data filtering to improve the overall data quality of a data source. Finally, the invention further includes processes, systems and method for data quality assurance of groups of rows of a database.

1 . A data filtering method comprising:
 receiving data from a database;   determining if the data is valid by utilizing rules in a data hygiene engine, wherein the rules are:
 (1) at least one rule selected from the group consisting of: field rules, row rules, groups of rows rules, and combinations thereof, and 
 (2) at least one data assurance management metrics rules, wherein the data assurance management metrics rules are data quality metric rules used to analyze groups of rows based on user requirements, wherein data quality metrics are calculated for groups of rows and compared to a threshold; 
   filtering invalid data;   outputting a production database without the invalid data.   
     
     
         2 . The method of  claim 1 , wherein the field rules are related to only one field. 
     
     
         3 . The method of  claim 1 , wherein the field rules are field type or field value ranges. 
     
     
         4 . The method of  claim 1 , wherein the row rules are rules to intermix two different fields. 
     
     
         5 . The method of  claim 1 , wherein the groups of row rules group rows, calculate metrics for each group of rows, and comparing to a threshold. 
     
     
         6 . The method of  claim 5 , wherein an entire group is eliminated if the calculated metric is below the threshold. 
     
     
         7 . The method of  claim 5 , wherein the threshold is predetermined by a user. 
     
     
         8 . The method of  claim 1 , wherein a data assurance management metric is used to group the data into multiple groups to complete a score for the entire dataset. 
     
     
         9 . The method of  claim 8 , further comprising comparing the score to a threshold and eliminating all groups that do not meet the threshold. 
     
     
         10 . The method of  claim 1 , wherein a data assurance management metric is used to group the data into multiple groups and compute a completeness score for the entire dataset. 
     
     
         11 . The method of  claim 10 , further comprising comparing the completeness score to a minimum threshold and eliminating all groups that do not meet the threshold. 
     
     
         12 . The method of  claim 1 , wherein a plurality of data assurance management metrics are used together to group the data into multiple groups and compute a score for the entire dataset. 
     
     
         13 . The method of  claim 12 , further comprising comparing the completeness score to a threshold and eliminating all groups that do not meet the threshold. 
     
     
         14 . The method of  claim 13 , wherein the threshold is determined by a rule related to the plurality of data assurance management metrics. 
     
     
         15 . The method of  claim 1 , wherein an acceptable range of values or numbers is provided to the data hygiene engine. 
     
     
         16 . The method of  claim 1 , wherein invalid data is subject to further review before being filtered. 
     
     
         17 . The method of  claim 1 , wherein the invalid data is modified to create valid data that is then considered valid. 
     
     
         18 . The method of  claim 1 , further comprising running a search engine query coupled to a threshold for the at least one data assurance management metrics. 
     
     
         19 . The method of  claim 18 , further comprising setting reliability thresholds. 
     
     
         20 . The method of  claim 18 , further comprising setting confidence thresholds.