Systems and methods for manipulation of inexact semi-structured data

Abstract

The data constraint framework solution of the present invention addresses data quality issues by standardizing, verifying, matching, consolidating and merging data records using powerful inexact matching logic and search reduction technologies. The data conditioning framework uses these technologies to more efficiently condition data to improve the quality of data and/or resolve quality data issues such as incomplete, inaccurate and duplicate data records. For example, the data conditioning framework is used to “cleanse” incorrect, incomplete and duplicate data from a data source, such as an information system. The data conditioning framework uses the following approximate searching and matching techniques to improve the efficiency of the approximate matching, reduce the search space for approximate matching, and improve the speed of executing approximate searches and matches: 1) inexact trimmed matching, 2) adaptive search ordering, 3) cascading search space reduction, 4) tiered and metric indexing, and 5) domain knowledge matching.

1 . A method for improving an approximate match of a string with a reference data element by approximately matching the string with one or more data elements identifying alternative representations of the reference data element, the method comprising:
 (a) receiving, by a device, a request to perform approximate matching of a string with a reference data element;   (b) determining, by an approximate matching engine executing on the device, an approximate match of the string to the reference data element and one or more alternative data elements, the one or more alternative data elements comprising alternative representations of a value of the reference data element; and   (c) identifying, by the device, as a result one of the alternative data elements having the approximate match within a predetermined threshold.   
     
     
         2 . The method of  claim 1 , wherein one of the string, the reference data element, or the one or more alternative data elements identifies one of a person, an institute or a location. 
     
     
         3 . The method of  claim 1 , wherein the string, the reference data element, or the one or more alternative data elements represent data of one of the following systems: accounting, manufacturing, customer relationship management, enterprise resource planning, product data management, product lifecycle management, supply chain management, bioinformatics system or lab information management system. 
     
     
         4 . The method of  claim 1 , further comprising identifying, by the device, the result having the closest match within the predetermined threshold. 
     
     
         5 . The method of  claim 1 , wherein step(c) comprises identifying, by the device the reference data element as the result having the approximate match within the predetermined threshold. 
     
     
         6 . The method of  claim 1 , wherein the reference data element comprises an abbreviation of a state, and the one or more alternative data elements comprise alternative representations of identifying the state. 
     
     
         7 . The method of  claim 1 , wherein the reference data element comprises an abbreviation of a country, and the one or more alternative data elements comprise alternative representations of identifying the country. 
     
     
         8 . The method of  claim 1 , wherein the reference data element comprises a zip code, and the one or more alternative data elements comprise a city associated with the zip code. 
     
     
         9 . The method of  claim 1 , wherein the reference data element comprises a portion of an address, and the one or more alternative data elements comprise alternative representations of the portion of the address. 
     
     
         10 . The method of  claim 1 , wherein the reference data element comprises a type of business, and the one or more alternative data elements comprise alternative representations of the type of business. 
     
     
         11 . The method of  claim 1 , wherein the reference data element comprises a portion of a city name, and the one or more alternative data elements comprise alternative representations of the portion of the city name. 
     
     
         12 . The method of  claim 1 , wherein the reference data element comprises a name of a company, and the one or more alternative data elements comprise alternative representations of the name of the company. 
     
     
         13 . The method of  claim 1 , wherein the reference data element comprises a secondary address unit designator, and the one or more alternative data elements comprise alternative representations of the secondary address unit designator. 
     
     
         14 . A system for improving an approximate match of a string with a reference data element by approximately matching the string with one or more data elements identifying alternative representations of the reference data element, the system comprising:
 a device receiving a request to perform approximate matching of a string with a reference data element;   an approximate matching engine executing on the device determining an approximate match of the string to the reference data element and one or more alternative data elements, the one or more alternative data elements comprising alternative representations of a value of the reference data element; and   wherein the device identifies as a result one of the alternative data elements having the approximate match within a predetermined threshold.   
     
     
         15 . The system of  claim 14 , wherein one of the string, the reference data element, or the one or more alternative data elements identifies one of a person, an institute or a location. 
     
     
         16 . The system of  claim 14 , wherein the string, the reference data element, or the one or more alternative data elements represent data of one of the following systems: accounting, manufacturing, customer relationship management, enterprise resource planning, product data management, product lifecycle management, supply chain management, bioinformatics system or lab information management system. 
     
     
         17 . The system of  claim 14 , further comprising identifying, by the device, the result having the closest match within the predetermined threshold. 
     
     
         18 . The system of  claim 14 , wherein the reference data element comprises an abbreviation of a country, and the one or more alternative data elements comprise alternative representations of identifying the country. 
     
     
         19 . The system of  claim 14 , wherein the reference data element comprises a zip code, and the one or more alternative data elements comprise a city associated with the zip code. 
     
     
         20 . The system of  claim 14 , wherein the reference data element comprises a portion of an address, and the one or more alternative data elements comprise alternative representations of the portion of the address.