Data Disambiguation Systems and Methods

Abstract

Various embodiments provide a state-based, regular expression parser in which data, such as generally unstructured text, is received into the system and undergoes a tokenization process which permits structure to be imparted to the data. Tokenization of the data effectively enables various patterns in the data to be identified. In some embodiments, one or more components can utilize stimulus/response paradigms to recognize and react to patterns in the data.

1 . A system comprising: 
 one or more computer-readable media;    a functional presence engine embodied on the one or more computer-readable media and configured as a probabilistic parser to receive documents and perform lexical analysis on the documents effective to tokenize document portions; and    a knowledge base component embodied on the one or more computer-readable media and operably associated with the functional presence engine and configured to define: 
 cases of text that can appear in documents and be matched to documents received by the functional presence engine; and  
 responses that are triggered in an event of a match;  
   a table embodied on the one or more computer-readable media and associated with the functional presence engine and knowledge base component and configured to contain entries associated with tokens assigned in the tokenization process and documents in which the tokens appear; and    an information retrieval component configured to search for tokens and identify associated documents.    
   
   
       2 . The system of  claim 1 , wherein the tokenized document portions comprise individual words that appear in the documents.  
   
   
       3 . The system of  claim 1 , wherein the information retrieval component is configured to use regular expressions to search for tokens.  
   
   
       4 . The system of  claim 1 , wherein the functional presence engine is configured to use: 
 one or more lexical files, individual lexical files specifying one or more matching rules and corresponding output symbols; and    a lexical analysis program which processes documents to produce tokenized document portions in accordance with the one or more matching rules specified by the one or more lexical files.    
   
   
       5 . The system of  claim 4 , wherein the one or more rules are specified as regular expressions.  
   
   
       6 . The system of  claim 1 , wherein the knowledge base component is associated with one or more files that define cases of text that can be matched to text in documents received by the functional presence engine and responses that are triggered in an event of a match, the files being defined in a hierarchical tag-based language.  
   
   
       7 . The system of  claim 6 , wherein the hierarchical tag-based language comprises an extensible markup language.  
   
   
       8 . The system of  claim 6 , wherein one tag comprises a top level tag that encapsulates all other tags.  
   
   
       9 . The system of  claim 6 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file to be loaded.  
   
   
       10 . The system of  claim 6 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file that is to be loaded and used to perform the lexical analysis.  
   
   
       11 . The system of  claim 6 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match.  
   
   
       12 . The system of  claim 6 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match, wherein the text patterns are defined as regular expressions.  
   
   
       13 . A method comprising: 
 providing a functional presence engine configured as a probabilistic parser to receive documents and perform lexical analysis on the documents effective to tokenize document portions;    providing a knowledge base component operably associated with the functional presence engine and configured to define: 
 cases of text that can appear in documents and be matched to documents received by the functional presence engine, and  
 responses that are triggered in an event of a match;  
   using the functional presence engine and the knowledge base component, building a table that contains entries associated with tokens assigned in the tokenization process and documents in which the tokens appear; and    using an information retrieval component to search for tokens and identify associated documents.    
   
   
       14 . The method of  claim 13 , wherein the tokenized document portions comprise individual words that appear in the documents.  
   
   
       15 . The method of  claim 13 , wherein the information retrieval component is configured to use regular expressions to search for tokens.  
   
   
       16 . The method of  claim 13 , wherein the functional presence engine is configured to use: 
 one or more lexical files, individual lexical files specifying one or more matching rules and corresponding output symbols; and    a lexical analysis program which processes documents to produce tokenized document portions in accordance with the one or more matching rules specified by the one or more lexical files.    
   
   
       17 . The method of  claim 16 , wherein the one or more rules are specified as regular expressions.  
   
   
       18 . The method of  claim 13 , wherein the knowledge base component is associated with one or more files that define: 
 cases of text that can be matched to text in documents received by the functional presence engine; and    responses that are triggered in an event of a match, the files being defined in a hierarchical tag-based language.    
   
   
       19 . The method of  claim 18 , wherein the hierarchical tag-based language comprises an extensible markup language.  
   
   
       20 . The method of  claim 18 , wherein one tag comprises a top level tag that encapsulates all other tags.  
   
   
       21 . The method of  claim 18 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file to be loaded.  
   
   
       22 . The method of  claim 18 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file that is to be loaded and used to perform the lexical analysis.  
   
   
       23 . The method of  claim 18 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match.  
   
   
       24 . The method of  claim 18 , wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match, wherein text patterns are defined as regular expressions.  
   
   
       25 . A method comprising: 
 receiving text obtained from a data origination entity;    probabilistically parsing the text effective to tokenize text portions with tokens, said probabilistically parsing comprising using one or more matching rules defined as regular expressions in an attempt to match to text received from the data origination entity;    conducting one or more searches on the tokens.    
   
   
       26 . The method of  claim 25 , wherein the act of probabilistically parsing comprises using one or more files defining a hierarchical tag-based structure to provide said one or more matching rules.  
   
   
       27 . The method of  claim 25 , wherein the act of probabilistically parsing comprises using one or more files defining a hierarchical tag-based structure to provide said one or more matching rules, and wherein the hierarchical tag-based structure comprises a tag that specifies a file that is to be loaded and used to perform said probabilistically parsing.  
   
   
       28 . The method of  claim 25 , wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place.  
   
   
       29 . The method of  claim 25 , wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place, and wherein individual runtime objects can be associated with more than one knowledge base object.  
   
   
       30 . The method of  claim 25 , wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place, and wherein runtime objects are configured to maintain state information associated with text that is received.  
   
   
       31 . The method of  claim 25 , wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place, and wherein the knowledge base objects are configured to process text received from the runtime objects and return scores generated by processing the text, to the run time objects.