Hybrid approach for short form detection and expansion to long forms

Abstract

Embodiments provide a system and method for short form and long form detection. Given candidate short forms, the system can generate one or more n-gram combinations, resulting in one or more candidate short form and n-gram combination pairs. For each candidate short form and n-gram combination pair, the system can calculate an approximate string matching distance, calculate a best possible alignment score, calculate a confidence score, calculate a topic similarity score, and calculate a semantic similarity score. The system can determine the validity, through a meta learner, of the one or more valid candidate short form and n-gram combination pairs based upon each short form and n-gram combination pair's confidence score, topic similarity score, and semantic similarity score, and store the valid short form and n-gram combination pairs in a repository. The system has no language specific constraints and can extract short form and long form pairs from documents written in various languages. The system is also not limited to whether the language of the given corpus is case sensitive or not.

What is claimed is: 
     
         1 . A computer implemented method, in a data processing system comprising a processor and a memory comprising instructions which are executed by the processor to cause the processor to implement a short form detection system, the method comprising:
 ingesting a corpus comprising one or more documents;   splitting the one or more documents into one or more tokens;   filtering the one or more tokens by:
 discarding the one or more tokens that are not nouns; 
 discarding the one or more tokens that have more than a pre-determined number of digits; and 
 discarding the one or more tokens belonging to a pre-determined percentage of the most frequently appearing tokens in the corpus; 
   wherein all remaining tokens comprise short form candidates; and   storing the short form candidates in the memory.   
     
     
         2 . The method as recited in  claim 1 , further comprising:
 for each short form candidate, generating one or more n-gram combinations, resulting in one or more short form candidate and n-gram combination pairs.   
     
     
         3 . The method as recited in  claim 2 , further comprising:
 for each short form candidate and n-gram combination pair, determining an approximate string matching distance.   
     
     
         4 . The method as recited in  claim 3 , wherein the step of determining the approximate string matching distance further comprises:
 defining an array of the remaining tokens for each n-gram combination;   defining an array of characters from each short form candidate;   comparing the array of the remaining tokens with the array of characters to determine a number of matches and a number of skipped tokens; and   calculating the approximate string matching distance using the number of matches, the number of skipped tokens, and a total number of tokens.   
     
     
         5 . The method as recited in  claim 4 , wherein the approximate string matching distance is calculated as the absolute value of the total number of tokens minus the number of skipped tokens minus the number of matches. 
     
     
         6 . The method as recited in  claim 1 , wherein the filtering step further includes automatically detecting if the one or more documents are written in a case-sensitive language. 
     
     
         7 . The method as recited in  claim 6 , wherein the filtering step further includes discarding the one or more tokens that lack at least one uppercase letter. 
     
     
         8 . The method as recited in  claim 1 , wherein the pre-determined number of digits comprises symbols. 
     
     
         9 . The method as recited in  claim 1 , wherein a maximum number of digits is one. 
     
     
         10 . The method as recited in  claim 1 , wherein the pre-determined percentage is 0.5%. 
     
     
         11 . A computer program product for short form detection, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
 ingest a corpus comprising one or more documents;   split the one or more documents into one or more tokens;   filter the one or more tokens by:
 discarding the one or more tokens that are not nouns; 
 discarding the one or more tokens that have more than a pre-determined number of digits; and 
 discarding the one or more tokens belonging to a pre-determined percentage of the most frequently appearing tokens in the corpus; 
   wherein all remaining tokens comprise short form candidates; and   store the short form candidates in a memory.   
     
     
         12 . The computer program product as recited in  claim 1 , the processor further configured to:
 generate one or more n-gram combinations for each short form candidate, resulting in one or more short form candidate and n-gram combination pairs.   
     
     
         13 . The computer program product as recited in  claim 12 , the processor further configured to:
 determine an approximate string matching distance for each short form candidate and n-gram combination pair.   
     
     
         14 . The computer program product as recited in  claim 13 , the processor further configured to:
 define an array of the remaining tokens for each n-gram combination;   define an array of characters from each short form candidate;   compare the array of the remaining tokens with the array of characters to determine a number of matches and a number of skipped tokens; and   calculate the approximate string matching distance using the number of matches, the number of skipped tokens, and a total number of tokens.   
     
     
         15 . The computer program product as recited in  claim 14 , the processor further configured to:
 determine the approximate string matching distance as the absolute value of the total number of tokens minus the number of skipped tokens minus the number of matches.   
     
     
         16 . The computer program product as recited in  claim 11 , the processor further configured to:
 automatically detect if the one or more documents are written in a case-sensitive language.   
     
     
         17 . The computer program product as recited in  claim 16 , the processor further configured to:
 discard the one or more tokens that lack at least one uppercase letter.   
     
     
         18 . The computer program product as recited in  claim 11 , wherein the pre-determined number of digits comprises symbols. 
     
     
         19 . The computer program product as recited in  claim 11 , wherein a maximum number of digits is one. 
     
     
         20 . A system for short form detection, comprising:
 a short form detection processor configured to:   ingest a corpus comprising one or more documents;   split the one or more documents into one or more tokens;   filter the one or more tokens by:
 discarding the one or more tokens that are not nouns; 
 discarding the one or more tokens that have more than a pre-determined number of digits; and 
 discarding the one or more tokens belonging to a pre-determined percentage of the most frequently appearing tokens in the corpus; 
   wherein all remaining tokens comprise short form candidates; and   store the short form candidates in a memory.