Text segmentation with multiple granularity levels

Abstract

Text processing includes: segmenting received text based on a lexicon of smallest semantic units to obtain medium-grained segmentation results; merging the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results; looking up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and forming fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results.

What is claimed is: 
     
         1 . (canceled) 
     
     
         2 . A method, comprising:
 building a lexicon of smallest semantic units, comprising:
 receiving an entry to be classified; 
 using a trained classifier to determine whether the received entry to be classified is a smallest semantic unit or a compound semantic unit; and 
 in the event that the received entry is determined to be a smallest semantic unit, adding the received entry to the lexicon of smallest semantic units; 
   segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;   merging the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;   forming a word segmentation tree structure, wherein the received text is used to form a root node of the word segmentation tree structure, and a segment in the medium-grained segmentation results is used to form a sub-node of the root node;   looking up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results;   forming a leaf node in the word segmentation tree structure based at least in part on a corresponding search element; and   forming fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results, and obtaining the leaf node as a fine-grained segmentation result.   
     
     
         3 . The method of  claim 2 , further comprising:
 training a classifier to form the trained classifier, wherein:
 the training is based at least in part on a plurality of training sample entries; 
 a training sample entry in the plurality of training sample entries includes:
 a character count; 
 an independent use rate; 
 a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules; 
 a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries; 
 an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and 
 a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit. 
 
   
     
     
         4 . The method of  claim 2 , wherein the received text is in a non-divider marked language. 
     
     
         5 . The method of  claim 2 , further comprising, in the event that the received entry is determined to be a compound semantic unit, adding the received entry to a lexicon of compound semantic units. 
     
     
         6 . The method of  claim 3 , wherein using the trained classifier to determine whether the received entry is a smallest semantic unit or a compound semantic unit includes inputting into the trained classifier: a character count of the received entry, an independent use rate of the received entry, a phrase structure rule indicator indicating whether the received entry complies with phrase structure rules, a semantic attribute indicating an inclusion state of the received entry in the predetermined set of enumerated entries, and an overlap attribute of the received entry. 
     
     
         7 . The method of  claim 3 , further comprising:
 determining a search element that corresponds to the received entry; and   saving the search element in the lexicon of smallest semantic units.   
     
     
         8 . The method of  claim 3 , wherein determining a search element that corresponds to the received entry includes:
 determining whether the received entry is segmentable;   in the event that the received entry is segmentable, setting the search element to correspond to a fine-grained word included within the received entry; and   in the event that the entry is not segmentable, setting the search element to correspond to the received entry.   
     
     
         9 . The method of  claim 2 , wherein segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results. 
     
     
         10 . A system, comprising:
 one or more processors configured to:
 build a lexicon of smallest semantic units, comprising to:
 receive an entry to be classified; 
 use a trained classifier to determine whether the received entry to be classified is a smallest semantic unit or a compound semantic unit; and 
 in the event that the received entry is determined to be a smallest semantic unit, add the received entry to the lexicon of smallest semantic units; 
 
 segment received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results; 
 merge the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results; 
 form a word segmentation tree structure, wherein the received text is used to form a root node of the word segmentation tree structure, and a segment in the medium-grained segmentation results is used to form a sub-node of the root node; 
 look up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; 
 form a leaf node in the word segmentation tree structure based at least in part on a corresponding search element; and 
 form fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation result, and obtain the leaf node as a fine-grained segmentation result; and 
   one or more memories coupled to the one or more processors, configured to provide the one or more processors with instructions.   
     
     
         11 . The system of  claim 10 , wherein the one or more processors are further configured to:
 train a classifier to form the trained classifier, wherein:
 the training is based at least in part on a plurality of training sample entries; 
 a training sample entry in the plurality of training sample entries includes:
 a character count; 
 an independent use rate; 
 a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules; 
 a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries; 
 an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and 
 a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit. 
 
   
     
     
         12 . The system of  claim 10 , wherein the text is in a non-divider marked language. 
     
     
         13 . The system of  claim 10 , wherein the one or more processors are further configured to:
 in the event that the received entry is determined to be a compound semantic unit, add the received entry to a lexicon of compound semantic units.   
     
     
         14 . The system of  claim 11 , wherein using the trained classifier to determine whether the received entry is a smallest semantic unit or a compound semantic unit includes inputting into the trained classifier: a character count of the received entry, an independent use rate of the received entry, a phrase structure rule indicator indicating whether the received entry complies with phrase structure rules, a semantic attribute indicating an inclusion state of the received entry in the predetermined set of enumerated entries, and an overlap attribute of the received entry. 
     
     
         15 . The system of  claim 11 , wherein the one or more processors are further configured to:
 determine a search element that corresponds to the received entry; and   save the search element in the lexicon of smallest semantic units.   
     
     
         16 . The system of  claim 11 , wherein determining a search element that corresponds to the received entry includes:
 determining whether the received entry is segmentable;   in the event that the received entry is segmentable, setting the search element to correspond to a fine-grained word included within the received entry; and   in the event that the received entry is not segmentable, setting the search element to correspond to the received entry.   
     
     
         17 . The system of  claim 10 , wherein segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results. 
     
     
         18 . A computer program product being embodied in a computer readable storage medium and comprising computer instructions for:
 building a lexicon of smallest semantic units, comprising:
 receiving an entry to be classified; 
 using a trained classifier to determine whether the received entry to be classified is a smallest semantic unit or a compound semantic unit; and 
 in the event that the received entry is determined to be a smallest semantic unit, adding the received entry to the lexicon of smallest semantic units; 
   segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;   merging the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;   forming a word segmentation tree structure, wherein the received text is used to form a root node of the word segmentation tree structure, and a segment in the medium-grained segmentation results is used to form a sub-node of the root node;   looking up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results;   forming a leaf node in the word segmentation tree structure based at least in part on a corresponding search element; and   forming fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results, and obtaining the leaf node as a fine-grained segmentation result.   
     
     
         19 . The computer program product of  claim 18 , further comprising computer instructions for:
 training a classifier to form the trained classifier, wherein:
 the training is based at least in part on a plurality of training sample entries; 
 a training sample entry in the plurality of training sample entries includes:
 a character count; 
 an independent use rate; 
 a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules; 
 a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries; 
 an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and 
 a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit. 
 
   
     
     
         20 . A system, comprising:
 one or more processors configured to:
 train a classifier to form the trained classifier, wherein: 
 the training is based at least in part on a plurality of training sample entries; 
 a training sample entry in the plurality of training sample entries includes:
 a character count; 
 an independent use rate; 
 a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules; 
 a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries; 
 an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and 
 a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit; and 
 
 build a lexicon of smallest semantic units, comprising to: 
 use the trained classifier to determine whether an entry to be classified is a smallest semantic unit or a compound semantic unit; and 
 in the event that the received entry is determined to be a smallest semantic unit, add the received entry to the lexicon of smallest semantic units; and 
   one or more memories coupled to the one or more processors, configured to provide the one or more processors with instructions.