Using context to extract entities from a document collection

Abstract

Described is using context information obtained from entity mentions in likely relevant documents to extract entity mentions from documents that are ambiguous with respect to their relevance to a domain. A list of entities is input into an entity extraction mechanism, which processes a large collection of documents to determine data (counts) corresponding to frequency of entity mentions. Infrequently mentioned entities are specific entities, while frequently mentioned entities are non-specific (generic or ambiguous) entities. The context surrounding mentions of the specific entities is processed to obtain interesting context terms (words, phrases or both) for the domain. The interesting context terms are then compared against the contexts of non-specific entity mentions to determine whether each non-specific entity mention is relevant to the domain. A result set containing only relevant documents or relevant mentions collection is output.

What is claimed is: 
     
         1 . A method, comprising:
 determining how frequently a plurality of entity names corresponding to a domain appear in a collection of documents;   classifying at least some of the entity names as specific entities if they appear less than a threshold number of times in the collection of documents;   identifying context terms located in the collection of documents within a word count proximity to the specific entities;   computing affinity counts of the context terms, each affinity count indicative of how frequently a corresponding context term appears in the collection of documents;   classifying and storing at least some of the specific entities as being related to the domain based on the computing affinity counts.   
     
     
         2 . The method of  claim 1 , further comprising classifying at least some of the specific entities as non-specific entities based on being mentioned in the collection of documents more than a threshold number of times. 
     
     
         3 . The method of  claim 2  further comprising:
 outputting results corresponding to a first set of documents that include a mention of a specific entity; and 
 outputting a second set of documents that each includes a mention of a non-specific entity and has one or more of the interesting context terms within the context of the non-specific entity. 
 
     
     
         4 . The method of  claim 3 , further comprising merging a plurality of instances of a common document into a single representation of that common document in the result. 
     
     
         5 . The method of  claim 1 , wherein classifying and storing at least some of the entity names as being related to the domain based on the computed affinity counts comprises comparing a percentage of the documents that contain the at least some of the entity names with respect to a total number of documents against a threshold percentage and classifying entities below the threshold percentage into a category corresponding to the non-specific entities. 
     
     
         6 . The method of  claim 1 , further comprising identifying interesting context terms for the specific entities based on the computed affinity counts by discarding some of the context terms based on the computed affinity counts indicating said some of the context terms are located in the collection of documents within the word county proximity to at least one of the specific terms more than a threshold number of times. 
     
     
         7 . The method of  claim 1 , further comprising identifying interesting context terms for the specific entities using the affinity counts to eliminate candidate context terms for being as likely to be mentioned within the context of a specific entity as mentioned outside the context. 
     
     
         8 . The method of  claim 1 , wherein said computing the affinity counts of the context terms indicative of how frequently a corresponding context term appears in the collection of documents comprises:
 identifying candidate context terms located in the collection of documents within the word count proximity to the specific entities,   identifying instances of the candidate context terms that are within the word count proximity to at least one of the specific entities, and   eliminating the instances of the candidate context terms that are located beyond the word count proximity from being counted in the affinity counts.   
     
     
         9 . An apparatus, comprising:
 at least one processor, and   memory communicatively coupled to the at least one processor and including:
 an entity extraction mechanism configured for:
 processing a collection of documents to determine how frequently a plurality of text entities corresponding to a domain are mentioned across the collection of documents, 
 classifying some of the text entities as ambiguous terms and other of the text entities as non-ambiguous terms based on how frequently the text entities appear in the collection of documents, wherein the at least some of the text entities are classified as ambiguous terms when located in the collection of documents more than a threshold number of times and the other of the text entities are classified as non-ambiguous when located less than a threshold number of times, 
 determining at least some of the non-ambiguous terms are related to the domain based on frequency counts of appearance in the collection of documents, and 
 storing an association that the at least some of the non-ambiguous terms are related to the domain. 
 
   
     
     
         10 . The apparatus of  claim 9 , further comprising:
 identifying interesting context terms within a word count proximity to the non-ambiguous terms, and   basing, in part, the determination that the at least some of the non-ambiguous terms are related to the domain on the interesting context terms.   
     
     
         11 . The apparatus of  claim 10 , wherein the entity extraction mechanism is further configured for outputting documents that contain one or more mentions of the non-ambiguous terms, wherein the results include a document identifier, data corresponding to the non-ambiguous term mention in a document, and data corresponding to a location of the non-ambiguous term in the document. 
     
     
         12 . The apparatus of  claim 10 , further comprising changing classification of at least one of the non-ambiguous terms to an ambiguous term based on at least one of the interesting context terms. 
     
     
         13 . The apparatus of  claim 9 , wherein the entity extraction mechanism is further configured for determining interesting context terms through determining candidate context terms and using affinity count information of the candidate context terms over the collection of documents to eliminate candidate context terms that appear frequently in the document collection. 
     
     
         14 . The apparatus of  claim 9  wherein the entity extraction mechanism is further configured for determining interesting context terms through determining candidate context terms and using count information to eliminate candidate context terms that are as likely to be mentioned within the context of a specific entity as mentioned outside the context. 
     
     
         15 . The apparatus of  claim 9  wherein the entity extraction mechanism is further configured for determining the interesting context terms through eliminating candidate context terms based upon a set of stopwords. 
     
     
         16 . The apparatus of  claim 9  wherein the domain corresponds to a movie domain, a medicine domain, a music-related domain, a consumer products domain, or a people domain. 
     
     
         17 . The apparatus of  claim 9  wherein the collection of documents comprises web documents. 
     
     
         18 . One or more storage memory having computer-executable instructions executable for analyzing text entities in a collection of documents and storing classifications of the text entities as being related to a domain, comprising:
 determining mention counts for the text entities, the mention counts indicative of frequencies that the text entities are mentioned in the collection of documents;   classifying entities with mention counts in the collection of documents less than a threshold number of times as specific entities;   classifying entities with mention counts exceeding the threshold number of times as non-specific entities;   extracting interesting context terms for the domain based on surrounding terms of the specific entities within the collection of documents and affinity counts of surrounding terms across the collection of documents; and   providing results that identify which of the documents are related to the domain based on the specific entities and the interesting context terms.   
     
     
         19 . The one or more storage memory of  claim 18 , wherein said extracting the interesting context terms comprises obtaining the affinity counts of candidate context terms over the collection of documents and eliminating the candidate context terms that appear frequently more than a threshold number of times in the collection of documents. 
     
     
         20 . The one or more storage memory of  claim 18  further comprising eliminating candidate context terms that are as likely to be mentioned within a context of a specific entity as mentioned outside the context.