Method, apparatus, and computer program product for classification of documents

Abstract

Provided herein are systems, methods and computer readable media for classification of documents using a location hierarchy. An example method may include receiving a feature vector r that represents occurrence counts of references in a document's text to each of a group of named entities, and determining whether the document is associated with the particular location by querying, to determine a query result, using feature vector r, at least one location-specific classifier from a group of location-specific classifiers, wherein the location-specific classifier is associated with the particular location, and wherein the location-specific classifier is configured to generate a positive output value in response to receiving an input feature vector representing occurrence count of at least one reference to the particular named entity and determining that the document is associated with the particular location in an instance in which the query result includes data indicating that the positive output value was generated by the location-specific classifier that is associated with the particular location.

1 - 30 . (canceled) 
     
     
         31 . An apparatus comprising one or more processors and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the one or more processors, cause the apparatus to:
 parse a dequeued work item using a markup language parser to extract a uniform resource locator (URL);   retrieve a web document using the URL;   determine, based at least in part on a web document, one or more candidate images for creation of a representative image by:
 extracting, by identifying image tags with source attributes, one or more image references from the parsed dequeued work item; 
 filtering the one or more image references using a blacklist of expressions designed to match those one or more image references comprising one or more predefined undesirable characteristics; 
 generating a candidate images set by retrieving those one or more image references which do not match any of the expressions; and 
 updating the candidate images set based at least in part on determining, for each candidate image of the candidate images set, if the retrieved image is a duplicate image; and 
   generate a representative image by modifying a chosen image selected from among the candidate images set in accordance with one or more desired characteristics and an environment in which the representative image will be used.   
     
     
         32 . The apparatus of  claim 31 , wherein determining if the retrieved image is a duplicate image comprises:
 computing a retrieved image cryptographic hash for the retrieved image; and   determining if the retrieved image cryptographic hash matches a previously stored retrieved image cryptographic hash.   
     
     
         33 . The apparatus of  claim 32 , further caused to:
 responsive to a determination of a match, exclude the retrieved image from the candidate images set.   
     
     
         34 . The apparatus of  claim 33 , further caused to:
 responsive to a determination of no match, store the retrieved image cryptographic hash.   
     
     
         35 . The apparatus of  claim 31 , further caused to:
 use an HTTP client to retrieve those one or more image references which do not match any of the expressions.   
     
     
         36 . The apparatus of  claim 31 , wherein values of the image references comprise URLs locating images. 
     
     
         37 . The apparatus of  claim 31 , wherein modifying the chosen image selected from among the candidate images set comprises cropping the chosen image to a predefined size. 
     
     
         38 . The apparatus of  claim 31 , further caused to:
 extract one or more video references from the parsed dequeued work item;   retrieve the videos using an HTTP client;   decode the retrieved videos; and   add the extracted image frames from the videos from predefined points in the video to the candidate images set.   
     
     
         39 . The apparatus of  claim 38 , wherein the one or more video references are extracted by identifying video tags containing source tags with source attributes, embed tags with source attributes, or object tags with data attributes, values of which are URLs locating videos. 
     
     
         40 . The apparatus of  claim 38 , wherein a video retriever stops retrieval of a video after receiving an initial segment of a predefined minimum size. 
     
     
         41 . A computer-implemented method, comprising:
 parsing a dequeued work item using a markup language parser to extract a uniform resource locator (URL);   retrieving a web document using the URL;   determining, based at least in part on a web document, one or more candidate images for creation of a representative image by:
 extracting, by identifying image tags with source attributes, one or more image references from the parsed dequeued work item; 
 filtering the one or more image references using a blacklist of expressions designed to match those one or more image references comprising one or more predefined undesirable characteristics; 
 generating a candidate images set by retrieving those one or more image references which do not match any of the expressions; and 
 updating the candidate images set based at least in part on determining, for each candidate image of the candidate images set, if the retrieved image is a duplicate image; and 
   generating a representative image by modifying a chosen image selected from among the candidate images set in accordance with one or more desired characteristics and an environment in which the representative image will be used.   
     
     
         42 . The computer-implemented method of  claim 41 , wherein determining if the retrieved image is a duplicate image comprises:
 computing a retrieved image cryptographic hash for the retrieved image; and   determining if the retrieved image cryptographic hash matches a previously stored retrieved image cryptographic hash.   
     
     
         43 . The computer-implemented method of  claim 42 , further comprising:
 responsive to a determination of a match, exclude the retrieved image from the candidate images set.   
     
     
         44 . The computer-implemented method of  claim 43 , further comprising:
 responsive to a determination of no match, storing the retrieved image cryptographic hash.   
     
     
         45 . The computer-implemented method of  claim 41 , further comprising:
 use an HTTP client to retrieve those one or more image references which do not match any of the expressions.   
     
     
         46 . The computer-implemented method of  claim 41 , wherein values of the image references comprise URLs locating images. 
     
     
         47 . The computer-implemented method of  claim 41 , wherein modifying the chosen image selected from among the candidate images set comprises cropping the chosen image to a predefined size. 
     
     
         48 . The computer-implemented method of  claim 41 , further comprising:
 extracting one or more video references from the parsed dequeued work item;   retrieving the videos using an HTTP client;   decoding the retrieved videos; and   adding the extracted image frames from the videos from predefined points in the video to the candidate images set.   
     
     
         49 . The computer-implemented method of  claim 48 , wherein the one or more video references are extracted by identifying video tags containing source tags with source attributes, embed tags with source attributes, or object tags with data attributes, values of which are URLs locating videos. 
     
     
         50 . The computer-implemented method of  claim 48 , wherein a video retriever stops retrieval of a video after receiving an initial segment of a predefined minimum size. 
     
     
         51 . A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, configured to:
 parse a dequeued work item using a markup language parser to extract a uniform resource locator (URL);   retrieve a web document using the URL;   determine, based at least in part on a web document, one or more candidate images for creation of a representative image by:
 extracting, by identifying image tags with source attributes, one or more image references from the parsed dequeued work item; 
 filtering the one or more image references using a blacklist of expressions designed to match those one or more image references comprising one or more predefined undesirable characteristics; 
 generating a candidate images set by retrieving those one or more image references which do not match any of the expressions; and 
 updating the candidate images set based at least in part on determining, for each candidate image of the candidate images set, if the retrieved image is a duplicate image; and 
   generate a representative image by modifying a chosen image selected from among the candidate images set in accordance with one or more desired characteristics and an environment in which the representative image will be used.