Systems and methods for categorizing domains using artificial intelligence

Abstract

In an embodiment, a set of labeled training data that includes indicators of webpages is received. Each indicated webpage is labeled with one or more categories that were determined for the webpage by a human reviewer. Features, such as text and scripts, are extracted from each indicated webpage, and are used along with the labels to train a classifier to predict one or more categories for a webpage based on the features of the webpage. The trained classifier may be used to associate one or more categories with each domain of a plurality of domains given the categories predicted for some or all of the webpages associated with the domain. A list of domains and associated categories may be used for a variety of purposes including search engine optimization and content filtering.

1 . A method for training a classifier comprising:
 receiving a training set of webpages by a computing device, wherein each webpage in the training set is labeled with a category of a first plurality of categories;   retrieving a second plurality of categories stored on the computing device by the computing device, wherein the second plurality of categories has fewer categories than the first plurality of categories;   for each webpage of the training set of webpages that is labeled with a category that is in the first plurality of categories but not in the second plurality of categories, replacing the category that the webpage is labeled with using a category from the second plurality of categories by the computing device;   for each webpage of the training set of webpages, extracting one or more features from the webpage by the computing device, wherein the one or more features comprise script features; and   for each webpage of the training set of webpages, training a neural network classifier using the one or more extracted features and the category that the webpage is labeled with by the computing device.   
     
     
         2 . The method of  claim 1 , wherein each category of the first plurality of categories and the second plurality of categories relates to a topic or a subject, and the categories of the second plurality of categories are more general and/or generic than the categories of the first plurality of categories. 
     
     
         3 . The method of  claim 1 , wherein the one or more features further comprise video features or image features. 
     
     
         4 . The method of  claim 1 , further comprising:
 for each domain of a plurality of domains:
 retrieving a set of webpages from the domain by the computing device; 
 for each webpage of the set of webpages:
 extracting one or more features from the webpage of the set of webpages by the computing device; and 
 associating a category of the second plurality of categories with the webpage using the neural network classifier and the one or more features extracted from the webpage by the computing device. 
 
   
     
     
         5 . The method of  claim 4 , further comprising:
 for each domain of the plurality of domains, associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain.   
     
     
         6 . The method of  claim 5 , wherein associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain comprises:
 determining each category associated with more than a threshold percentage of webpages of the set of webpages; and   associating the determined categories with the domain.   
     
     
         7 . The method of  claim 6 , wherein each category is associated with a different threshold percentage and is identified by the neural network. 
     
     
         8 . A system for training a classifier comprising:
 at least one processor; and   a computer-readable medium storing computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
 receive a training set of webpages, wherein each webpage in the training set is labeled with a category of a first plurality of categories; 
 retrieve a second plurality of categories stored on the system, wherein the second plurality of categories has fewer categories than the first plurality of categories; 
 for each webpage of the training set of webpages that is labeled with a category that is in the first plurality of categories but not in the second plurality of categories, replacing the category that the webpage is labeled with using a category from the second plurality of categories; 
 for each webpage of the training set of webpages, extract one or more features from the webpage, wherein the one or more features comprise script features; and 
 for each webpage of the training set of webpages, train a neural network classifier using the one or more extracted features and the category that the webpage is labeled with. 
   
     
     
         9 . The system of  claim 8 , wherein each category of the first plurality of categories and the second plurality of categories relates to a topic or a subject, and the categories of the second plurality of categories are more general than the categories of the first plurality of categories. 
     
     
         10 . The system of  claim 8 , wherein the one or more features further comprise video features or image features. 
     
     
         11 . The system of  claim 8 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
 for each domain of a plurality of domains:
 retrieve a set of webpages from the domain; 
 for each webpage of the set of webpages:
 extract one or more features from the webpage of the set of webpages; and 
 associate a category of the second plurality of categories with the webpage using the neural network classifier and the one or more features extracted from the webpage. 
 
   
     
     
         12 . The system of  claim 11 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
 for each domain of the plurality of domains, associate a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain.   
     
     
         13 . The system of  claim 12 , wherein associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain comprises:
 determining each category associated with more than a threshold percentage of webpages of the set of webpages; and   associating the determined categories with the domain.   
     
     
         14 . The system of  claim 13 , wherein each category is associated with a different threshold percentage and is identified by the neural network. 
     
     
         15 . A non-transitory computer-readable medium storing computer executable instructions stored therefore that when executed by at least one processor a system to:
 receive a training set of webpages, wherein each webpage in the training set is labeled with a category of a first plurality of categories;   retrieve a second plurality of categories stored on the system, wherein the second plurality of categories has fewer categories than the first plurality of categories;   for each webpage of the training set of webpages that is labeled with a category that is in the first plurality of categories but not in the second plurality of categories, replacing the category that the webpage is labeled with using a category from the second plurality of categories;   for each webpage of the training set of webpages, extract one or more features from the webpage, wherein the one or more features comprise script features; and   for each webpage of the training set of webpages, train a neural network classifier using the one or more extracted features and the category that the webpage is labeled with.   
     
     
         16 . The non-transitory computer-readable medium of  claim 15 , wherein each category of the first plurality of categories and the second plurality of categories relates to a topic or a subject, and the categories of the second plurality of categories are more general and/or generic than the categories of the first plurality of categories. 
     
     
         17 . The non-transitory computer-readable medium of  claim 15 , wherein the one or more features further comprise video features or image features. 
     
     
         18 . The non-transitory computer-readable medium of  claim 15 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
 for each domain of a plurality of domains:
 retrieve a set of webpages from the domain; 
 for each webpage of the set of webpages: 
 extract one or more features from the webpage of the set of webpages; and 
 associate a category of the second plurality of categories with the webpage using the neural network classifier and the one or more features extracted from the webpage. 
   
     
     
         19 . The non-transitory computer-readable medium of  claim 18 , further comprising computer executable instructions stored therefore that when executed by the at least one processor cause the system to:
 for each domain of the plurality of domains, associate a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain.   
     
     
         20 . The non-transitory computer-readable medium of  claim 19 , wherein associating a category of the second plurality of categories with the domain based on the category associated with each webpage of the set of webpages from the domain comprises:
 determining each category associated with more than a threshold percentage of webpages of the set of webpages; and   associating the determined categories with the domain.