Systems and Methods for Protecting Against Exposure to Content Violating a Content Policy

Abstract

A method for protecting against exposure to content violating a content policy, the method including receiving a number of content items including a first set of content items associated with a content group, determining a measurement associated with an amount of the first set of content items belonging to a specific content category, assigning one or more of the number of content items to be categorized by at least one of the machine learning algorithm or a manual review process, automatically applying the specific content category to one or more other content items of the content group such that the one or more other content items are not reviewed by the manual review process, and transmitting at least one of the number of content items, wherein the content category of each of the number of content items indicates whether the specific content item violates any content policies.

What is claimed is: 
     
         1 - 20 . (canceled) 
     
     
         21 . A method for moderating content, the method comprising:
 receiving, by a computing system comprising one or more processors, a plurality of content items for service to end user devices in conjunction with information resources;   moderating the content of the plurality of content items by, for each respective group of one or more groups of content items of the plurality of content items:
 obtaining, by the computing system, labels respectively generated for one or more content items of the respective group using a machine-learned model, the labels indicating an association of the one or more content items with a specific content category; 
 based on a comparison of a threshold measurement and a measurement of the one or more content items, automatically associating, by the computing system, the specific content category with all of the respective group; and 
 based on the association of the respective group with the specific content category, determining, by the computing system and based on a content policy, a respective set of information resources with which the respective group may be served; and 
   transmitting, by the computing system, at least one of the plurality of content items for service to an end user device in conjunction with an information resource associated with the respective set of information resources with which the at least one content item may be served.   
     
     
         22 . The method of  claim 21 , wherein the machine-learned model is configured to classify the one or more content items into categories that indicate what the one or more content items depict. 
     
     
         23 . The method of  claim 21 , comprising:
 obtaining, by the computing system, second labels respectively generated for one or more second content items of a second group using the machine-learned model, the second labels indicating an association of the one or more second content items with a second specific content category; and   based on a comparison of the threshold measurement and a measurement of the one or more second content items, sending the one or more second content items to a manual review system for review.   
     
     
         24 . The method of  claim 21 , wherein at least one of the one or more groups is formed from items sharing at least one characteristic selected from: a source, a category, a purpose, a medium of presentation, an intended presentation period, or a geographic association. 
     
     
         25 . The method of  claim 21 , wherein different content categories are associated with different threshold measurements. 
     
     
         26 . The method of  claim 21 , comprising:
 dynamically determining, by the computing system, the threshold measurement.   
     
     
         27 . The method of  claim 26 , comprising:
 determining, by the computing system, the labeling outcomes of a number of potential threshold values associated with each label to identify the threshold measurement.   
     
     
         28 . A computing system comprising:
 one or more processors; and   one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
 receiving a plurality of content items for service to end user devices in conjunction with information resources; 
 moderating the content of the plurality of content items by, for each respective group of one or more groups of content items of the plurality of content items:
 obtaining labels respectively generated for one or more content items of the respective group using a machine-learned model, the labels indicating an association of the one or more content items with a specific content category; 
 based on a comparison of a threshold measurement and a measurement of the one or more content items, automatically associating the specific content category with all of the respective group; and 
 based on the association of the respective group with the specific content category, determining, based on a content policy, a respective set of information resources with which the respective group may be served; and 
 
 transmitting at least one of the plurality of content items for service to an end user device in conjunction with an information resource associated with the respective set of information resources with which the at least one content item may be served. 
   
     
     
         29 . The computing system of  claim 28 , wherein the machine-learned model is configured to classify the one or more content items into categories that indicate what the one or more content items depict. 
     
     
         30 . The computing system of  claim 28 , the operations comprising:
 obtaining second labels respectively generated for one or more second content items of a second group using the machine-learned model, the second labels indicating an association of the one or more second content items with a second specific content category; and   based on a comparison of the threshold measurement and a measurement of the one or more second content items, sending the one or more second content items to a manual review system for review.   
     
     
         31 . The computing system of  claim 28 , wherein at least one of the one or more groups is formed from items sharing at least one characteristic selected from: a source, a category, a purpose, a medium of presentation, an intended presentation period, or a geographic association. 
     
     
         32 . The computing system of  claim 28 , wherein different content categories are associated with different threshold measurements. 
     
     
         33 . The computing system of  claim 28 , the operations comprising:
 dynamically determining the threshold measurement.   
     
     
         34 . The computing system of  claim 33 , the operations comprising:
 determining the labeling outcomes of a number of potential threshold values associated with each label to identify the threshold measurement.   
     
     
         35 . One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to cause a computing system to perform operations, the operations comprising:
 receiving a plurality of content items for service to end user devices in conjunction with information resources;   moderating the content of the plurality of content items by, for each respective group of one or more groups of content items of the plurality of content items:
 obtaining labels respectively generated for one or more content items of the respective group using a machine-learned model, the labels indicating an association of the one or more content items with a specific content category; 
 based on a comparison of a threshold measurement and a measurement of the one or more content items, automatically associating the specific content category with all of the respective group; and 
 based on the association of the respective group with the specific content category, determining, based on a content policy, a respective set of information resources with which the respective group may be served; and 
   transmitting at least one of the plurality of content items for service to an end user device in conjunction with an information resource associated with the respective set of information resources with which the at least one content item may be served.   
     
     
         36 . The one or more non-transitory computer-readable media of  claim 35 , wherein the machine-learned model is configured to classify the one or more content items into categories that indicate what the one or more content items depict. 
     
     
         37 . The one or more non-transitory computer-readable media of  claim 35 , wherein at least one of the one or more groups is formed from items sharing at least one characteristic selected from: a source, a category, a purpose, a medium of presentation, an intended presentation period, or a geographic association. 
     
     
         38 . The one or more non-transitory computer-readable media of  claim 35 , wherein different content categories are associated with different threshold measurements. 
     
     
         39 . The one or more non-transitory computer-readable media of  claim 35 , the operations comprising:
 dynamically determining the threshold measurement.   
     
     
         40 . The one or more non-transitory computer-readable media of  claim 39 , the operations comprising:
 determining the labeling outcomes of a number of potential threshold values associated with each label to identify the threshold measurement.