System and method for large-scale multi-label learning using incomplete label assignments

Abstract

At least one label prediction model is trained, or learned, using training data that may comprise training instances that may be missing one or more labels. The at least one label prediction model may be used in identifying a content item's ground-truth label set comprising an indicator for each label in the label set indicating whether or not the label is applicable to the content item.

1 . A method comprising:
 training, by a computing device, a multi-level label prediction model to predict an applicability, to a digital content item, of each label of a plurality of labels, the multi-level label prediction model comprising a first level and a second level, the first level being trained using training data missing a number of applicable labels, label prediction output of the trained first level including an applicability prediction for each of the number of applicable labels missing from the training data, the second level being trained using the label prediction output generated by the trained first level;   generating, by the computing device, the digital content item's label prediction output using the trained multi-level label prediction model, the digital content item's labeling prediction output indicating an applicability, to the digital content item, of each label of the plurality of labels; and   automatically generating, by the computing device, an annotation for the digital content item based on the digital content item's labeling prediction output.   
     
     
         2 . The method of  claim 1 , comprising:
 identifying, via the computing device, a set of digital content items using the digital content item's annotation.   
     
     
         3 . The method of  claim 2 , each digital content item in the set of digital content items having a respective annotation generated using label prediction output from the trained multi-level label prediction model. 
     
     
         4 . The method of  claim 2 , identifying the set of digital content items is responsive to a received search request. 
     
     
         5 . The method of  claim 4 , the received search request comprising information indicating the digital content item. 
     
     
         6 . The method of  claim 1 , the training data comprising a plurality of training instances corresponding to a plurality of digital content items, labeling data corresponding to at least one training instance is missing at least one applicable label from the plurality of labels. 
     
     
         7 . The method of  claim 6 , a training instance, of the plurality of training instances, comprising associated labeling data and feature data. 
     
     
         8 . The method of  claim 7 , further comprising:
 analyzing, a digital content item of the plurality of digital content items, and based on the analysis, generating the feature data for the analyzed digital content item.   
     
     
         9 . The method of  claim 1 , the number of levels of the multi-level label prediction model is empirically determined, a final level of the multi-level label prediction model being identified based on a determined convergence in the label prediction output generated by the final level and a potential next level. 
     
     
         10 . The method of  claim 1 , the digital content item is a document and the annotation comprises a number of words contained in the document. 
     
     
         11 . The method of  claim 1 , the digital content item comprises an image. 
     
     
         12 . A computer readable non-transitory storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:
 training a multi-level label prediction model to predict an applicability, to a digital content item, of each label of a plurality of labels, the multi-level label prediction model comprising a first level and a second level, the first level being trained using training data missing a number of applicable labels, label prediction output of the trained first level including an applicability prediction for each of the number of applicable labels missing from the training data, the second level being trained using the label prediction output generated by the trained first level;   generating the digital content item's label prediction output using the trained multi-level label prediction model, the digital content item's labeling prediction output indicating an applicability, to the digital content item, of each label of the plurality of labels; and   automatically generating annotation for the digital content item based on the digital content item's labeling prediction output.   
     
     
         13 . The computer readable non-transitory storage medium of  claim 12 , the method further comprising:
 identifying a set of digital content items using the digital content item's annotation.   
     
     
         14 . The computer readable non-transitory storage medium of  claim 13 , each digital content item in the set of digital content items having a respective annotation generated using label prediction output from the trained multi-level label prediction model. 
     
     
         15 . The computer readable non-transitory storage medium of  claim 13 , identifying the set of digital content items is responsive to a received search request. 
     
     
         16 . The computer readable non-transitory storage medium of  claim 15 , the received search request comprising information indicating the digital content item. 
     
     
         17 . The computer readable non-transitory storage medium of  claim 12 , the training data comprising a plurality of training instances corresponding to a plurality of digital content items, labeling data corresponding to at least one training instance is missing at least one applicable label from the plurality of labels. 
     
     
         18 . The computer readable non-transitory storage medium of  claim 17 , a training instance, of the plurality of training instances, comprising associated labeling data and feature data. 
     
     
         19 . The computer readable non-transitory storage medium of  claim 18 , further comprising:
 analyzing, a digital content item of the plurality of digital content items, and based on the analysis, generating the feature data for the analyzed digital content item.   
     
     
         20 . A system comprising:
 a processor;   a storage medium for tangibly storing thereon program logic for execution by the processor, the stored logic comprising:
 training logic executed by the processor for training a multi-level label prediction model to predict an applicability, to a digital content item, of each label of a plurality of labels, the multi-level label prediction model comprising a first level and a second level, the first level being trained using training data missing a number of applicable labels, label prediction output of the trained first level including an applicability prediction for each of the number of applicable labels missing from the training data, the second level being trained using the label prediction output generated by the trained first level; 
 generating logic executed by the processor for generating the digital content item's label prediction output using the trained multi-level label prediction model, the digital content item's labeling prediction output indicating an applicability, to the digital content item, of each label of the plurality of labels; and 
 generating logic executed by the processor for automatically generating annotation for the digital content item based on the digital content item's labeling prediction output.