Active learning via a sample consistency assessment

Abstract

A method includes obtaining a set of unlabeled training samples. For each training sample in the set of unlabeled training samples generating, the method includes using a machine learning model and the training sample, a corresponding first prediction, generating, using the machine learning model and a modified unlabeled training sample, a second prediction, the modified unlabeled training sample based on the training sample, and determining a difference between the first prediction and the second prediction. The method includes selecting, based on the differences, a subset of the set of unlabeled training samples. For each training sample in the subset of the set of unlabeled training samples, the method includes obtaining a ground truth label for the training sample, and generating a corresponding labeled training sample based on the training sample paired with the ground truth label. The method includes training the machine learning model using the corresponding labeled training samples.

What is claimed is: 
     
         1 . A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
 obtaining a set of unlabeled training samples;   for each particular unlabeled training sample in the set of unlabeled training samples:
 generating, using a machine learning model and the particular unlabeled training sample, a corresponding first prediction; 
 generating, using the machine learning model and a modified unlabeled training sample, a corresponding second prediction, the modified unlabeled training sample based on the particular unlabeled training sample; and 
 determining a corresponding difference between the corresponding first prediction and the corresponding second prediction; 
   selecting, based on the corresponding differences, a subset of the set of unlabeled training samples;   for each particular unlabeled training sample in the subset of the set of unlabeled training samples:
 obtaining a corresponding ground truth label for the particular unlabeled training sample; and 
 generating a corresponding labeled training sample based on the particular unlabeled training sample paired with the corresponding ground truth label; and 
   training the machine learning model using the corresponding labeled training samples.   
     
     
         2 . The method of  claim 1 , wherein a number of unlabeled training samples in the subset of the set of unlabeled training samples is less than a cardinality of the set of unlabeled training samples. 
     
     
         3 . The method of  claim 1 , wherein selecting, based on the corresponding differences, the subset of the set of unlabeled training samples comprises selecting unlabeled training samples having a corresponding difference satisfying a threshold. 
     
     
         4 . The method of  claim 1 , wherein selecting, based on the corresponding differences, the subset of the set of unlabeled training samples comprises selecting a threshold number of the unlabeled training samples having the largest corresponding differences. 
     
     
         5 . The method of  claim 1 , wherein the operations further comprise, during an initial active learning cycle:
 randomly selecting a random set of unlabeled training samples from the set of unlabeled training samples;   for each particular unlabeled training sample in the random set of unlabeled training samples, obtaining a corresponding ground truth label; and   training the machine learning model using the random set of unlabeled training samples and the corresponding ground truth labels.   
     
     
         6 . The method of  claim 5 , wherein the operations further comprise, during the initial active learning cycle:
 identifying a candidate set of unlabeled training samples from the set of unlabeled training samples, wherein a cardinality of the candidate set of unlabeled training samples is less than a cardinality of the set of unlabeled training samples;   determining a first cross entropy between a distribution of ground truth labels and a distribution of predicted labels generated using the machine learning model for the unlabeled training samples in the candidate set of unlabeled training samples;   determining a second cross entropy between a distribution of ground truth labels and a distribution of predicted labels generated using the machine learning model for the unlabeled training samples in the set of unlabeled training samples;   determining that the first cross entropy is greater than or equal to the second cross entropy; and   based on determining that the first cross entropy is greater than or equal to the second cross entropy, selecting the candidate set of unlabeled training samples as a starting size for initially training the machine learning model.   
     
     
         7 . The method of  claim 6 , wherein identifying the candidate set of unlabeled training samples from the set of unlabeled training samples comprises determining the corresponding difference for each unlabeled training sample of the set of unlabeled training samples. 
     
     
         8 . The method of  claim 7 , wherein the operations further comprise, when the first cross entropy is less than the second cross entropy:
 randomly selecting an expanded set of unlabeled training samples from the set of unlabeled training samples;   updating the candidate set of unlabeled training samples to include the expanded set of unlabeled training samples randomly selected from the set of unlabeled training samples;   updating the set of unlabeled training samples by removing each unlabeled training sample from the expanded set of unlabeled training samples from the set of unlabeled training samples; and   during an immediately subsequent active learning cycle:
 determining the first cross entropy between a distribution of ground truth labels and a distribution of predicted labels generated using the machine learning model for the unlabeled training samples in the updated candidate set of unlabeled training samples; 
 determining the second cross entropy between the distribution of ground truth labels and a distribution of predicted labels generating using the machine learning model for the unlabeled training samples in the updated candidate set of unlabeled training samples; 
 determining that the first cross entropy is greater than or equal to the second cross entropy; and 
 based on determining that the first cross entropy is greater than or equal to the second cross entropy, selecting a size of the updated candidate set of unlabeled training samples as a starting size for initially training the machine learning model. 
   
     
     
         9 . The method of  claim 1 , wherein the machine learning model comprises a convolutional neural network. 
     
     
         10 . The method of  claim 1 , wherein the corresponding difference between the corresponding first prediction and the corresponding second prediction represents a variance between the corresponding first prediction and the corresponding second prediction. 
     
     
         11 . A system comprising:
 data processing hardware; and   memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
 obtaining a set of unlabeled training samples; 
 for each particular unlabeled training sample in the set of unlabeled training samples:
 generating, using a machine learning model and the particular unlabeled training sample, a corresponding first prediction; 
 generating, using the machine learning model and a modified unlabeled training sample, a corresponding second prediction, the modified unlabeled training sample based on the particular unlabeled training sample; and 
 determining a corresponding difference between the corresponding first prediction and the corresponding second prediction; 
 
 selecting, based on the corresponding differences, a subset of the set of unlabeled training samples; 
 for each particular unlabeled training sample in the subset of the set of unlabeled training samples:
 obtaining a corresponding ground truth label for the particular unlabeled training sample; and 
 generating a corresponding labeled training sample based on the particular unlabeled training sample paired with the corresponding ground truth label; and 
 
 training the machine learning model using the corresponding labeled training samples. 
   
     
     
         12 . The system of  claim 11 , wherein a number of unlabeled training samples in the subset of the set of unlabeled training samples is less than a cardinality of the set of unlabeled training samples. 
     
     
         13 . The system of  claim 11 , wherein selecting, based on the corresponding differences, the subset of the set of unlabeled training samples comprises selecting unlabeled training samples having a corresponding difference satisfying a threshold. 
     
     
         14 . The system of  claim 11  wherein selecting, based on the corresponding differences, the subset of the set of unlabeled training samples comprises selecting a threshold number of the unlabeled training samples having the largest corresponding differences. 
     
     
         15 . The system of  claim 11 , wherein the operations further comprise, during an initial active learning cycle:
 randomly selecting a random set of unlabeled training samples from the set of unlabeled training samples;   for each particular unlabeled training sample in the random set of unlabeled training samples, obtaining a corresponding ground truth label; and   training the machine learning model using the random set of unlabeled training samples and the corresponding ground truth labels.   
     
     
         16 . The system of  claim 15 , wherein the operations further comprise, during the initial active learning cycle:
 identifying a candidate set of unlabeled training samples from the set of unlabeled training samples, wherein a cardinality of the candidate set of unlabeled training samples is less than a cardinality of the set of unlabeled training samples;   determining a first cross entropy between a distribution of ground truth labels and a distribution of predicted labels generated using the machine learning model for the unlabeled training samples in the candidate set of unlabeled training samples;   determining a second cross entropy between a distribution of ground truth labels and a distribution of predicted labels generated using the machine learning model for the unlabeled training samples in the set of unlabeled training samples;   determining that the first cross entropy is greater than or equal to the second cross entropy; and   based on determining that the first cross entropy is greater than or equal to the second cross entropy, selecting the candidate set of unlabeled training samples as a starting size for initially training the machine learning model.   
     
     
         17 . The system of  claim 16 , wherein identifying the candidate set of unlabeled training samples from the set of unlabeled training samples comprises determining the corresponding difference for each unlabeled training sample of the set of unlabeled training samples. 
     
     
         18 . The system of  claim 17 , further comprising, when the first cross entropy is less than the second cross entropy:
 randomly selecting an expanded set of unlabeled training samples from the set of unlabeled training samples;   updating the candidate set of unlabeled training samples to include the expanded set of unlabeled training samples randomly selected from the set of unlabeled training samples;   updating the set of unlabeled training samples by removing each unlabeled training sample from the expanded set of unlabeled training samples from the set of unlabeled training samples; and   during an immediately subsequent active learning cycle:
 determining the first cross entropy between a distribution of ground truth labels and a distribution of predicted labels generated using the machine learning model for the unlabeled training samples in the updated candidate set of unlabeled training samples; 
 determining the second cross entropy between the distribution of ground truth labels and a distribution of predicted labels generating using the machine learning model for the unlabeled training samples in the updated candidate set of unlabeled training samples; 
 determining that the first cross entropy is greater than or equal to the second cross entropy; and 
 based on determining that the first cross entropy is greater than or equal to the second cross entropy, selecting a size of the updated candidate set of unlabeled training samples as a starting size for initially training the machine learning model. 
   
     
     
         19 . The system of  claim 11 , wherein the machine learning model comprises a convolutional neural network. 
     
     
         20 . The system of  claim 11 , wherein the corresponding difference between the corresponding first prediction and the corresponding second prediction represents a variance between the corresponding first prediction and the corresponding second prediction.