Machine Learning Systems and Methods for Using Perturbation to Identify High Value Vector Labeling Targets

Abstract

Among a great deal of other disclosure and scope, systems and methods are enclosed that adapt adversarial learning principles to an active learning regime. Given a problem space of note, a set of labeled vectors, a machine learning model trained on the set of labeled vectors, and a set of unlabeled vectors, we identify the unlabeled vectors our model is most unsure of. Each of our unlabeled vectors in our set of unlabeled vectors is initially classified by our model, and the prediction probabilities are taken note of. Then, each of our unlabeled vectors in our set of unlabeled vectors is perturbed by adding some random noise. The perturbed vectors are reclassified by our model, with the prediction probabilities taken note of once again. The vectors with the greatest divergence in prediction probabilities before and after perturbation are those that lie in regions within the problem space our model has little to no confidence and knowledge regarding and are sent to our oracle first in order to improve our labeled set of vectors.

What is claimed is: 
     
         1 . A method for identifying a high value labeling target, the method comprising:
 receiving, by a processing resource, a first set of data elements including at least a first data element and a second data element;   applying, by the processing resource, a mathematical model to the first set of data elements to yield at least: a first predicted output corresponding to the first data element, and a second predicted output corresponding to the second data element;   adding, by the processing resource, a perturbation to the first data element to yield a perturbed data element;   applying, by the processing resource, the mathematical model to a second set of data elements including the perturbed data element to yield at least a third predicted output corresponding to the perturbed data element; and   using, by the processing resource, a combination of the first predicted output and the third predicted output to determine a labeling value of the first data element.   
     
     
         2 . The method of  claim 1 , wherein the first data element is a first vector, wherein the second data element is a second vector, and wherein the set of data elements is a set of vectors. 
     
     
         3 . The method of  claim 1 , wherein using the combination of the first predicted output and the third predicted output to determine the labeling value of the first data element includes:
 calculating, by the processing resource, divergence of the first predicted output to yield a first divergence;   calculating, by the processing resource, divergence of the third predicted output to yield a second divergence; and   using, by the processing resource, a combination of the first divergence and the second divergence to determine a labeling value of the first data element.   
     
     
         4 . The method of  claim 3 , wherein both the first divergence and the second divergence are calculated using a Kullback-Leibler algorithm in accordance with the following equation:
     D   KL ( p ( | x )∥ p ( | x +∈).
   
     
     
         5 . The method of  claim 3 , wherein using the combination of the first divergence and the second divergence to determine a labeling value of the first data element includes:
 calculating, by the processing resource, a difference between the first divergence and the second divergence to yield a divergence difference; and   comparing, by the processing resource, the divergence difference to a threshold value, wherein upon determining that the divergence difference exceeds the threshold value, the first data element is identified as a high value labeling target.   
     
     
         6 . The method of  claim 3 , wherein the perturbed data element is a first perturbed data element, the method further comprising:
 calculating, by the processing resource, divergence of the second predicted output to yield a third divergence;   adding, by the processing resource, the perturbation to the second data element to yield a second perturbed data element, wherein the second set of data elements includes the second perturbed data element, and wherein applying the mathematical model to the second set of data elements further yields: a fourth predicted output corresponding to the second perturbed data element;   calculating, by the processing resource, divergence of the fourth predicted output to yield a fourth divergence; and   using, by the processing resource, a combination of the third divergence and the fourth divergence to determine a labeling value of the second data element.   
     
     
         7 . The method of  claim 1 , wherein the first predicted output is a first class, wherein the third predicted output is a second class, and wherein using the combination of the first predicted output and the third predicted output to determine labeling value of the first data element includes:
 identifying, by the processing resource, the first data element as a high value labeling target where the first class is different from the second class.   
     
     
         8 . The method of  claim 1 , wherein the first predicted output is a first class, wherein the second predicted output is a second class, and wherein using the combination of the first predicted output and the second predicted output to determine labeling value of the first data element includes:
 identifying, by the processing resource, the first data element as a low value labeling target where the first class is the same as the second class.   
     
     
         9 . The method of  claim 2 , the method further comprising:
 using, by the processing resource, the labeling value of the first vector along with the result of at least one other heuristic to rank the first vector relative to the second vector.   
     
     
         10 . The method of  claim 9 , wherein the at least one other heuristic is selected from a group consisting of: a Shannon's entropy heuristic, a confidence based heuristic, an orthogonality to labelled points heuristic, a distance from decision hyperplane heuristic, an information density heuristic, an expected gradient length heuristic, and a consensus based heuristic. 
     
     
         11 . A system for identifying a high value labeling target, the system comprising:
 a processing resource;   a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to:
 receive a first set of data elements including at least a first data element and a second data element; 
 apply a mathematical model to the first set of data elements to yield at least: a first predicted output corresponding to the first data element, and a second predicted output corresponding to the second data element; 
 add a perturbation to the first data element to yield a perturbed data element; 
 apply the mathematical model to a second set of data elements including the perturbed data element to yield at least a third predicted output corresponding to the perturbed data element; and 
 use a combination of the first predicted output and the third predicted output to determine a labeling value of the first data element. 
   
     
     
         12 . The system of  claim 11 , wherein the instructions that when executed by the processing resource cause the processing resource to use the combination of the first predicted output and the third predicted output to determine the labeling value of the first data element include instructions that cause the processing resource to:
 calculate divergence of the first predicted output to yield a first divergence;   calculate divergence of the third predicted output to yield a second divergence; and   using a combination of the first divergence and the second divergence to determine a labeling value of the first data element.   
     
     
         13 . The system of  claim 12 , wherein both the first divergence and the second divergence are calculated using a Kullback-Leibler algorithm in accordance with the following equation:
     D   KL ( p ( | x )∥ p ( | x +∈).
   
     
     
         14 . The system of  claim 12 , wherein the instructions that when executed by the processing resource cause the processing resource to use the combination of the first divergence and the second divergence to determine a labeling value of the first data element include instructions that cause the processing resource to:
 calculate a difference between the first divergence and the second divergence to yield a divergence difference; and   compare the divergence difference to a threshold value, wherein upon determining that the divergence difference exceeds the threshold value, the first data element is identified as a high value labeling target.   
     
     
         15 . The system of  claim 12 , wherein the perturbed data element is a first perturbed data element, and wherein the non-transitory computer-readable medium, coupled to the processing resource, further has stored therein instructions that when executed by the processing resource cause the processing resource to:
 calculate divergence of the second predicted output to yield a third divergence;   add the perturbation to the second data element to yield a second perturbed data element, wherein the second set of data elements includes the second perturbed data element, and wherein applying the mathematical model to the second set of data elements further yields: a fourth predicted output corresponding to the second perturbed data element;   calculate divergence of the fourth predicted output to yield a fourth divergence; and   use a combination of the third divergence and the fourth divergence to determine a labeling value of the second data element.   
     
     
         16 . The system of  claim 11 , wherein the first predicted output is a first class, wherein the third predicted output is a second class, and wherein the instructions that when executed by the processing resource cause the processing resource to use the combination of the first predicted output and the third predicted output to determine the labeling value of the first data element include instructions that cause the processing resource to:
 identify the first data element as a high value labeling target where the first class is different from the second class.   
     
     
         17 . A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a computer system, causes the one or more processing resources to:
 receive a first set of data elements including at least a first data element and a second data element;   apply a mathematical model to the first set of data elements to yield at least: a first predicted output corresponding to the first data element, and a second predicted output corresponding to the second data element;   add a perturbation to the first data element to yield a perturbed data element;   apply the mathematical model to a second set of data elements including the perturbed data element to yield at least a third predicted output corresponding to the perturbed data element; and   use a combination of the first predicted output and the third predicted output to determine a labeling value of the first data element.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , wherein the set of instructions, which when executed by one or more processing resources of a computer system causes the one or more processing resources to use the combination of the first predicted output and the third predicted output to determine the labeling value of the first data element include instructions that cause the processing resource to:
 calculate divergence of the first predicted output to yield a first divergence;   calculate divergence of the third predicted output to yield a second divergence; and   using a combination of the first divergence and the second divergence to determine a labeling value of the first data element.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 18 , wherein both the first divergence and the second divergence are calculated using a Kullback-Leibler algorithm in accordance with the following equation:
     D   KL ( p ( | x )∥ p ( | x +∈).
   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 18 , wherein the set of instructions, which when executed by one or more processing resources of a computer system causes the one or more processing resources to use the combination of the first divergence and the second divergence to determine a labeling value of the first data element include instructions that cause the processing resource to:
 calculate a difference between the first divergence and the second divergence to yield a divergence difference; and   compare the divergence difference to a threshold value, wherein upon determining that the divergence difference exceeds the threshold value, the first data element is identified as a high value labeling target.