Machine Learning Systems and Methods For Dual Network Multi-Class Classification

Abstract

Among a great deal of other disclosure and scope, systems and methods are disclosed in relation to a dual network entity designed for classification in problem spaces where the target can be one of multiple possibilities with as few labeled training examples as possible. In one of many possible implementations, a network is first used to identify vectors considered to possess immense amounts of information regarding the problem space. An oracle is then tasked with labeling such vectors. The secondary network uses the new insights gleaned about the problem space to identify unlabeled vectors that our target model has correctly identified. These vectors are pseudolabeled, providing further information about the problem space to our target model. The cycle continues until the operator is satisfied with the performance of the target model.

What is claimed is: 
     
         1 . A method for modeling data, the method comprising:
 ranking, by a processing resource, a set of unlabelled data elements based upon an expected impact of each of the unlabelled data elements on operation of a first mathematical model to yield a subset of high ranked data elements, wherein the subset of high ranked data elements includes at least one unlabelled data element having a ranking that is higher than another data element of the set of data elements not included in the subset of high ranked data elements;   training, by the processing resource, the first mathematical model using a first dataset including both the subset of high ranked data elements and a set of previously labelled data elements to yield at least a first decision output and a first confidence value corresponding to the first decision output and a second decision output and a second confidence value corresponding to the second decision ouput;   applying, by the processing resource, a second mathematical model to a second dataset including at least the first decision output, wherein the second mathematical model provides an updated decision output corresponding to the first decision output; and   selecting, by the processing resource, the first decision output for labelling based at least in part on a combination of the first decision output and the updated decision output.   
     
     
         2 . The method of  claim 1 , the method further comprising:
 selecting, by the programming resource, the first decision output for inclusion in the second dataset based upon the first confidence value exceeding a confidence threshold value; and   excluding, by the programming resource, the second decision output from inclusion in the second dataset based upon the second confidence value being less than the confidence threshold value.   
     
     
         3 . The method of  claim 1 , the method further comprising:
 comparing, by the processing resource, the first decision output with one of the previously labelled decision ouputs to yield a comparison result; and   wherein selecting the first decision output for labelling is done based at least in part on the combination of the first decision output and the updated decision output, and upon the comparison result.   
     
     
         4 . The method of  claim 3 , wherein the comparison result indicates that the one of the previously labelled decision ouputs is similar to the first decision output. 
     
     
         5 . The method of  claim 1 , wherein selecting the first decision output for labelling based at least in part on the combination of the first decision output and the updated decision output includes:
 selecting, by the processing resource, the first decision output for labelling based at least in part on the first decision output matching the updated decision output.   
     
     
         6 . The method of  claim 1 , wherein ranking the set of unlabelled data elements based upon the expected impact of each of the unlabelled data elements on operation of the first mathematical model comprises:
 selecting, by the processing resource, an unlabelled data element from a set of unlabelled data elements to yield a selected, unlabelled data element, wherein non-selected unlabelled data elements in the set of unlabelled data elements are a non-selected set of unlabelled data elements;   selecting, by the processing resource, a subset of the non-selected set of unlabelled data elements;   merging, by the processing device, the selected, unlabelled data element with the subset of the non-selected set of unlabelled data elements to yield a merged, unlabelled dataset;   forming, by the processing device, a union of the merged, unlabelled dataset and a labelled dataset to yield a union dataset; and   calculating, by the processing resource, an expected performance value of the union dataset.   
     
     
         7 . The method of  claim 6 , wherein the set of unlabelled data elements is a set of unlabelled vectors, and wherein the labelled dataset is a set of labelled vectors, and wherein the method further comprises:
 generating the set of unlabelled vectors using a combination of at least a first heuristic and a second heuristic.   
     
     
         8 . The method of  claim 7 , wherein:
 the first heuristic is selected from a group consisting of: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an orthogonality to labelled points heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic; and   the second heuristic is different from the first heuristic and selected from a group consisting of: the Shannon's entropy heuristic, the confidence based heuristic, the distance from decision hyperplane heuristic, the orthogonality to labelled points heuristic, the information density heuristic, the perturbation heuristic, the expected gradient length heuristic, and the consensus based heuristic.   
     
     
         9 . The method of  claim 7 , the method further comprising:
 generating the set of unlabelled vectors using a combination of four or more of the following heuristics: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an orthogonality to labelled points heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic.   
     
     
         10 . A system for modeling data, the system comprising:
 a processing resource;   a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to:
 rank a set of unlabelled data elements based upon an expected impact of each of the unlabelled data elements on operation of a first mathematical model to yield a subset of high ranked data elements, wherein the subset of high ranked data elements includes at least one unlabelled data element having a ranking that is higher than another data element of the set of data elements not included in the subset of high ranked data elements; 
 train the first mathematical model using a first dataset including both the subset of high ranked data elements and a set of previously labelled data elements to yield at least a first decision output and a first confidence value corresponding to the first decision output and a second decision output and a second confidence value corresponding to the second decision ouput; 
 apply a second mathematical model to a second dataset including at least the first decision output, wherein the second mathematical model provides an updated decision output corresponding to the first decision output; and 
 select the first decision output for labelling based at least in part on a combination of the first decision output and the updated decision output. 
   
     
     
         11 . The system of  claim 10 , wherein the non-tranitory computer readable medium further includes instructions, which when executed by the one or more processing resources of the computer system, causes the one or more processing resources to:
 comparing the first decision output with one of the previously labelled decision ouputs to yield a comparison result; and   wherein selecting the first decision output for labelling is done based at least in part on the combination of the first decision output and the updated decision output, and upon the comparison result.   
     
     
         12 . A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a computer system, causes the one or more processing resources to:
 rank a set of unlabelled data elements based upon an expected impact of each of the unlabelled data elements on operation of a first mathematical model to yield a subset of high ranked data elements, wherein the subset of high ranked data elements includes at least one unlabelled data element having a ranking that is higher than another data element of the set of data elements not included in the subset of high ranked data elements;   train the first mathematical model using a first dataset including both the subset of high ranked data elements and a set of previously labelled data elements to yield at least a first decision output and a first confidence value corresponding to the first decision output and a second decision output and a second confidence value corresponding to the second decision ouput;   apply a second mathematical model to a second dataset including at least the first decision output, wherein the second mathematical model provides an updated decision output corresponding to the first decision output; and   select the first decision output for labelling based at least in part on a combination of the first decision output and the updated decision output.   
     
     
         13 . The non-transitory computer-readable storage medium of  claim 12 , wherein the non-tranitory computer readable medium further includes instructions, which when executed by the one or more processing resources of the computer system, causes the one or more processing resources to:
 select the first decision output for inclusion in the second dataset based upon the first confidence value exceeding a confidence threshold value; and   exclude the second decision output from inclusion in the second dataset based upon the second confidence value being less than the confidence threshold value.   
     
     
         14 . The non-transitory computer-readable storage medium of  claim 12 , wherein the non-tranitory computer readable medium further includes instructions, which when executed by the one or more processing resources of the computer system, causes the one or more processing resources to:
 comparing the first decision output with one of the previously labelled decision ouputs to yield a comparison result; and   wherein selecting the first decision output for labelling is done based at least in part on the combination of the first decision output and the updated decision output, and upon the comparison result.   
     
     
         15 . The non-transitory computer-readable storage medium of  claim 14 , wherein the comparison result indicates that the one of the previously labelled decision ouputs is similar to the first decision output. 
     
     
         16 . The non-transitory computer-readable storage medium of  claim 12 , wherein the instructions that cause the one or more processing resources to select the first decision output for labelling based at least in part on the combination of the first decision output and the updated decision output include instructions, which when executed by the one or more processing resources of the computer system, causes the one or more processing resources to:
 select the first decision output for labelling based at least in part on the first decision output matching the updated decision output.   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 12 , wherein the instructions that cause the one or more processing resources to rank the set of unlabelled data elements based upon the expected impact of each of the unlabelled data elements on operation of the first mathematical model include instructions, which when executed by the one or more processing resources of the computer system, causes the one or more processing resources to: comprises:
 select an unlabelled data element from a set of unlabelled data elements to yield a selected, unlabelled data element, wherein non-selected unlabelled data elements in the set of unlabelled data elements are a non-selected set of unlabelled data elements;   select a subset of the non-selected set of unlabelled data elements;   merge the selected, unlabelled data element with the subset of the non-selected set of unlabelled data elements to yield a merged, unlabelled dataset;   form a union of the merged, unlabelled dataset and a labelled dataset to yield a union dataset; and   calculate an expected performance value of the union dataset.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , wherein the set of unlabelled data elements is a set of unlabelled vectors, and wherein the labelled dataset is a set of labelled vectors, and wherein the non-tranitory computer readable medium further includes instructions, which when executed by the one or more processing resources of the computer system, causes the one or more processing resources to:
 generate the set of unlabelled vectors using a combination of at least a first heuristic and a second heuristic.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 18 , wherein:
 the first heuristic is selected from a group consisting of: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an orthogonality to labelled points heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic; and   the second heuristic is different from the first heuristic and selected from a group consisting of: the Shannon's entropy heuristic, the confidence based heuristic, the distance from decision hyperplane heuristic, the orthogonality to labelled points heuristic, the information density heuristic, the perturbation heuristic, the expected gradient length heuristic, and the consensus based heuristic.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 18 , wherein the non-tranitory computer readable medium further includes instructions, which when executed by the one or more processing resources of the computer system, causes the one or more processing resources to:
 generate the set of unlabelled vectors using a combination of four or more of the following heuristics: a Shannon's entropy heuristic, a confidence based heuristic, a distance from decision hyperplane heuristic, an orthogonality to labelled points heuristic, an information density heuristic, a perturbation heuristic, an expected gradient length heuristic, and a consensus based heuristic.