Customizable machine learning models

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for customizable machine learning models. In some implementations, data is received, including (i) example data sets and (ii) data specifying one or more criteria to be assessed. A set of multiple models is trained, where each model in the set of models is trained using a training data set comprising a different subset of the example data sets. Output of the models is obtained for various example data sets, and a combination of n-grams is selected based on the outputs. The example data sets are used to train a classifier to evaluate input data with respect to the specified one or more criteria based on whether the input data includes the n-grams in the selected combination of n-grams.

1 . A computer-implemented method comprising:
 receiving, by a computing system comprising one or more computers, (i) example data sets that each include information about a different individual in an organization and (ii) data specifying one or more criteria to be assessed;   for each of different combinations of n-grams that occur in the example data sets, training, by the computing system, a set of multiple models that are each configured to classify input data based on whether the n-grams in the combination are present in the input data, wherein each model in the set of models is respectively trained using a training data set comprising a different subset of the example data sets;   obtaining, by the computing system and for each model in each of the sets of models, output that the model generates for a test data set comprising example data sets different from those of the training data set with which the model was trained;   selecting, by the computing system, one of the combinations of n-grams based on the outputs; and   using, by the computing system, the example data sets to train a classifier to evaluate input data with respect to the specified one or more criteria based on whether the input data includes the n-grams in the selected combination of n-grams.   
     
     
         2 . The method of  claim 1 , further comprising:
 for each of multiple n-grams extracted from text of the example data sets, determining, by the computing system, a correlation measure indicative of a frequency that the n-gram occurs in example data sets determined to satisfy the specified one or more criteria; and   identifying, by the computing system, a subset of the n-grams selected based on the correlation measures,   wherein the different combinations of n-grams that occur in the example data sets are different combinations of n-grams that occur in the subset of the n-grams selected based on the correlation measures.   
     
     
         3 . The method of  claim 2 , wherein selecting the subset of the n-grams based on the correlation measures comprises:
 generating a rank ordered list of n-grams according to associated correlation measures; and   selecting a number of the top ranked n-grams as the subset of the n-grams.   
     
     
         4 . The method of  claim 3 , wherein the rank ordered list of n-grams is generated using a binary logistic regression. 
     
     
         5 . The method of  claim 2 , wherein the models in each set of multiple models are configured to classify input data by:
 obtaining, with a machine learning model, weights associated with each n-gram in the combination of n-grams using the training data set.   
     
     
         6 . The method of  claim 2 , wherein selecting one of the combinations of n-grams based on the outputs comprises:
 determining, based on the data specifying one or more criteria to be assessed, a cost function that used to define a top performance tier;   determining, for each model in each of the sets of models, an efficacy for each model based on (i) a performance metric of example data sets ranked within the top performance tier by the model and (ii) an average performance metric of the example data sets within the test data set; and   selecting the one of the combinations of n-grams based on the efficacy determined for the model that corresponds to the one of the combinations of n-grams.   
     
     
         7 . The method of  claim 6 , wherein the performance metric of example data sets ranked within the top performance tier by the model is an average of performance metrics of example data sets ranked within the top performance tier by the model. 
     
     
         8 . The method of  claim 6 , wherein selecting one of the combinations of n-grams based on the outputs further comprises:
 determining, for each combination of n-grams, an average efficacy and a consistency of efficacy across the multiple models corresponding to the combination of n-grams; and   selecting the one of the combinations of n-grams based on the average efficacy and consistency associated with the one of the combinations of n-grams.   
     
     
         9 . The method of  claim 8 , wherein the consistency of efficacy across the multiple models corresponding to the combination of n-grams is a standard deviation of the efficacy across the multiple models corresponding to the combination of n-grams. 
     
     
         10 . The method of  claim 2 , wherein using the example data sets to train the classifier to evaluate input data comprises using all of the example data sets to train the classifier to evaluate input data. 
     
     
         11 . The method of  claim 2 , further comprising:
 for each of multiple search ranges determined from the example data sets, determining a correlation measure indicative of a frequency that information falling within the search range occurs in the example data sets determined to satisfy the specified one or more criteria, and   wherein selecting the subset of n-grams based on the correlation measures comprises selecting a subset of n-grams and search ranges based on the correlation measures.   
     
     
         12 . The method of  claim 2 , wherein the example data sets include job application data and job performance data associated with current or former employees of a company. 
     
     
         13 . The method of  claim 12 , wherein the job application data associated with the current or former employees of the company includes resume data, curriculum vitae data, or job application data. 
     
     
         14 . The method of  claim 2 , wherein the specified one or more criteria indicate a performance outcome, the method further comprising:
 using the trained classifier to generate, for each candidate in a set of candidates, an output indicating a likelihood that the candidate will achieve the performance outcome; and   selecting, based on outputs of the classifier, a group of the candidates that are likely to achieve the performance outcome.   
     
     
         15 . A system comprising:
 one or more processors; and a data store coupled to the one or more processors having instructions stored thereon which, when executed by the at least one processor, causes the one or more processors to perform operations comprising:
 receiving (i) example data sets that each include information about a different individual in an organization and (ii) data specifying one or more criteria to be assessed; 
 for each of different combinations of n-grams that occur in the example data sets, training a set of multiple models that are each configured to classify input data based on whether the n-grams in the combination are present in the input data, wherein each model in the set of models is respectively trained using a training data set comprising a different subset of the example data sets; 
 obtaining, for each model in each of the sets of models, output that the model generates for a test data set comprising example data sets different from those of the training data set with which the model was trained; 
 selecting one of the combinations of n-grams based on the outputs; and 
 using the example data sets to train a classifier to evaluate input data with respect to the specified one or more criteria based on whether the input data includes the n-grams in the selected combination of n-grams. 
   
     
     
         16 . The system of  claim 15 , wherein the operations further comprise:
 for each of multiple n-grams extracted from text of the example data sets, determining a correlation measure indicative of a frequency that the n-gram occurs in example data sets determined to satisfy the specified one or more criteria; and   identifying a subset of the n-grams selected based on the correlation measures,   wherein the different combinations of n-grams that occur in the example data sets are different combinations of n-grams that occur in the subset of the n-grams selected based on the correlation measures.   
     
     
         17 . The system of  claim 16 , wherein selecting the subset of the n-grams based on the correlation measures comprises:
 generating a rank ordered list of n-grams according to associated correlation measures; and   selecting a number of the top ranked n-grams as the subset of the n-grams.   
     
     
         18 . A non-transient computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
 receiving (i) example data sets that each include information about a different individual in an organization and (ii) data specifying one or more criteria to be assessed;   for each of different combinations of n-grams that occur in the example data sets, training a set of multiple models that are each configured to classify input data based on whether the n-grams in the combination are present in the input data, wherein each model in the set of models is respectively trained using a training data set comprising a different subset of the example data sets;   obtaining, for each model in each of the sets of models, output that the model generates for a test data set comprising example data sets different from those of the training data set with which the model was trained;   selecting one of the combinations of n-grams based on the outputs; and   using the example data sets to train a classifier to evaluate input data with respect to the specified one or more criteria based on whether the input data includes the n-grams in the selected combination of n-grams.   
     
     
         19 . The medium of  claim 18 , wherein the operations further comprise:
 for each of multiple n-grams extracted from text of the example data sets, determining a correlation measure indicative of a frequency that the n-gram occurs in example data sets determined to satisfy the specified one or more criteria; and   identifying a subset of the n-grams selected based on the correlation measures,   wherein the different combinations of n-grams that occur in the example data sets are different combinations of n-grams that occur in the subset of the n-grams selected based on the correlation measures.   
     
     
         20 . The medium of  claim 19 , wherein selecting the subset of the n-grams based on the correlation measures comprises:
 generating a rank ordered list of n-grams according to associated correlation measures; and   selecting a number of the top ranked n-grams as the subset of the n-grams.