Automated label generation using a machine-learned language model

Abstract

An online system may provide an instruction prompt to a machine-learned language model. The instruction prompt may include an instruction to generate an evaluation label of a training sample of a classification model and a textual format related to how data is arranged. The evaluation label may be used in a supervised training of the classification model. The online system may provide a batch of evaluation request prompts to the machine-learned language model. Each evaluation request prompt includes data that is at least partially arranged in the textual format described in the instruction prompt. The online system may receive a plurality of responses from the machine-learned language model. Each response includes the evaluation label corresponding to each evaluation request prompt. The online system may store at least evaluation labels and the data in the evaluation request prompts as training samples for the supervised training of the classification model.

What is claimed is: 
     
         1 . A method comprising, at an online system comprising one or more processors and one or more computer-readable media:
 providing an instruction prompt to a machine-learned language model, the instruction prompt comprising (1) an instruction for the machine-learned language model to generate an evaluation label of a training sample of a classification model, and (2) a textual format related to how data is arranged, wherein the evaluation label is to be used as a training label for the training sample in a supervised training of the classification model;   providing a batch of evaluation request prompts to the machine-learned language model, each evaluation request prompt comprising data that is at least partially arranged in the textual format described in the instruction prompt;   receiving a plurality of responses from the machine-learned language model, each response comprising the evaluation label corresponding to each evaluation request prompt; and   storing at least evaluation labels and the data in the evaluation request prompts as training samples for the supervised training of the classification model.   
     
     
         2 . The method of  claim 1 , wherein providing the instruction prompt to the machine-learned language model comprises:
 specifying an input textual format of an input that the machine-learned language model is to receive;   specifying an output textual format of an output that the machine-learned language model is to generate; and   including a chain-of-thoughts instruction for the machine-learned language model to follow in performing evaluation, the chain-of-thoughts instruction including an explanation for each evaluation label on how the training sample should be classified to the evaluation label.   
     
     
         3 . The method of  claim 1 , wherein providing the instruction prompt to the machine-learned language model comprises:
 providing a multimodal input example that follows the textual format, the multimodal input example including an input to the classification model, an output of the classification model, attribute data retrieved from a database of the online system, and an image; and   specifying an output format that includes the evaluation label and a reason for assigning the evaluation label.   
     
     
         4 . The method of  claim 1 , wherein providing the instruction prompt to the machine-learned language model comprises:
 specifying a set of candidate evaluation labels in the instruction for the machine-learned language model to generate the evaluation label;   providing an example for each candidate evaluation label on one or more selection criteria for the candidate evaluation label; and   providing a chain-of-thoughts instruction in applying the one or more selection criteria for each candidate evaluation label.   
     
     
         5 . The method of  claim 1 , wherein the instruction prompt further comprises one or more of:
 a set of candidate evaluation labels;   an example of input of the machine-learned language model, the input being in the textual format;   an example of output of the machine-learned language model;   a chain-of-thought instruction; or   an explanation for selecting each candidate evaluation label.   
     
     
         6 . The method of  claim 1 , wherein the classification model is a search engine for an item based on a search query, and wherein providing the instruction prompt to the machine-learned language model comprises:
 including a first instruction for the machine-learned language model to identify a search intent from the search query;   including a second instruction for the machine-learned language model to identify one or more attributes of the item returned by the search engine; and   providing one or more examples to the machine-learned language model on how the evaluation label is assigned based on comparing the one or more attributes of the item to the search intent.   
     
     
         7 . The method of  claim 1 , wherein providing the batch of evaluation request prompts to the machine-learned language model comprises:
 retrieving historical inputs and historical outputs of the classification model that are stored in a database;   formatting the historical inputs and historical outputs according to the textual format that is provided in the instruction prompt; and   generating the batch of evaluation request prompts, each evaluation request prompt includes the historical inputs and historical outputs.   
     
     
         8 . The method of  claim 1 , further comprising training the classification model, wherein training the classification model comprises:
 retrieving the training samples of the classification model, the training samples each comprising the evaluation label generated by the machine-learned language model;   applying, in forward propagation, the classification model to the training samples to generate predicted labels;   comparing the predicted labels to the evaluation label generated by the machine-learned language model; and   adjusting, in backpropagation, one or more parameters of the machine-learned language model based on the comparing.   
     
     
         9 . The method of  claim 1 , further comprising:
 receiving a query for the classification model;   identifying, by the classification model, an item based on the query;   applying the machine-learned language model to generate the evaluation label for the item; and   providing the evaluation label as a classification in response to the query.   
     
     
         10 . The method of  claim 1 , further comprising:
 receiving a query for the classification model;   identifying, by the classification model, a plurality of items based on the query;   applying the machine-learned language model to generate evaluation labels for the plurality of items; and   selecting, based on the evaluation labels, one of the plurality of items as a sponsored item to be displayed as a response to the query.   
     
     
         11 . A computer program product comprising one or more non-transitory computer-readable storage media configured to store code comprising executable code instructions, wherein the executable code instructions, when executed, cause one or more processors to:
 provide an instruction prompt to a machine-learned language model, the instruction prompt comprising (1) an instruction for the machine-learned language model to generate an evaluation label of a training sample of a classification model and (2) a textual format related to how data is arranged, wherein the evaluation label is to be used as a training label for the training sample in a supervised training of the classification model;   provide a batch of evaluation request prompts to the machine-learned language model, each evaluation request prompt comprising data that is at least partially arranged in the textual format described in the instruction prompt;   receive a plurality of responses from the machine-learned language model, each response comprising the evaluation label corresponding to each evaluation request prompt; and   store at least evaluation labels and the data in the evaluation request prompts as training samples for the supervised training of the classification model.   
     
     
         12 . The computer program product of  claim 11 , wherein the executable code instruction to provide the instruction prompt to the machine-learned language model comprises instructions to:
 specify an input textual format of an input that the machine-learned language model is to receive;   specify an output textual format of an output that the machine-learned language model is to generate; and   include a chain-of-thoughts instruction for the machine-learned language model to follow in performing evaluation, the chain-of-thoughts instruction include an explanation for each evaluation label on how the training sample should be classified to the evaluation label.   
     
     
         13 . The computer program product of  claim 11 , wherein the executable code instruction to provide the instruction prompt to the machine-learned language model comprises instructions to:
 provide a multimodal input example that follows the textual format, the multimodal input example include an input to the classification model, an output of the classification model, attribute data retrieved from a database of an online system, and an image; and   specify an output format that includes the evaluation label and a reason for assigning the evaluation label.   
     
     
         14 . The computer program product of  claim 11 , wherein the executable code instruction to provide the instruction prompt to the machine-learned language model comprises instructions to:
 specify a set of candidate evaluation labels in the instruction for the machine-learned language model to generate the evaluation label;   provide an example for each candidate evaluation label on one or more selection criteria for the candidate evaluation label; and   provide a chain-of-thoughts instruction in applying the one or more selection criteria for each candidate evaluation label.   
     
     
         15 . The computer program product of  claim 11 , wherein the instruction prompt further comprises one or more of:
 a set of candidate evaluation labels;   an example of input of the machine-learned language model, the input being in the textual format;   an example of output of the machine-learned language model;   a chain-of-thought instruction; or   an explanation for selecting each candidate evaluation label.   
     
     
         16 . The computer program product of  claim 11 , wherein the instruction to the classification model is a search engine for an item based on a search query, and wherein the executable code instruction to provide the instruction prompt to the machine-learned language model comprises instructions to:
 include a first instruction for the machine-learned language model to identify a search intent from the search query;   include a second instruction for the machine-learned language model to identify one or more attributes of the item returned by the search engine; and   provide one or more examples to the machine-learned language model on how the evaluation label is assigned based on compare the one or more attributes of the item to the search intent.   
     
     
         17 . The computer program product of  claim 11 , wherein the executable code instruction to provide the batch of evaluation request prompts to the machine-learned language model comprises instructions to:
 retrieve historical inputs and historical outputs of the classification model that are stored in a database;   format the historical inputs and historical outputs according to the textual format that is provided in the instruction prompt; and   generate the batch of evaluation request prompts, each evaluation request prompt includes the historical inputs and historical outputs.   
     
     
         18 . The computer program product of  claim 11 , wherein the executable code instructions, when executed, further cause the one or more processors to train the classification model, wherein the executable code instruction to train the classification model comprises instructions to:
 retrieve the training samples of the classification model, the training samples each comprising the evaluation label generated by the machine-learned language model;   apply, in forward propagation, the classification model to the training samples to generate predicted labels;   compare the predicted labels to the evaluation label generated by the machine-learned language model; and   adjust, in backpropagation, one or more parameters of the machine-learned language model based on the comparing.   
     
     
         19 . The computer program product of  claim 11 , wherein the executable code instructions, when executed, further cause the one or more processors to:
 receive a query for the classification model;   identify, by the classification model, an item based on the query;   applying the machine-learned language model to generate the evaluation label for the item; and   provide the evaluation label as a classification in response to the query.   
     
     
         20 . A system comprising:
 one or more processors; and   one or more non-transitory computer-readable storage media configured to store code comprising executable code instructions, wherein the executable code instructions, when executed, cause the one or more processors to:
 provide an instruction prompt to a machine-learned language model, the instruction prompt comprising (1) an instruction for the machine-learned language model to generate an evaluation label of a training sample of a classification model and (2) a textual format related to how data is arranged, wherein the evaluation label is to be used as a training label for the training sample in a supervised training of the classification model; 
 provide a batch of evaluation request prompts to the machine-learned language model, each evaluation request prompt comprising data that is at least partially arranged in the textual format described in the instruction prompt; 
 receive a plurality of responses from the machine-learned language model, each response comprising the evaluation label corresponding to each evaluation request prompt; and 
 store at least evaluation labels and the data in the evaluation request prompts as training samples for the supervised training of the classification model.