Automated data extraction pipeline for large language model training

Abstract

An automated data extraction pipeline for large language model (LLM) training may include extracting a set of code segments from a set of natural language question-answer (Q&A) combinations that each include a provided input, a provided output, and a provided code segment formatted to transform the provided input into the provided output. The data extraction pipeline may then generate a predicted output from a question portion of a first natural language Q&A combination using a first LLM. A first extracted code segment from the extracted set of code segments may then be executed to generate a first actual output of the first extracted code segment. One or more data samples may then be generated for training a second LLM based on a comparison of the first actual output to the predicted output. The second LLM may then be trained using the one or more data samples.

What is claimed is: 
     
         1 . A method for data processing, comprising:
 extracting a plurality of code segments from a plurality of natural language question-answer combinations, wherein each question-answer combination comprises a provided input, a provided output, and a provided code segment formatted to transform the provided input into the provided output;   generating, using a first large language model, a predicted output from a question portion of a first natural language question-answer combination of the plurality of natural language question-answer combinations;   executing a first extracted code segment from the extracted plurality of code segments to generate a first actual output of the first extracted code segment;   generating one or more data samples for a second large language model based at least in part on a comparison of the first actual output of the first extracted code segment to the predicted output from the first large language model; and   training the second large language model using the one or more data samples.   
     
     
         2 . The method of  claim 1 , further comprising:
 generating, using the first large language model, a predicted input from the question portion of the first natural language question-answer combination, wherein executing the first extracted code segment comprises inputting the predicted input into the first extracted code segment.   
     
     
         3 . The method of  claim 2 , further comprising:
 generating, using the first large language model, a first predicted code segment from the question portion of the first natural language question-answer combination, wherein the first predicted code segment is formatted by the first large language model to transform the provided input into the provided output; and   executing the first predicted code segment to generate a first actual output of the first predicted code segment, wherein generating the one or more data samples for the second large language model is based at least in part on comparing the first actual output of the first predicted code segment to the predicted output from the first large language model.   
     
     
         4 . The method of  claim 3 , wherein executing the first predicted code segment comprises inputting the predicted input into the first predicted code segment. 
     
     
         5 . The method of  claim 3 , further comprising:
 categorizing the one or more data samples into a first training set comprising a first plurality of data samples or into a second training set comprising a second plurality of data samples based at least in part on the comparison of the first actual output of the first extracted code segment to the predicted output from the first large language model, or a comparison of the first actual output of the first predicted code segment to the predicted output from the first large language model, or both; and   training the second large language model using the first training set, the second training set, or both.   
     
     
         6 . The method of  claim 3 , further comprising:
 discarding the first extracted code segment from a list of candidate training samples if executing the first extracted code segment fails to generate a valid actual output; and   discarding the first predicted code segment from the list of candidate training samples if executing the first predicted code segment fails to generate the valid actual output.   
     
     
         7 . The method of  claim 1 , wherein the first large language model comprises a publicly-hosted generative pre-training transformer model. 
     
     
         8 . An apparatus for data processing, comprising:
 one or more memories storing processor-executable code; and   one or more processors coupled with the one or more memories and individually or collectively operable to execute the code to cause the apparatus to:
 extract a plurality of code segments from a plurality of natural language question-answer combinations, wherein each question-answer combination comprises a provided input, a provided output, and a provided code segment formatted to transform the provided input into the provided output; 
 generate, using a first large language model, a predicted output from a question portion of a first natural language question-answer combination of the plurality of natural language question-answer combinations; 
 execute a first extracted code segment from the extracted plurality of code segments to generate a first actual output of the first extracted code segment; 
 generate one or more data samples for a second large language model based at least in part on a comparison of the first actual output of the first extracted code segment to the predicted output from the first large language model; and 
 train the second large language model using the one or more data samples. 
   
     
     
         9 . The apparatus of  claim 8 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
 generate, using the first large language model, a predicted input from the question portion of the first natural language question-answer combination, wherein executing the first extracted code segment comprises inputting the predicted input into the first extracted code segment.   
     
     
         10 . The apparatus of  claim 9 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
 generate, using the first large language model, a first predicted code segment from the question portion of the first natural language question-answer combination, wherein the first predicted code segment is formatted by the first large language model to transform the provided input into the provided output; and   execute the first predicted code segment to generate a first actual output of the first predicted code segment, wherein generating the one or more data samples for the second large language model is based at least in part on comparing the first actual output of the first predicted code segment to the predicted output from the first large language model.   
     
     
         11 . The apparatus of  claim 10 , wherein executing the first predicted code segment comprises inputting the predicted input into the first predicted code segment. 
     
     
         12 . The apparatus of  claim 10 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
 categorize the one or more data samples into a first training set comprising a first plurality of data samples or into a second training set comprising a second plurality of data samples based at least in part on the comparison of the first actual output of the first extracted code segment to the predicted output from the first large language model, or a comparison of the first actual output of the first predicted code segment to the predicted output from the first large language model, or both; and   train the second large language model using the first training set, the second training set, or both.   
     
     
         13 . The apparatus of  claim 10 , wherein the one or more processors are individually or collectively further operable to execute the code to cause the apparatus to:
 discard the first extracted code segment from a list of candidate training samples if executing the first extracted code segment fails to generate a valid actual output; and   discard the first predicted code segment from the list of candidate training samples if executing the first predicted code segment fails to generate the valid actual output.   
     
     
         14 . The apparatus of  claim 8 , wherein the first large language model comprises a publicly-hosted generative pre-training transformer model. 
     
     
         15 . A non-transitory computer-readable medium storing code for data processing, the code comprising instructions executable by one or more processors to:
 extract a plurality of code segments from a plurality of natural language question-answer combinations, wherein each question-answer combination comprises a provided input, a provided output, and a provided code segment formatted to transform the provided input into the provided output;   generate, using a first large language model, a predicted output from a question portion of a first natural language question-answer combination of the plurality of natural language question-answer combinations;   execute a first extracted code segment from the extracted plurality of code segments to generate a first actual output of the first extracted code segment;   generate one or more data samples for a second large language model based at least in part on a comparison of the first actual output of the first extracted code segment to the predicted output from the first large language model; and   train the second large language model using the one or more data samples.   
     
     
         16 . The non-transitory computer-readable medium of  claim 15 , wherein the instructions are further executable by the one or more processors to:
 generate, using the first large language model, a predicted input from the question portion of the first natural language question-answer combination, wherein executing the first extracted code segment comprises inputting the predicted input into the first extracted code segment.   
     
     
         17 . The non-transitory computer-readable medium of  claim 16 , wherein the instructions are further executable by the one or more processors to:
 generate, using the first large language model, a first predicted code segment from the question portion of the first natural language question-answer combination, wherein the first predicted code segment is formatted by the first large language model to transform the provided input into the provided output; and   execute the first predicted code segment to generate a first actual output of the first predicted code segment, wherein generating the one or more data samples for the second large language model is based at least in part on comparing the first actual output of the first predicted code segment to the predicted output from the first large language model.   
     
     
         18 . The non-transitory computer-readable medium of  claim 17 , wherein executing the first predicted code segment comprises inputting the predicted input into the first predicted code segment. 
     
     
         19 . The non-transitory computer-readable medium of  claim 17 , wherein the instructions are further executable by the one or more processors to:
 categorize the one or more data samples into a first training set comprising a first plurality of data samples or into a second training set comprising a second plurality of data samples based at least in part on the comparison of the first actual output of the first extracted code segment to the predicted output from the first large language model, or a comparison of the first actual output of the first predicted code segment to the predicted output from the first large language model, or both; and   train the second large language model using the first training set, the second training set, or both.   
     
     
         20 . The non-transitory computer-readable medium of  claim 17 , wherein the instructions are further executable by the one or more processors to:
 discard the first extracted code segment from a list of candidate training samples if executing the first extracted code segment fails to generate a valid actual output; and   discard the first predicted code segment from the list of candidate training samples if executing the first predicted code segment fails to generate the valid actual output.