Automating test-driven development with transformers

Abstract

A test-driven development system utilizes a neural transformer model with attention to generate method bodies for a focal method given its associated test cases, and optionally a method signature and a docstring of the focal method. The candidate method bodies are validated for syntactic correctness, tested using the given test cases, and tested with a donor class in a target system. Those candidate method bodies passing the validation and testing are then ranked based on a PLUM score that analyzes the candidate method bodies against various quality and performance metrics.

What is claimed: 
     
         1 . A computer-implemented method, comprising:
 pre-training a deep learning model on an unsupervised training dataset of natural language text;   pre-training the deep learning model on an unsupervised training dataset of source code snippets;   fine-tuning the deep learning model on a supervised training dataset, wherein the supervised training dataset includes a plurality of tuples, wherein a tuple includes a method signature of a focal method and a plurality of test cases for the focal method; and   deploying the deep learning model to generate a method body for a target method given at least one test case for the target method.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 obtaining a first plurality of source code snippets; and   generating the unsupervised training dataset of source code snippets by applying a denoising function to each training sample of the first plurality of source code snippets.   
     
     
         3 . The computer-implemented method of  claim 1 , further comprising:
 obtaining a first plurality of natural language text; and   generating the unsupervised training dataset of natural language text by applying a denoising function to each sample of the first plurality of natural language text.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein the deep learning model is deployed in an integrated development environment (IDE). 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the deep learning model is deployed in a source code editor. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the deep learning model is deployed as a web service that generates the method body for the target method when given the at least one test case for the target method. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein the deep learning model is a neural transformer with attention. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the deep learning model is a neural transformer with attention in an encoder-decoder configuration. 
     
     
         9 . A system comprising:
 a processor; and   a memory that stores a program configured to be executed by the processor, the program includes instructions to perform actions that:   train a deep learning model to predict a method body given at least one test case associated with the method body, wherein the deep learning model is pre-trained on an unsupervised training dataset of natural language text and on an unsupervised training dataset of source code snippets, wherein the pre-trained deep learning model is then fine-tuned on a supervised training dataset, wherein the supervised training dataset includes a plurality of samples, wherein a sample includes a method signature of a focal method and a plurality of test cases for the focal method; and   deploy the deep learning model to predict the method body given the at least one test case associated with the method body.   
     
     
         10 . The system of  claim 9 , wherein the program includes instructions to perform actions that:
 create the unsupervised training dataset of natural language text through application of a denoising function to each natural language training sample of the unsupervised training dataset of natural language text.   
     
     
         11 . The system of  claim 9 , wherein the program includes instructions to perform actions that:
 create the unsupervised training dataset of source code snippets through application of a denoising function to each source code snippet of the unsupervised training dataset of source code snippets.   
     
     
         12 . The system of  claim 9 , wherein the deep learning model is deployed in an integrated development environment (IDE). 
     
     
         13 . The system of  claim 9 , wherein the deep learning model is deployed in a source code editor. 
     
     
         14 . The system of  claim 9 , wherein the deep learning model is deployed as a web service that predicts the method body when provided with the given at least one test case. 
     
     
         15 . The system of  claim 9 , wherein the deep learning model is a neural transformer with attention. 
     
     
         16 . The system of  claim 9 , wherein the deep learning model is a neural transformer with attention configured as a sequence-to-sequence model.