Long-range modeling of source code files by syntax hierarchy

Abstract

The syntax elements of a source code program used to represent the context of a focal method are selected based on a priority order. The selected syntax elements are input into a fixed-size context window that is used to train a neural transformer with attention model to learn to generate source code and used by the neural transformer model to generate source code. The context window contains prioritized sequences of tokens that extend beyond the target focus in order to provide a longer visibility back into the source code program for the model to learn predictive patterns. This gives the model a file-level context of the source code program without increasing the size of the context window.

What is claimed: 
     
         1 . A system comprising:
 one or more processors; and   a memory that stores one or more programs that are configured to be executed by the one or more processors, the one or more programs including instructions to perform actions that:   extract a first plurality of syntax elements from a source code program to represent a focal method and a second plurality of syntax elements to represent a context of the focal method;   generate a rolling window comprising a plurality of context windows;   fill a first one of the plurality of context windows of the rolling window with select ones of the first plurality of syntax elements and a second one of the plurality of context windows of the rolling window with select ones of the second plurality of syntax elements, wherein the select ones of the second plurality of syntax elements are selected based on a pre-configured order; and   obtain from a deep learning model source code based on the data of the rolling window.   
     
     
         2 . The system of  claim 1 , wherein the pre-configured order includes a hierarchical list of syntax elements. 
     
     
         3 . The system of  claim 1 , wherein the second plurality of syntax elements have a scope that differs from a local scope of the focal method. 
     
     
         4 . The system of  claim 1 , wherein the first plurality of syntax elements includes a method signature of the focal method, a method docstring of the focal method, and/or a class name of the focal method. 
     
     
         5 . The system of  claim 1 , wherein the second plurality of syntax elements include a global import statement, a method signature of a peer method of a class of the focal method, a docstring of a class of the method signature, a global expression, and/or a method body of a method of the class of the focal method. 
     
     
         6 . The system of  claim 1 , wherein the pre-configured order priorities global import statements over other syntax elements. 
     
     
         7 . The system of  claim 1 , wherein the pre-configured order priorities assigned values over other syntax elements than global import statements. 
     
     
         8 . The system of  claim 1 , wherein the pre-configured order priorities class attributes over other syntax elements than global import statements and assigned values. 
     
     
         9 . A computer-implemented method, comprising:
 extracting a first plurality of syntax elements from a source code program to represent features of a focal method;   generating an embedding for the first plurality of syntax elements;   extracting a second plurality of syntax elements from the source code program to represent a context of the focal method;   generating an embedding for each syntax element of the second plurality of syntax elements;   constructing a rolling window having a plurality of context windows;   populating a first context window with the embedding for the first plurality of syntax elements;   populating the second context window with select ones of the second plurality of syntax elements having a closest embedding distance to the embedding of the first plurality of syntax elements; and   applying the rolling window to a deep learning model to generate source code based on data of the rolling window.   
     
     
         10 . The computer-implemented method of  claim 9 , wherein generating an embedding for the first plurality of syntax elements further comprises:
 obtaining a bi-encoder; and   applying the first plurality of syntax elements to the bi-encoder to generate the embedding for the first plurality of syntax elements.   
     
     
         11 . The computer-implemented method of  claim 9 , wherein the deep learning model is a neural transformer model with attention. 
     
     
         12 . The computer-implemented method of  claim 9 , further comprising:
 determining the closest embedding distance based on a cosine similarity between the embedding of the first plurality of syntax elements with the embedding of a select one of the second plurality of syntax elements.   
     
     
         13 . The computer-implemented method of  claim 10 , wherein the bi-encoder includes a first encoder that generates an embedding for the first plurality of syntax elements jointly with a second encoder that generates an embedding for a select one of the second plurality of syntax elements. 
     
     
         14 . The computer-implemented method of  claim 9 , wherein the first plurality of syntax elements includes a focal method signature, docstring of the focal method and/or class name of the focal method. 
     
     
         15 . The computer-implemented method of  claim 9 , wherein the second plurality of syntax elements have a scope that differs from a local scope of the focal method. 
     
     
         16 . A computer-implemented method, comprising:
 analyzing a source code snippet having a focal method for focal method features and related syntax elements, wherein the focal method features identify a focal method, wherein the related syntax elements represent a context of the focal method;   producing an embedding for the focal method features and embeddings for each of the related syntax elements;   constructing a rolling window having a plurality of context windows;   filling a first context window of the plurality of context windows with the embeddings of the focal method features;   utilizing the embeddings of the focal method features and the embeddings for each of the related syntax elements to prioritize select ones of the related syntax elements to fill a second context window;   filling a second context window of the plurality of context windows with the embeddings of the prioritized select ones of the related syntax elements; and   applying data of the rolling widow to a deep learning model to generate source code based on the data of the rolling window.   
     
     
         17 . The computer-implemented method of  claim 16 , wherein the deep learning model is a neural transformer model with attention. 
     
     
         18 . The computer-implemented method of  claim 16 , further comprising:
 determining a priority of the related syntax elements based on an embedding distance measure between an embedding of the focal method features and each embedding of the related syntax elements.   
     
     
         19 . The computer-implemented method of  claim 18 , further comprising:
 generating the embedding of the focal method features and each embedding of the related syntax elements using a bi-encoder,   wherein the bi-encoder comprises a joint embedding space for the focal method features and syntax elements of the focal method features.   
     
     
         20 . The computer-implemented method of  claim 16 , wherein the joint embedding space is based on pairs of embeddings comprising an embedding of the focal method features and an embedding of a related syntax element.