Code translations of resilient distributed datasets in databases

Abstract

The subject technology generates, by executing first code in a first programming language, a resilient distributed dataset (RDD). The subject technology generates, by executing the first code in the first programming language, a second RDD based on the generated RDD. The subject technology generates, by executing the first code in the first programming language, a directed acyclic graph (DAG) including the generated RDD and the generated second RDD. The subject technology serializes the DAG. The subject technology generates a database table based on the serialized DAG. The subject technology converts the first code to second code in a second programming language different than the first programming language, and at least one database query in a second format different than the first programming language. The subject technology sends the converted first code and the at least one database query to a database system for execution.

What is claimed is: 
     
         1 . A system comprising:
 at least one hardware processor; and   a memory storing instructions that cause the at least one hardware processor to perform operations comprising:   generating, by executing first code in a first programming language, a resilient distributed dataset (RDD);   generating, by executing the first code in the first programming language, a second RDD based on the generated RDD, the first code further including:
 a first particular code statement invoking a first set of operations to generate the RDD, 
 a second particular code statement invoking a second set of operations to map data associated with the RDD and generating the second RDD, the second RDD including a reference to the RDD generated by the first particular code statement, and 
 a third particular code statement initiating execution of a particular operation to run the second RDD, provided by the second particular code statement, and the particular operation providing a set of results; 
   generating, by executing the first code in the first programming language, a directed acyclic graph (DAG) including the generated RDD and the generated second RDD;   serializing the DAG;   generating a database table based on the serialized DAG;   converting the first code to second code in a second programming language different than the first programming language, and at least one database query in a second format different than the first programming language; and   sending the converted first code and the at least one database query to a database system for execution.   
     
     
         2 . The system of  claim 1 , wherein the RDD comprises information defining a set of partitions and including a set of data partitioned into the set of partitions, the RDD being in a first format that is incompatible with the database system. 
     
     
         3 . The system of  claim 1 , wherein the operations further comprise:
 prior to generating the RDD, receiving the first code in the first programming language, the first code including a reference to a first dataset, the first dataset corresponding to the RDD to be generated.   
     
     
         4 . The system of  claim 1 , wherein the database table comprises a column corresponding to an object including the serialized DAG, a first column corresponding to partition identifier, and a second column corresponding to a serialized partition object. 
     
     
         5 . The system of  claim 1 , wherein the RDD comprises a first object and the second RDD comprises a second object. 
     
     
         6 . The system of  claim 1 ,
 wherein the first code is incompatible with the database system and converting the first code to second code in the second programming language comprise further operations comprising:   generating a user defined function (UDF) based at least in part on the first code, the UDF including a set of code statements that are in the second programming language compatible with the database system.   
     
     
         7 . The system of  claim 6 , wherein the database query calls the generated UDF, the first particular code statement comprises a first code statement in a Scala programming language,
 the second particular code statement comprises a second code statement in the Scala programming language,   the third particular code statement comprises a third code statement in the Scala programming language, and   the converted first code includes Java code and a set of SQL statements converted from the Scala programming language.   
     
     
         8 . The system of  claim 1 , wherein the operations further comprise:
 determining that the first code includes an action command that triggers an output of results or storing the results.   
     
     
         9 . The system of  claim 8 , wherein the RDD represents a transformation operation, the transformation operation applies a function on the RDD and creates a new RDD, and operations further comprise:
 generating a tree of RDD operations based on the RDD; and   evaluating the tree of RDD operations based on the action command.   
     
     
         10 . The system of  claim 1 , wherein the first code in the first programming language corresponds to an external system different than the database system including a storage platform, the first code being incompatible with the database system. 
     
     
         11 . A method comprising:
 generating, by executing first code in a first programming language, a resilient distributed dataset (RDD);   generating, by executing the first code in the first programming language, a second RDD based on the generated RDD, the first code further including:
 a first particular code statement invoking a first set of operations to generate the RDD, 
 a second particular code statement invoking a second set of operations to map data associated with the RDD and generating the second RDD, the second RDD including a reference to the RDD generated by the first particular code statement, and 
 a third particular code statement initiating execution of a particular operation to run the second RDD, provided by the second particular code statement, and the particular operation providing a set of results; 
   generating, by executing the first code in the first programming language, a directed acyclic graph (DAG) including the generated RDD and the generated second RDD;   serializing the DAG;   generating a database table based on the serialized DAG;   converting the first code to second code in a second programming language different than the first programming language, and at least one database query in a second format different than the first programming language; and   sending the converted first code and the at least one database query to a database system for execution.   
     
     
         12 . The method of  claim 11 , wherein the RDD comprises information defining a set of partitions and including a set of data partitioned into the set of partitions, the RDD being in a first format that is incompatible with the database system. 
     
     
         13 . The method of  claim 11 , further comprising:
 prior to generating the RDD, receiving the first code in the first programming language, the first code including a reference to a first dataset, the first dataset corresponding to the RDD to be generated.   
     
     
         14 . The method of  claim 11 , wherein the database table comprises a column corresponding to an object including the serialized DAG, a first column corresponding to partition identifier, and a second column corresponding to a serialized partition object. 
     
     
         15 . The method of  claim 11 , wherein the RDD comprises a first object and the second RDD comprises a second object. 
     
     
         16 . The method of  claim 11 ,
 wherein the first code is incompatible with the database system and converting the first code to second code in the second programming language comprise further operations comprising:   generating a user defined function (UDF) based at least in part on the first code, the UDF including a set of code statements that are in the second programming language compatible with the database system.   
     
     
         17 . The method of  claim 16 , wherein the database query calls the generated UDF, the first particular code statement comprises a first code statement in a Scala programming language,
 the second particular code statement comprises a second code statement in the Scala programming language,   the third particular code statement comprises a third code statement in the Scala programming language, and   the converted first code includes Java code and a set of SQL statements converted from the Scala programming language.   
     
     
         18 . The method of  claim 11 , further comprising:
 determining that the first code includes an action command that triggers an output of results or storing the results.   
     
     
         19 . The method of  claim 18 , wherein the RDD represents a transformation operation, the transformation operation applies a function on the RDD and creates a new RDD, and operations further comprise:
 generating a tree of RDD operations based on the RDD; and   evaluating the tree of RDD operations based on the action command.   
     
     
         20 . A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
 generating, by executing first code in a first programming language, a resilient distributed dataset (RDD);   generating, by executing the first code in the first programming language, a second RDD based on the generated RDD, the first code further including:
 a first particular code statement invoking a first set of operations to generate the RDD, 
 a second particular code statement invoking a second set of operations to map data associated with the RDD and generating the second RDD, the second RDD including a reference to the RDD generated by the first particular code statement, and 
 a third particular code statement initiating execution of a particular operation to run the second RDD, provided by the second particular code statement, and the particular operation providing a set of results; 
   generating, by executing the first code in the first programming language, a directed acyclic graph (DAG) including the generated RDD and the generated second RDD;   serializing the DAG;   generating a database table based on the serialized DAG;   converting the first code to second code in a second programming language different than the first programming language, and at least one database query in a second format different than the first programming language; and   sending the converted first code and the at least one database query to a database system for execution.