Data pipeline branching

Abstract

An example method of data pipeline branching includes: identifying a master branch of a data processing pipeline comprising ordered data transformation operations, such that the master branch references a first dataset, a first logic implementing a first data transformation operation, and a second dataset resulting from applying the first logic to the first dataset; creating a first test branch of the data processing pipeline, such that the first test branch references the first dataset and a second logic implementing a second data transformation operation; generating a third dataset by applying the second logic to the initial dataset; and merging the first test branch into the master branch by causing the master branch to reference the first dataset, the second logic, and the third dataset.

What is claimed is: 
     
         1 . A method, comprising:
 identifying a master branch of a data processing pipeline comprising ordered data transformation operations, wherein the master branch references a first dataset, a first logic implementing a first data transformation operation, and a second dataset resulting from applying the first logic to the first dataset;   creating a first test branch of the data processing pipeline, wherein the first test branch references the first dataset and a second logic implementing a second data transformation operation;   generating a third dataset by applying the second logic to the first dataset; and   merging the first test branch into the master branch by causing the master branch to reference the first dataset, the second logic, and the third dataset;   wherein the method is performed using one or more processors.   
     
     
         2 . The method of  claim 1 , wherein merging the first test branch into the master branch further comprises:
 identifying a difference between the first logic and the second logic; and   receiving, via a user interface, an approval of the difference.   
     
     
         3 . The method of  claim 1 , wherein merging the first test branch into the master branch further comprises:
 identifying a difference between the second dataset logic and the third dataset; and   receiving, via a user interface, an approval of the difference.   
     
     
         4 . The method of  claim 1 , wherein merging the first test branch into the master branch further comprises:
 identifying a conflict between the first logic and the second logic; and   receiving, via a user interface, a selection of a third logic to resolve the conflict.   
     
     
         5 . The method of  claim 1 , wherein merging the first test branch into the master branch further comprises:
 performing a data health check operation by applying one or more conditions to the second dataset, the one or more conditions comprising a verification that a creation of the second dataset completed successfully and a verification that the second dataset is not stale.   
     
     
         6 . The method of  claim 1 , further comprising:
 responsive to receiving, via a user interface, a first request to protect the first test branch, preventing other users from further modifying the first test branch; and   responsive to receiving, via the user interface, a second request to further modify the first test branch, creating a child test branch associated with the first test branch.   
     
     
         7 . The method of  claim 1 , further comprising:
 responsive to receiving user input requesting a second test branch corresponding to the master branch, creating the second test branch, wherein the second test branch references the first dataset and a copy of the first logic;   receiving a request to modify the second test branch, the request comprising at least one change to the copy of the first logic;   modifying the second test branch independently of the master branch to include a third logic reflecting the at least one change to the copy of the first logic, the third logic to be applied to the first dataset to produce a fourth dataset; and   responsive to user input requesting a merge of the modified second test branch into the master branch, merging the modified second test branch into the master branch by causing the master branch to reference the first dataset, the third logic, and the fourth dataset.   
     
     
         8 . The method of  claim 1 , further comprising:
 rendering a visual representation of the data pipeline, including a first graph corresponding to the master branch and a second graph corresponding to the first test branch,   wherein the first graph includes a first node representing the first dataset, a second node representing the second dataset, and a first edge connecting the first node and the second node, wherein the first edge references the first logic to be applied to the first dataset in order to produce the second dataset, and   wherein the second graph includes a third node representing the first dataset, a fourth node representing the third dataset, and a second edge connecting the third node and the fourth node, wherein the second edge references the third logic to be applied to the first dataset in order to produce the third dataset.   
     
     
         9 . A system comprising:
 a memory; and   one or more processors coupled to the memory, the one or more processors to execute instructions to cause the one or more processors to perform operations comprising:
 identifying a master branch of a data processing pipeline comprising ordered data transformation operations, wherein the master branch references a first dataset, a first logic implementing a first data transformation operation, and a second dataset resulting from applying the first logic to the first dataset; 
 creating a first test branch of the data processing pipeline, wherein the first test branch references the first dataset and a second logic implementing a second data transformation operation; 
 generating a third dataset by applying the second logic to the first dataset; and 
 merging the first test branch into the master branch by causing the master branch to reference the first dataset, the second logic, and the third dataset. 
   
     
     
         10 . The system of  claim 9 , wherein merging the first test branch into the master branch further comprises:
 identifying a difference between the first logic and the second logic; and   receiving, via a user interface, an approval of the difference.   
     
     
         11 . The system of  claim 9 , wherein merging the first test branch into the master branch further comprises:
 identifying a difference between the second dataset logic and the third dataset; and   receiving, via a user interface, an approval of the difference.   
     
     
         12 . The system of  claim 9 , wherein merging the first test branch into the master branch further comprises:
 identifying a conflict between the first logic and the second logic; and   receiving, via a user interface, a selection of a third logic to resolve the conflict.   
     
     
         13 . The system of  claim 9 , wherein merging the first test branch into the master branch further comprises:
 performing a data health check operation by applying one or more conditions to the second dataset, the one or more conditions comprising a verification that a creation of the second dataset completed successfully and a verification that the second dataset is not stale.   
     
     
         14 . The system of  claim 9 , wherein the operations further comprise:
 responsive to receiving, via a user interface, a first request to protect the first test branch, preventing other users from further modifying the first test branch; and   responsive to receiving, via the user interface, a second request to further modify the first test branch, creating a child test branch associated with the first test branch.   
     
     
         15 . The system of  claim 9 , wherein the operations further comprise:
 responsive to receiving user input requesting a second test branch corresponding to the master branch, creating the second test branch, wherein the second test branch references the first dataset and a copy of the first logic;   receiving a request to modify the second test branch, the request comprising at least one change to the copy of the first logic;   modifying the second test branch independently of the master branch to include a third logic reflecting the at least one change to the copy of the first logic, the third logic to be applied to the first dataset to produce a fourth dataset; and   responsive to user input requesting a merge of the modified second test branch into the master branch, merging the modified second test branch into the master branch by causing the master branch to reference the first dataset, the third logic, and the fourth dataset.   
     
     
         16 . The system of  claim 9 , wherein the operations further comprise:
 rendering a visual representation of the data pipeline, including a first graph corresponding to the master branch and a second graph corresponding to the first test branch,   wherein the first graph includes a first node representing the first dataset, a second node representing the second dataset, and a first edge connecting the first node and the second node, wherein the first edge references the first logic to be applied to the first dataset in order to produce the second dataset, and   wherein the second graph includes a third node representing the first dataset, a fourth node representing the third dataset, and a second edge connecting the third node and the fourth node, wherein the second edge references the third logic to be applied to the first dataset in order to produce the third dataset.   
     
     
         17 . A non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
 identifying a master branch of a data processing pipeline comprising ordered data transformation operations, wherein the master branch references a first dataset, a first logic implementing a first data transformation operation, and a second dataset resulting from applying the first logic to the first dataset;   creating a first test branch of the data processing pipeline, wherein the first test branch references the first dataset and a second logic implementing a second data transformation operation;   generating a third dataset by applying the second logic to the first dataset; and   merging the first test branch into the master branch by causing the master branch to reference the first dataset, the second logic, and the third dataset.   
     
     
         18 . The non-transitory computer readable storage medium of  claim 17 , wherein merging the first test branch into the master branch further comprises:
 identifying a difference between the first logic and the second logic; and   receiving, via a user interface, an approval of the difference.   
     
     
         19 . The non-transitory computer readable storage medium of  claim 17 , wherein merging the first test branch into the master branch further comprises:
 identifying a difference between the second dataset logic and the third dataset; and   receiving, via a user interface, an approval of the difference.   
     
     
         20 . The non-transitory computer readable storage medium of  claim 17 , wherein merging the first test branch into the master branch further comprises:
 identifying a conflict between the first logic and the second logic; and   receiving, via a user interface, a selection of a third logic to resolve the conflict.