Method and system for development and tracking of dynamic data lineage

Abstract

A method and a system for dynamically tracking data lineage is provided. The method includes: assigning a first context key to a first data set; inputting the first data set into a first software module; receiving a second data set as an output of the first software module; and assigning a second context key to the second data set. The second context key includes information that identifies the inputted first data set and information that identifies the first software module. When the second data set is subsequently inputted into a second software module, a third context key that identifies the inputted second data set and the second software module is assigned to the resultant output data set.

What is claimed is: 
     
       1. A method for tracking a lineage of data, the method being implemented by at least one processor, the method comprising:
 assigning, by the at least one processor, a first context key to a first data set; 
 inputting, by the at least one processor, the first data set into a first software module from among a plurality of software modules; 
 receiving, by the at least one processor, a second data set as an output of the first software module; and 
 assigning, by the at least one processor, a second context key to the second data set, the second context key including information that identifies the inputted first data set and information that identifies the first software module. 
 
     
     
       2. The method of  claim 1 , further comprising:
 assigning, by the at least one processor, a third context key to a third data set; 
 inputting, by the at least one processor, each of the second data set and the third data set into a second software module from among the plurality of software modules; 
 receiving, by the at least one processor, a fourth data set as an output of the second software module; and 
 assigning, by the at least one processor, a fourth context key to the third data set, the fourth context key including information that identifies the inputted second data set, information that identifies the inputted third data set, and information that identifies the second software module. 
 
     
     
       3. The method of  claim 2 , wherein each of the first context key and the third context key includes information that uniquely identifies the corresponding data set and information that indicates a source of the corresponding data set. 
     
     
       4. The method of  claim 2 , further comprising registering each of the plurality of software modules, wherein the information that identifies the first software module and the information that identifies the second software module is determined based on a result of the registration. 
     
     
       5. The method of  claim 4 , wherein each of the first context key, the second context key, the third context key, and the fourth context key includes information that indicates a time at which the corresponding data set is originally generated. 
     
     
       6. The method of  claim 5 , further comprising:
 receiving a fifth data set that includes an assigned fifth context key; 
 determining, based on the fifth context key, an identification of a third software module from among the plurality of software modules from which the fifth data set is originally generated; and 
 determining, based on the fifth context key, an identification of at least one data set used as an input to the third software module in order to generate the fifth data set. 
 
     
     
       7. The method of  claim 6 , further comprising displaying, on a graphical user interface, a result of the tracking of the lineage of data that illustrates a data flow with respect to each of the first data set, the second data set, the third data set, and the plurality of software modules. 
     
     
       8. The method of  claim 7 , further comprising modifying the displayed result of the tracking of the lineage of data by including an indication that relates to an elimination of at least one data set based on conditional logic provided in at least one from among the plurality of software modules. 
     
     
       9. The method of  claim 7 , further comprising displaying, on the graphical user interface, a set of tables that correspond to the displayed result of the tracking of the lineage of data. 
     
     
       10. A computing apparatus for tracking a lineage of data, the computing apparatus comprising:
 a processor; 
 a memory; and 
 a communication interface coupled to each of the processor and the memory, 
 wherein the processor is configured to:
 assign a first context key to a first data set; 
 input the first data set into a first software module from among a plurality of software modules; 
 receive a second data set as an output of the first software module; and 
 assign a second context key to the second data set, the second context key including information that identifies the inputted first data set and information that identifies the first software module. 
 
 
     
     
       11. The computing apparatus of  claim 10 , wherein the processor is further configured to:
 assign a third context key to a third data set; 
 input each of the second data set and the third data set into a second software module from among the plurality of software modules; 
 receive a fourth data set as an output of the second software module; and 
 assign a fourth context key to the third data set, the fourth context key including information that identifies the inputted second data set, information that identifies the inputted third data set, and information that identifies the second software module. 
 
     
     
       12. The computing apparatus of  claim 11 , wherein each of the first context key and the third context key includes information that uniquely identifies the corresponding data set and information that indicates a source of the corresponding data set. 
     
     
       13. The computing apparatus of  claim 11 , wherein the processor is further configured to register each of the plurality of software modules, wherein the information that identifies the first software module and the information that identifies the second software module is determined based on a result of the registration. 
     
     
       14. The computing apparatus of  claim 13 , wherein each of the first context key, the second context key, the third context key, and the fourth context key includes information that indicates a time at which the corresponding data set is originally generated. 
     
     
       15. The computing apparatus of  claim 14 , wherein the processor is further configured to:
 receive, via the communication interface, a fifth data set that includes an assigned fifth context key; 
 determine, based on the fifth context key, an identification of a third software module from among the plurality of software modules from which the fifth data set is originally generated; and 
 determine, based on the fifth context key, an identification of at least one data set used as an input to the third software module in order to generate the fifth data set. 
 
     
     
       16. The computing apparatus of  claim 15 , wherein the processor is further configured to display, on a graphical user interface, a result of the tracking of the lineage of data that illustrates a data flow with respect to each of the first data set, the second data set, the third data set, and the plurality of software modules. 
     
     
       17. The computing apparatus of  claim 16 , wherein the processor is further configured to modify the displayed result of the tracking of the lineage of data by including an indication that relates to an elimination of at least one data set based on conditional logic provided in at least one from among the plurality of software modules. 
     
     
       18. The computing apparatus of  claim 16 , wherein the processor is further configured to display, on the graphical user interface, a set of tables that correspond to the displayed result of the tracking of the lineage of data. 
     
     
       19. A non-transitory computer readable storage medium storing instructions for tracking a lineage of data, the storage medium comprising executable code which, when executed by a processor, causes the processor to:
 assign a first context key to a first data set; 
 input the first data set into a first software module from among a plurality of software modules; 
 receive a second data set as an output of the first software module; and 
 assign a second context key to the second data set, the second context key including information that identifies the inputted first data set and information that identifies the first software module. 
 
     
     
       20. The storage medium of  claim 19 , wherein when executed by the processor, the executable code further causes the processor to:
 assign a third context key to a third data set; 
 input each of the second data set and the third data set into a second software module from among the plurality of software modules; 
 receive a fourth data set as an output of the second software module; and 
 assign a fourth context key to the third data set, the fourth context key including information that identifies the inputted second data set, information that identifies the inputted third data set, and information that identifies the second software module.