Intelligent data processing system with metadata generation from iterative data analysis

Abstract

A method includes obtaining a first data model from a data exploration phase performed in a first environment, where the first data model includes first metadata. The method also includes obtaining a second data model from the data exploration phase performed in a second environment different from the first environment, where the second data model includes second metadata. The method further includes generating a third data model including one or more software artifacts using the first metadata and the second metadata. Each of the one or more software artifacts is configured as one or more files that are configured for execution of at least one artificial intelligence (AI)/machine learning (ML) application.

What is claimed is: 
     
         1 . A method comprising:
 obtaining a first data model from a data exploration phase performed in a first environment, the first data model comprising first metadata;   obtaining a second data model from the data exploration phase performed in a second environment different from the first environment, the second data model comprising second metadata; and   generating a third data model comprising one or more software artifacts using the first metadata and the second metadata;   wherein each of the one or more software artifacts is configured as one or more files that are configured for execution of at least one artificial intelligence (AI)/machine learning (ML) application.   
     
     
         2 . The method of  claim 1 , wherein:
 generating the third data model comprises generating third metadata associated with the third data model using the first metadata and the second metadata; and   each of the first, second, and third metadata comprises information defining data transformations for creating one or more features or feature sets for use in a machine learning model.   
     
     
         3 . The method of  claim 1 , wherein generating the third data model comprises:
 performing one or more operations on at least one of the first data model and the second data model, the one or more operations defining one or more data transformations; and   generating the one or more software artifacts using the one or more data transformations.   
     
     
         4 . The method of  claim 3 , wherein the one or more operations are performed using an intermediate representation that maintains a sequence of the one or more data transformations, the intermediate representation comprising a context associated with the one or more data transformations. 
     
     
         5 . The method of  claim 3 , wherein generating the third data model further comprises combining at least a portion of a first graph associated with the first data model and at least a portion of a second graph associated with the second data model into a third graph associated with the third data model. 
     
     
         6 . The method of  claim 1 , wherein generating the third data model comprises iteratively generating multiple versions of the third data model based on input from multiple users. 
     
     
         7 . The method of  claim 1 , wherein the one or more files are human-readable and machine-executable. 
     
     
         8 . An apparatus comprising:
 at least one processing device configured to:
 obtain a first data model from a data exploration phase performed in a first environment, the first data model comprising first metadata; 
 obtain a second data model from the data exploration phase performed in a second environment different from the first environment, the second data model comprising second metadata; and 
 generate a third data model comprising one or more software artifacts using the first metadata and the second metadata; 
   wherein each of the one or more software artifacts is configured as one or more files that are configured for execution of at least one artificial intelligence (AI)/machine learning (ML) application.   
     
     
         9 . The apparatus of  claim 8 , wherein:
 to generate the third data model, the at least one processing device is configured to generate third metadata associated with the third data model using the first metadata and the second metadata; and   each of the first, second, and third metadata comprises information defining data transformations for creating one or more features or feature sets for use in a machine learning model.   
     
     
         10 . The apparatus of  claim 8 , wherein, to generate the third data model, the at least one processing device is configured to:
 perform one or more operations on at least one of the first data model and the second data model, the one or more operations defining one or more data transformations; and   generate the one or more software artifacts using the one or more data transformations.   
     
     
         11 . The apparatus of  claim 10 , wherein the at least one processing device is configured to perform the one or more operations using an intermediate representation that maintains a sequence of the one or more data transformations, the intermediate representation comprising a context associated with the one or more data transformations. 
     
     
         12 . The apparatus of  claim 10 , wherein, to generate the third data model, the at least one processing device is further configured to combine at least a portion of a first graph associated with the first data model and at least a portion of a second graph associated with the second data model into a third graph associated with the third data model. 
     
     
         13 . The apparatus of  claim 8 , wherein, to generate the third data model, the at least one processing device is configured to iteratively generate multiple versions of the third data model based on input from multiple users. 
     
     
         14 . The apparatus of  claim 8 , wherein the one or more files are human-readable and machine-executable. 
     
     
         15 . A non-transitory computer readable medium containing computer readable program code that when executed causes one or more processors to:
 obtain a first data model from a data exploration phase performed in a first environment, the first data model comprising first metadata;   obtain a second data model from the data exploration phase performed in a second environment different from the first environment, the second data model comprising second metadata; and   generate a third data model comprising one or more software artifacts using the first metadata and the second metadata;   wherein each of the one or more software artifacts is configured as one or more files that are configured for execution of at least one artificial intelligence (AI)/machine learning (ML)application.   
     
     
         16 . The non-transitory computer readable medium of  claim 15 , wherein:
 the computer readable program code that when executed causes the one or more processors to generate the third data model comprises:
 computer readable program code that when executed causes the one or more processors to generate third metadata associated with the third data model using the first metadata and the second metadata; and 
   each of the first, second, and third metadata comprises information defining data transformations for creating one or more features or feature sets for use in a machine learning model.   
     
     
         17 . The non-transitory computer readable medium of  claim 15 , wherein the computer readable program code that when executed causes the one or more processors to generate the third data model comprises:
 computer readable program code that when executed causes the one or more processors to:
 perform one or more operations on at least one of the first data model and the second data model, the one or more operations defining one or more data transformations; and 
 generate the one or more software artifacts using the one or more data transformations. 
   
     
     
         18 . The non-transitory computer readable medium of  claim 17 , wherein the computer readable program code when executed causes the one or more processors to perform the one or more operations using an intermediate representation that maintains a sequence of the one or more data transformations, the intermediate representation comprising a context associated with the one or more data transformations. 
     
     
         19 . The non-transitory computer readable medium of  claim 17 , wherein the computer readable program code that when executed causes the one or more processors to generate the third data model further comprises:
 computer readable program code that when executed causes the one or more processors to combine at least a portion of a first graph associated with the first data model and at least a portion of a second graph associated with the second data model into a third graph associated with the third data model.   
     
     
         20 . The non-transitory computer readable medium of  claim 15 , wherein the computer readable program code that when executed causes the one or more processors to generate the third data model comprises:
 computer readable program code that when executed causes the one or more processors to iteratively generate multiple versions of the third data model based on input from multiple users.