Framework for calculating grouped optimization algorithms within a distributed data store

Abstract

A framework for executing iterative grouped optimization algorithms such as machine learning and other analytic algorithms directly on unsorted data within a SQL data store without first redistributing the data comprises an architecture that provides C++ abstraction layers that include the algorithms over a SQL data store, and a higher Python abstraction layer that includes grouping and iteration controllers and call functionality to the C++ layer for invocation of the algorithms.

What is claimed is: 
     
         1 . A method of analyzing data within a distributed database having a plurality of database segments, comprising:
 grouping, using a grouping process running within the database, instances of data into a one or more groups without redistributing the data;   generating a predictive model for at least one of the one or more groups, wherein generating the predictive model includes running a first iteration of a first analytic algorithm within the database on each of the one or more groups;   updating the predictive model for the at least one of the one or more groups, wherein the updating of the predictive model comprises running subsequent iterations of a second analytic algorithm on the at least one of the one or more groups using as an input model for each subsequent iteration for each of the at least one of the one or more groups the predictive model for such group generated by a preceding iteration; and   updating the predictive model generated by the preceding iteration in the database with results of the updating of the predictive model based on the subsequent iteration.   
     
     
         2 . The method of  claim 1 , wherein the second analytic algorithm is the same as the first analytic algorithm. 
     
     
         3 . The method of  claim 1 , wherein the instances of data are grouped into the one or more groups such that each group comprises data instances having one or more common attribute values that characterize the group. 
     
     
         4 . The method of  claim 1 , wherein the grouping process comprises grouping the data groups without redistributing the data instances by running a GROUPBY query language operation for each group directly on data instances of a database segment. 
     
     
         5 . The method of  claim 4 , wherein the data instances comprise rows of table data in the database, and the attribute values that characterize the groups comprise one or more columns of the table data in each data instance. 
     
     
         6 . The method of  claim 1 , wherein the second analytic algorithm comprise an iterative machine learning algorithm that learns from the data groups to improve and update stored prediction models for each of the groups upon each iteration of the machine learning algorithm. 
     
     
         7 . The method of  claim 1 , wherein the running the first iteration comprises running the first analytic algorithm sequentially on each group within a database segment while concurrently running the first analytic algorithm on groups of all other database segments. 
     
     
         8 . The method of  claim 1 , wherein the running subsequent iterations comprises repeatedly running iterations of the second analytic algorithm for a predetermined number of iterations or until the models for each group converge, wherein each subsequent iteration runs the second analytic algorithm on the data groups using the updated predictive model from the preceding iteration. 
     
     
         9 . The method of  claim 8 , further comprising, upon the iterations reaching the predetermined number or upon the converging, aggregating and returning all grouped predictive models to a user. 
     
     
         10 . The method of  claim 8 , further comprising storing within the database the predictive models for each of the one or more groups from the first iteration, and updating the stored models using the results of the subsequent iterations. 
     
     
         11 . The method of  claim 1 , wherein the second analytic algorithm is implemented in a first application program within the database system, and the second analytic algorithm is invoked and iterated by a second application program within the database system that controls the first application program. 
     
     
         12 . A computer program product comprising a non-transitory computer readable medium storing executable instructions for controlling the operation of a computer in a distributed database having a plurality of database segments to perform a method comprising:
 grouping, using a grouping process running within the database, instances of data into a one or more groups without redistributing the data;   generating a predictive model for at least one of the one or more groups, wherein generating the predictive model includes running a first iteration of a first analytic algorithm within the database on each of the one or more groups;   updating the predictive model for the at least one of the one or more groups, wherein the updating of the predictive model comprises running subsequent iterations of a second analytic algorithm on the at least one of the one or more groups using as an input model for each subsequent iteration for each of the at least one of the one or more groups the predictive model for such group generated by a preceding iteration; and   updating the predictive model generated by the preceding iteration in the database with results of the updating of the predictive model based on the subsequent iteration.   
     
     
         13 . The computer program product of  claim 12 , wherein the second analytic algorithm is the same as the first analytic algorithm. 
     
     
         14 . The computer program product of  claim 12 , wherein the instances of data are grouped into the one or more groups such that each group comprises data instances having one or more common attribute values that characterize the group. 
     
     
         15 . The computer program product of  claim 12 , wherein the grouping process comprises grouping the data groups without redistributing the data instances by running a GROUPBY query language operation for each group directly on data instances of a database segment. 
     
     
         16 . The computer program product of  claim 12 , wherein the second analytic algorithm comprise an iterative machine learning algorithm that learns from the data groups to improve the prediction models associated with each of the groups upon each iteration of the machine learning algorithm. 
     
     
         17 . The computer program product of  claim 12 , wherein the running the first iteration comprises running the first analytic algorithm sequentially on each group within a database segment while concurrently running the first analytic algorithm on groups of all other database segments. 
     
     
         18 . The computer program product of  claim 12 , wherein the running subsequent iterations comprises repeatedly running iterations of the second analytic algorithm for a predetermined number of iterations or until the models for each group converge, wherein each iteration runs the second analytic algorithm on the data groups using the updated predictive model from the preceding iteration. 
     
     
         19 . The computer program product of  claim 18 , further comprising, upon the iterations reaching the predetermined number or upon the converging, aggregating and returning all grouped predictive models to a user. 
     
     
         20 . The computer program product of  claim 18 , further comprising storing within the database the predictive models for each of the one or more groups from the first iteration, and updating the stored models using the results of the subsequent iterations. 
     
     
         21 . The computer program product of  claim 12 , wherein the second analytic algorithm comprises first application program instructions running on the computer in the database system, and the second analytic algorithm is invoked and iterated by second application program layer instructions running on the computer in the database system, the second application program controlling the first application program. 
     
     
         22 . The computer system of  claim 21 , wherein the data instances comprise rows of table data in the database, and the attribute values that characterize the groups comprise one or more columns of the table data in each data instance. 
     
     
         23 . The computer system of  claim 21 , wherein the second analytical algorithm is one of a plurality of iterative machine learning algorithms in the first abstraction layer, and the second executable instructions interface with the first executable instructions to execute the analytic algorithm on the data instances. 
     
     
         24 . A computer system for a distributed database having a plurality of database segments, comprising:
 at least one processor configured to:
 group, using a grouping process running within the database, instances of data into a one or more groups without redistributing the data; 
 generate a predictive model for at least one of the one or more groups, wherein to generate the predictive model includes to run a first iteration of a first analytic algorithm within the database on each of the one or more groups; 
 update the predictive model for the at least one of the one or more groups, wherein to update of the predictive model comprises to run subsequent iterations of a second analytic algorithm on the at least one of the one or more groups using as an input model for each subsequent iteration for each of the at least one of the one or more groups the predictive model for such group generated by a preceding iteration; and 
 update the predictive model generated by the preceding iteration in the database with results of the updating of the predictive model based on the subsequent iteration 
   a memory coupled to the at least one processor and configured to provide the at least one processor with instructions.   
     
     
         25 . The computer system of  claim 24 , wherein the second analytic algorithm is the same as the first analytic algorithm. 
     
     
         26 . The computer system of  claim 24 , wherein the data instances comprise rows of table data in the database, and the attribute values that characterize the groups comprise one or more columns of the table data in each data instance. 
     
     
         27 . The computer system of  claim 24 , wherein the second analytical algorithm is one of a plurality of iterative machine learning algorithms in the first abstraction layer, and the second executable instructions interface with the first executable instructions to execute the analytic algorithm on the data instances.