Dynamic Predictive Modeling Platform

Abstract

Methods, systems, and apparatus, including computer programs encoded on one or more computer storage devices, for training and retraining predictive models. A series of training data sets are received and added to a training data queue. In response to a first condition being satisfied, multiple retrained predictive models are generated using the training data queue, multiple updateable trained predictive models obtained from a repository of trained predictive models, and multiple training functions. In response to a second condition being satisfied, multiple new trained predictive models are generated using the training data queue, at least some training data stored in a training data repository and training functions. The new trained predictive models include static trained predictive models and updateable trained predictive models. The repository of trained predictive models is updated with at least some of the retrained predictive models and new trained predictive models.

What is claimed is: 
     
         1 . A system comprising:
 one or more computers; and   one or more storage devices coupled to the one or more computers and storing:
 a repository of training functions, 
 a repository of trained predictive models comprising static trained predictive models and updateable trained predictive models, 
 a training data queue, 
 a training data repository, and 
 instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
 receiving a series of training data sets; 
 adding the training data sets to the training data queue; 
 in response to a first condition being satisfied, generating a plurality of retrained predictive models using the training data queue, a plurality of updateable trained predictive models obtained from the repository of trained predictive models, and a plurality of training functions obtained from the repository of training functions; 
 updating the repository of trained predictive models by storing one or more of the plurality of generated retrained predictive models; 
 in response to a second condition being satisfied, generating a plurality of new trained predictive models using the training data queue and at least some of the training data stored in the training data repository and using a plurality of training functions obtained from the repository of training functions, wherein the plurality of new trained predictive models comprise static trained predictive models and updateable trained predictive models; and 
 updating the repository of trained predictive models by storing at least some of the plurality of new trained predictive models. 
 
   
     
     
         2 . The system of  claim 1 , wherein the series of training data sets are received incrementally. 
     
     
         3 . The system of  claim 1 , wherein the series of training data sets are received together in a batch. 
     
     
         4 . The system of  claim 1 , wherein the first condition is satisfied when a size of the training data queue is greater than or equal to a threshold size. 
     
     
         5 . The system of  claim 1 , wherein the first condition is satisfied in response to receiving a command to update the plurality of updateable trained predictive models included in the repository of trained predictive models. 
     
     
         6 . The system of  claim 1 , wherein the first condition is satisfied after a predetermined time period has expired. 
     
     
         7 . The system of  claim 1 , wherein the second condition is satisfied in response to receiving a command to update the static models and the updateable models included in the repository of trained predictive models. 
     
     
         8 . The system of  claim 1 , wherein the second condition is satisfied after a predetermined time period has expired. 
     
     
         9 . The system of  claim 1 , wherein the second condition is satisfied when a size of the training data queue is greater than or equal to a threshold size. 
     
     
         10 . The system of  claim 1 , further comprising:
 a user interface configured to receive user input specifying a data retention policy that defines rules for maintaining and deleting training data included in the training data repository.   
     
     
         11 . The system of  claim 1 , where the operations further comprise:
 generating updated training data that includes at least some of the training data from the training data queue and at least some of the training data from the training data repository; and   updating the training data repository by storing the updated training data.   
     
     
         12 . The system of  claim 11 , wherein generating updated training data comprises implementing a data retention policy that defines rules for maintaining and deleting training data included in at least one of the training data queue or the training data repository. 
     
     
         13 . The system of  claim 12 , wherein the data retention policy includes a rule for deleting training data from the training data repository when the training data repository size reaches a predetermined size limit. 
     
     
         14 . The system of  claim 1 , wherein updating the repository of trained predictive models by storing one or more of the plurality of generated retrained predictive models comprises:
 for each of the plurality of retrained predictive models:
 comparing an effectiveness score of the retrained predictive model to an effectiveness score of the updateable trained predictive model from the predictive model repository that was used to generate the retrained predictive model; and 
 based on the comparison, selecting a first of the two predictive models to store in the repository of predictive models and not storing a second of the two predictive models in the repository; 
   wherein the effectiveness scores are each scores that represents an estimation of the effectiveness of the respective trained predictive model.   
     
     
         15 . A computer-implemented method comprising:
 receiving new training data;   adding the new training data to a training data queue;   determining whether a size of the training data queue size is greater than a threshold size;   when the training data queue size is greater than the threshold size, retrieving a stored plurality of trained predictive models and a stored training data set, wherein each of the trained predictive models were generated using the training data set and a plurality of training functions, and wherein each of the trained predictive models is associated with a score that represents an estimation of the effectiveness of the predictive model;   generating a plurality of retrained predictive models using the training data queue, the retrieved plurality of trained predictive models and the plurality of training functions;   generating a new score associated each of the generated retrained predictive models; and   adding at least some of the training data queue to the stored training data set.   
     
     
         16 . The method of  claim 15 , wherein the threshold is a predetermined data size. 
     
     
         17 . The method of  claim 15 , wherein the threshold is a predetermined ratio of the training data queue size to a size of the stored training data set. 
     
     
         18 . A computer-implemented method comprising:
 receiving a series of training data sets;   adding the training data sets to a training data queue;   in response to a first condition being satisfied, generating a plurality of retrained predictive models using the training data queue, a plurality of updateable trained predictive models obtained from a repository of trained predictive models, and a plurality of training functions obtained from a repository of training functions;   updating the repository of trained predictive models by storing one or more of the plurality of generated retrained predictive models;   in response to a second condition being satisfied, generating a plurality of new trained predictive models using the training data queue and at least some of training data stored in a training data repository and using a plurality of training functions obtained from the repository of training functions, wherein the plurality of new trained predictive models comprise static trained predictive models and updateable trained predictive models; and   updating the repository of trained predictive models by storing at least some of the plurality of new trained predictive models.   
     
     
         19 . The method of  claim 18 , wherein the first condition is satisfied when a size of the training data queue is greater than or equal to a threshold size. 
     
     
         20 . The method of  claim 18 , wherein the second condition is satisfied when a predetermined period of time has expired.