Automatically scalable system for serverless hyperparameter tuning

Abstract

A scalable system and method for completing a model task using a serverless architecture is disclosed. The system may include memory storing instructions and one or more processors. The method may include receiving a request to complete a model task; retrieving a first model and a first hyperparameter based on the request; provisioning computing resources to a first development instance configured to train the first model based on the first hyperparameter and the model task; training, by the first development instance, an instance of the first model to produce a trained model and terminating said training upon satisfaction of a training criterion; receiving the trained model and a first performance metric; receiving a second performance metric associated with a second model; and terminating the development instance based on a determination that the termination condition is satisfied based on at least one of the first and second performance metrics.

1 - 20 . (canceled) 
     
     
         21 . A scalable system for completing a model task using a serverless architecture, the system comprising:
 a model optimizer comprising:
 memory for storing instructions; and 
 one or more processors configured to execute the stored instructions to perform operations comprising:
 receiving a request to complete a model task; 
 retrieving a first model from a model storage by selecting, based on the model task, the first model in an index of stored models; 
 retrieving a first hyperparameter based on the first model and the model task; 
 provisioning computing resources to a first development instance configured to train the first model based on the first hyperparameter and the model task; 
 creating an instance of the first model from the first development instance; 
 retrieving, by the first development instance, training data comprising synthetic data based on actual data; 
 training the instance of the first model by the first development instance using the training data to obtain a first trained model; 
 terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a dataset generated by the trained model and a reference dataset, a data similarity score between the dataset generated by the trained model and the reference dataset, or a data quality score for the dataset generated by the trained model; 
 generating, by the first development instance, a first performance metric of the trained model; 
 receiving, from the first development instance, the trained model and the first performance metric; 
 receiving, from a second development instance, a second performance metric associated with a second model; 
 determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and 
 terminating the first development instance based on the determination that the termination condition is satisfied. 
 
   
     
     
         22 . The scalable system of  claim 21 , wherein the model task is at least one of a classification task, a prediction task, or a regression task. 
     
     
         23 . The scalable system of  claim 21 , wherein the model storage comprises a cloud-based database. 
     
     
         24 . The scalable system of  claim 21 , wherein the index of stored models is configured to identify a suitable model based on the model task. 
     
     
         25 . The scalable system of  claim 21 , wherein retrieving the first model from the model storage is further based on a modeling characteristic, the modeling characteristic being at least one of a model type, a data schema, a training dataset type, or a training dataset identifier. 
     
     
         26 . The scalable system of  claim 21 , wherein the retrieved first hyperparameter is at least one of a training hyperparameter or an architectural hyperparameter. 
     
     
         27 . The scalable system of  claim 21 , wherein the first development instance is further configured to retrieve the training data from a dataset generator or a training data database. 
     
     
         28 . The scalable system of  claim 21 , wherein the training data comprises synthetic data generated by a generative adversarial network based on actual data. 
     
     
         29 . The scalable system of  claim 21 , wherein the first performance metric depends on a similarity between data generated by the trained model and the training data. 
     
     
         30 . The scalable system of  claim 21 , wherein the termination condition is based on at least one of a threshold value of the first performance metric, an improvement rate of the first performance metric, a threshold value of the second performance metric, or an improvement rate of the second performance metric. 
     
     
         31 . The scalable system of  claim 21 , the operations further comprising:
 storing the trained model in the model storage; and   updating the index of stored models.   
     
     
         32 . The scalable system of  claim 21 , wherein the operations further comprise, before determining whether the termination condition is satisfied:
 retrieving a second hyperparameter based on the first performance metric;   providing, to the first development instance, a command to generate a second trained model by training the first trained model based on the second hyperparameter;   receiving, from the first development instance, the second trained model and a model output associated with the second trained model; and   updating the first performance metric based on the model output.   
     
     
         33 . The scalable system of  claim 32 , wherein the first hyperparameter is an architectural hyperparameter and the second hyperparameter is a training hyperparameter. 
     
     
         34 . The scalable system of  claim 32 , wherein retrieving the second hyperparameter is further based on a search strategy. 
     
     
         35 . The scalable system of  claim 21 , wherein the operations further comprise performing, before determining whether the termination condition is satisfied:
 receiving a request to complete a new model task;   retrieving a second hyperparameter based on the first trained model and the new model task;   provisioning new computing resources to the first development instance, the first development instance being configured to train the first trained model based on the second hyperparameter and the new model task;   receiving, from the first development instance, a model output; and   updating the first performance metric based on the model output.   
     
     
         36 . The scalable system of  claim 21 , wherein the operations further comprise, before determining whether the termination condition is satisfied:
 receiving a request to complete a new model task;   retrieving a third model from the model storage based on the new model task;   retrieving a second hyperparameter based on the first trained model and the new model task;   providing, to the first development instance, a command to train the third model based on the second hyperparameter; and   receiving a third performance metric from the first development instance,   wherein determining whether the termination condition is satisfied is further based on the third performance metric.   
     
     
         37 . A method for completing a model task using a serverless architecture comprising:
 receiving a request to complete a model task;   retrieving a first model from a model storage by selecting the first model in an index of stored models based on the model task;   retrieving a hyperparameter based on the first model and the model task;   provisioning computing resources to a first development instance configured to train the first model based on the retrieved hyperparameter and the model task;   creating an instance of the first model from the first development instance;   retrieving, by the first development instance, training data comprising synthetic data based on actual data;   training the instance of the first model by the first development instance using the training data to obtain a trained model;   terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a dataset generated by the trained model and the reference dataset, a data similarity score between the dataset generated by the trained model and the reference dataset, or a data quality score for the dataset generated by the trained model;   generating, by the first development instance, a first performance metric of the trained model;   receiving, from the first development instance, the trained model and the first performance metric;   receiving, from a second development instance, a second performance metric associated with a second model;   determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and   terminating the first development instance based on the determination that the termination condition is satisfied.   
     
     
         38 . A scalable system for completing a model task using a serverless architecture, the system comprising:
 a dataset generator comprising:
 memory for storing instructions; and 
 one or more processors configured to execute the stored instructions to perform first operations comprising generating training data based on actual data; and 
   a model optimizer comprising:
 memory for storing instructions; and 
 one or more processors configured to execute the stored instructions to perform second operations comprising:
 receiving a request to complete a model task; 
 retrieving a first model from a model storage by selecting the first model in an index of stored models based on the model task; 
 retrieving a hyperparameter based on the first model and the model task; 
 provisioning computing resources to a first development instance configured to train the first model based on the retrieved hyperparameter and the model task; 
 creating an instance of the first model from a first development instance; 
 training the instance of the first model by the first development instance using the training data to obtain a trained model; 
 terminating, by the first development instance, the training upon satisfaction of a training criterion, wherein the training criterion is associated with at least one of a statistical correlation score between a dataset generated by the trained model and the reference dataset, a data similarity score between the dataset generated by the trained model and the reference dataset, or a data quality score for the dataset generated by the trained model; 
 generating, by the first development instance, a first performance metric of the trained model; 
 receiving, from the first development instance, the trained model and the first performance metric; 
 receiving, from a second development instance, a second performance metric associated with a second model; 
 determining that a termination condition is satisfied based on at least one of the first performance metric or the second performance metric; and 
 terminating the first development instance based on the determination that the termination condition is satisfied. 
 
   
     
     
         39 . The system of  claim 38 , wherein the training data comprises synthetic data generated by a generative adversarial network based on the actual data. 
     
     
         40 . The system of  claim 38 , wherein the first operations further comprise:
 identifying at least one sensitive data item in the actual data; and   generating at least one synthetic data item to replace the at least one sensitive data item in the actual data.