Predictive Modeling from Distributed Datasets

Abstract

Techniques for using data sets for a predictive model are described. According to various implementations, techniques described herein enable different data sets to be used to generate a predictive model, while minimizing the risk that individual data points of the data sets will be exposed by the predictive model. This aids in protecting individual privacy (e.g., protecting personally identifying information for individuals), while enabling robust predictive models to be generated using data sets from a variety of different sources

What is claimed is: 
     
         1 . A system comprising:
 at least one processor; and   one or more computer-readable storage media including instructions stored thereon that, responsive to execution by the at least one processor, cause the system perform operations including:
 calculating a gradient value based on a data set applied to a data model, the gradient value including a weight value calculated for the data model; 
 communicating the gradient value to an external service; 
 receiving an average gradient value from the external service; 
 applying the average gradient value to the data model; and 
 obtaining, based on ascertaining that a termination criterion occurs, a predictive model that represents a trained version of the data model. 
   
     
     
         2 . A system as recited in  claim 1 , wherein said calculating comprises using a backpropagation procedure to train the data model using the data set. 
     
     
         3 . A system as recited in  claim 1 , wherein said calculating comprises:
 dividing the data set into a set of mini-batches; and   calculating the gradient value using a particular mini-batch of the set of mini-batches.   
     
     
         4 . A system as recited in  claim 1 , wherein said calculating comprises:
 dividing the data set into a set of mini-batches; and   calculating the gradient value using a particular mini-batch of the set of mini-batches, wherein the termination criterion comprises determining that each mini-batch of the set of mini-batches is evaluated to generate a respective gradient value.   
     
     
         5 . A system as recited in  claim 1 , wherein said applying comprises applying the average gradient value to update a weight value of the data model. 
     
     
         6 . A system as recited in  claim 1 , wherein the predictive model comprises a neural network trained using the average gradient value. 
     
     
         7 . A system as recited in  claim 1 , wherein the operations further include:
 applying a set of input data to the predictive model;   ascertaining an output of the predictive model; and   performing an action based on the output of the predictive model.   
     
     
         8 . A computer-implemented method, comprising:
 receiving multiple gradient values from multiple different source systems;   generating an average gradient value from the multiple gradient values;   adding a noise term to the average gradient value to generate a noisy gradient average;   communicating the noisy gradient average to the multiple different source systems; and   obtaining a predictive model trained using the noisy gradient average.   
     
     
         9 . A method as described in  claim 8 , wherein said adding the noise term comprises adding a Laplace-distributed random number to the average gradient value to generate the noisy gradient average. 
     
     
         10 . A method as described in  claim 8 , wherein said adding the noise term comprises performing a garbled circuits protocol using the average gradient value. 
     
     
         11 . A method as described in  claim 8 , wherein the predictive model comprises a neural network trained using the noisy gradient average. 
     
     
         12 . A computer-implemented method, comprising:
 calculating a gradient value based on a data set applied to a data model;   generating a perturbed gradient value based on the gradient value and a perturbation value;   communicating the perturbed gradient value to a first host system;   communicating the perturbation value to a second host system;   receiving an average gradient value from one or more of the first host system or the second host system, the average gradient value calculated based on the perturbed gradient value and the perturbation value;   applying the average gradient value to the data model; and   obtaining a predictive model that represents a trained version of the data model, the data model trained at least in part using the average gradient value.   
     
     
         13 . A method as described in  claim 12 , wherein said calculating comprises applying backpropagation to the data model and using the data set to calculate the gradient value. 
     
     
         14 . A method as described in  claim 12 , wherein said calculating comprises:
 dividing the data set into a set of mini-batches; and   calculating the gradient value using a particular mini-batch of the set of mini-batches.   
     
     
         15 . A method as described in  claim 12 , wherein said generating the perturbed gradient value comprises generating the perturbation value as a random vector, and adding the random vector to the gradient value to generate the perturbed gradient value. 
     
     
         16 . A method as described in  claim 12 , wherein said applying comprises applying a weight value from the average gradient value to the data model. 
     
     
         17 . A method as described in  claim 12 , wherein said obtaining is performed in response to ascertaining that a termination criterion occurs. 
     
     
         18 . A method as described in  claim 12 , wherein the average gradient value is calculated using a garbled circuits protocol. 
     
     
         19 . A method as described in  claim 12 , wherein the predictive model comprises a neural network trained using the average gradient value. 
     
     
         20 . A method as described in  claim 12 , further comprising:
 applying a set of input data to the predictive model;   ascertaining an output of the predictive model;   performing an action based on the output of the predictive model.