Management and Evaluation of Machine-Learned Models Based on Locally Logged Data

Abstract

The present disclosure provides systems and methods for the management and/or evaluation of machine-learned models based on locally logged data. In one example, a user computing device can obtain a machine-learned model (e.g., from a server computing device) and can evaluate at least one performance metric for the machine-learned model. In particular, the at least one performance metric for the machine-learned model can be evaluated relative to data that is stored locally at the user computing device. The user computing device and/or the server computing device can determine whether to activate the machine-learned model on the user computing device based at least in part on the at least one performance metric. In another example, the user computing device can evaluate a plurality of machine-learned models against locally stored data. At least one of the models can be selected based on the evaluated performance metrics.

1 .- 20 . (canceled) 
     
     
         21 . A computer-implemented method to manage machine-learned models, the method comprising:
 obtaining, by a user computing device, a plurality of machine-learned models;   evaluating, by the user computing device, at least one performance metric for each of the plurality of machine-learned models, wherein the at least one performance metric for each machine-learned model is evaluated relative to data that is stored locally at the user computing device;   providing, by the user computing device, the performance metrics respectively evaluated for the plurality of machine-learned models to a server computing device;   receiving, by the user computing device, a selection of a first machine-learned model of the plurality of machine-learned models from the server computing device based at least in part on the performance metrics respectively evaluated for the plurality of machine-learned models; and   using, by the user computing device, the selected first machine-learned model to obtain one or more predictions.   
     
     
         22 . The computer-implemented method of  claim 21 , wherein evaluating, by the user computing device, the at least one performance metric for each of the plurality of machine-learned models comprises using, by the user computing device, historical data that was previously logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models. 
     
     
         23 . The computer implemented method of  claim 22 , wherein using, by the user computing device, the historical data that was previously logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models comprises:
 inputting, by the user computing device, at least a first portion of the historical data into each of the plurality of machine-learned models;   receiving, by the user computing device, at least one prediction from each of the plurality of machine-learned models in response to input of the first portion of the historical data; and   comparing, by the user computing device, the at least one prediction from each machine-learned model to a second portion of the historical data to respectively determine the at least one performance metric for each of the plurality of machine-learned models.   
     
     
         24 . The computer-implemented method of  claim 21 , wherein evaluating, by the user computing device, the at least one performance metric for each of the plurality of machine-learned models comprises, after obtaining, by the user computing device, the plurality of machine-learned models, using, by the user computing device, new data that is newly logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models. 
     
     
         25 . The computer-implemented method of  claim 24 , wherein using, by the user computing device, new data that is newly logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models comprises:
 inputting, by the user computing device, at least a first portion of the new data into each of the plurality of machine-learned models;   receiving, by the user computing device, at least one prediction from each of the plurality of machine-learned models in response to input of the first portion of the new data; and   comparing, by the user computing device, the at least one prediction from each machine-learned model to a second portion of the new data to respectively determine the at least one performance metric for each of the plurality of machine-learned models.   
     
     
         26 . The computer-implemented method of  claim 21 , wherein obtaining, by the user computing device, the plurality of machine-learned models comprises:
 receiving, by the user computing device, the plurality of machine-learned models from the server computing device.   
     
     
         27 . The computer-implemented method of  claim 21 , wherein the selection of the first machine-learned model from the plurality of machine-learned models is based at least in part on a global performance that has been aggregated from performance metrics reported by other computing devices. 
     
     
         28 . The computer-implemented method of  claim 21 , wherein evaluating, by the user computing device, at least one performance metric for each of the plurality of machine-learned models comprises evaluating, by the user computing device for each of the plurality of machine-learned models, an average probability of correct prediction. 
     
     
         29 . The computer-implemented method of  claim 21 , wherein the plurality of machine-learned models are evaluated prior to activation of any of the machine-learned models; and
 after the user computing device receives the selection of the first machine-learned model, the first machine-learned model is activated and used at the user computing device.   
     
     
         30 . The computer-implemented method of  claim 21 , further comprising:
 reporting, by the user computing device, one or more of the performance metrics evaluated for the plurality of machine-learned models to a server computing device.   
     
     
         31 . The computer-implemented method of  claim 30 , further comprising:
 aggregating, by the server computing device, the one or more performance metrics received from the user computing device with other performance metrics reported by other computing devices to assess a global performance for each of the plurality of machine-learned models.   
     
     
         32 . A computing device, the computing device comprising:
 one or more processors; and   one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to:
 obtain a machine-learned model, wherein the machine-learned model is one of a plurality of machine-learned models that have been previously trained on different types of training data; 
 train the machine-learned model further based on a first portion of data that is stored locally at the computing device to obtain a user-specific variant of the machine-learned model; 
 evaluate at least one performance metric for the user-specific variant of the machine-learned model, wherein the at least one performance metric for the user-specific variant of the machine-learned model is evaluated relative to a second portion of the data that is stored locally at the computing device; 
 determine whether to activate the user-specific variant of the machine-learned model based at least in part on the at least one performance metric evaluated for the user-specific variant of the machine-learned model; and 
 when it is determined that the user-specific variant of the machine-learned model should be activated, use the user-specific variant of the machine-learned model to obtain one or more predictions. 
   
     
     
         33 . The computing device of  claim 32 , wherein the computing device uses historical data that was previously logged at the user computing device prior to receipt of the machine-learned model to evaluate the at least one performance metric the machine-learned model. 
     
     
         34 . The computing device of  claim 32 , wherein the computing device uses new data that is newly logged at the user computing device after receipt of the machine-learned model to evaluate the at least one performance metric for each of the plurality of machine-learned models. 
     
     
         35 . The computing device of  claim 32 , wherein to determine whether to activate the user-specific variant of the machine-learned model based at least in part on the at least one performance metric the computing device:
 compares the at least one performance metric to at least one threshold value; and   determines whether the at least one performance metric compares favorably to the at least one threshold value, wherein it is determined that the user-specific variant of the machine-learned model should be activated when the at least one performance metric compares favorably to the at least one threshold value.   
     
     
         36 . The computing device of  claim 32 , wherein to determine whether to activate the user-specific variant of the machine-learned model based at least in part on the at least one performance metric the computing device:
 provides the at least one performance metric evaluated for the user-specific variant of the machine-learned model to a server computing device; and   receives a determination of whether to activate the user-specific variant of the machine-learned model from the server computing device.   
     
     
         37 . The computing device of  claim 32 , wherein the at least one performance metric comprises an average probability of correct prediction. 
     
     
         38 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors of a server computing system, cause the server computing system to:
 obtain a plurality of performance metrics respectively associated with a plurality of machine-learned models, wherein the performance metric for each machine-learned model indicates a performance of such machine-learned model when evaluated against a set of data that is stored locally at a user computing device;   select at least one of the plurality of machine-learned models based at least in part on the plurality of performance metrics; and   cause use of the selected at least one machine-learned model at the user computing device.   
     
     
         39 . The one or more non-transitory computer-readable media of  claim 38 , wherein to obtain the plurality of performance metrics, the server computing device receives the plurality of performance metrics from the user computing device. 
     
     
         40 . The one or more non-transitory computer-readable media of  claim 39 , wherein execution of the instructions further causes the server computing device to aggregate the plurality of performance metrics received from the user computing device with a plurality of additional performance metrics received from additional computing devices to determine a global performance for each of the plurality of machine-learned models.