Processing Sensor Data with Multi-Model System on Resource-Constrained Device

Abstract

Methods, systems, and computer-readable media for multi-model processing on resource-constrained devices. A resource-constrained device can determine, based on a battery-life for a battery of the device, whether to process input through a first model or a second model. The first model can be a gating model that is more energy efficient to execute, and the second model can be a main model that is more accurate than the gating model. Depending on the current battery-life and/or other criteria, the system can process, through the gating model, sensor input that can record activity performed by a user of the resource-constrained device. If the gating model predicts an activity performed by the user that is recorded by the sensor data, the device can process the same or additional input through the main model. Overall power consumption can be reduced with a minimum accuracy maintained over processing input only through the main model.

What is claimed is: 
     
         1 . A computing device comprising one or more processors configured to:
 receive an input;   determine whether the device is operating in a resource-constrained mode; and   determine, based on at least the determination of whether the device is operating in a resource-constrained mode, whether to process the input using a first machine learning model or a second machine learning model,   the first machine learning model trained to generate, from the input, a first output with an average accuracy meeting a minimum accuracy threshold, and   the second machine learning model trained to generate, from the input, a second output with an average accuracy exceeding the minimum accuracy threshold.   
     
     
         2 . The computing device of  claim 1 , wherein the one or more processors are further configured to, when the first machine learning model predicts that a primary input recorded an activity, process a secondary input through the second machine learning model to generate the second output predicting whether the secondary input recorded the activity. 
     
     
         3 . The computing device of  claim 1 , wherein the first machine learning model requires fewer operations to process the input than the second machine learning model requires to process the input. 
     
     
         4 . The computing device of  claim 1 , wherein the determination of whether to use the first machine learning model or the second machine learning model is based at least in part on how each machine learning model affects a resource of the device. 
     
     
         5 . The computing device of  claim 2  further comprising one or more sensors, wherein:
 the primary input is obtained from a first sensor of the one or more sensors operating at a first power level; and 
 the secondary input is at least partially obtained from the first sensor operating at a second power level higher than the first power level. 
 
     
     
         6 . The computing device of  claim 1 , wherein the computing device further comprises a display, and wherein the one or more processors are further configured to determine, from the second output, that the second machine learning model predicted that the input recorded an activity, and in response generate and transmit a third output to the display. 
     
     
         7 . The computing device of  claim 1  further comprising one or more sensors, wherein:
 the first machine learning model and the second machine learning model are trained to perform a machine learning task, wherein the machine learning task comprises detecting a plurality of types of activities recorded by at least one sensor of the one or more sensors; and 
 the second machine learning model is trained to detect at least one type of the plurality of types with an average accuracy greater than the first machine learning model. 
 
     
     
         8 . The computing device of  claim 1 , wherein the computing device further comprises a battery, and the one or more processors are further configured to:
 receive a current battery-life of the battery indicating a change in available energy stored in the battery; and   determine to process the input through the first machine learning model:
 at a first frequency in response to the current battery-life indicating a change in more available energy stored in the battery; and 
 at a second frequency in response to the current battery-life indicating a change in less available energy stored in the battery, wherein the first frequency is greater than the second frequency. 
   
     
     
         9 . The computing device of  claim 1 , wherein the computing device further comprises a battery, and the one or more processors are further configured to:
 receive one or more performance thresholds corresponding to an average measure of performance for the first machine learning model under respective one or more conditions and that are equal to or greater than a minimum average threshold;   receive a current battery-life of the battery indicating a change in available energy stored in the battery;   determine, based at least on the current battery-life, whether to cause the first machine learning model to perform at a level of performance meeting a performance threshold of the one or more performance thresholds; and   in response to the determination to cause the first machine learning model to perform at the level of performance, cause the first machine learning model to operate under respective one or more conditions for the performance threshold.   
     
     
         10 . The computing device of  claim 1 , wherein the first and the second machine learning models are neural networks, the second machine learning model having a first plurality of layers, and the first machine learning model having a second plurality of layers that is a subset of the first plurality of layers. 
     
     
         11 . The computing device of  claim 1 , wherein the first machine learning model is of a different model architecture than the second machine learning model. 
     
     
         12 . A method comprising:
 determining whether a computing device is operating in a resource-constrained mode; and   determining, based on at least the determination of whether the computing device is operating in a resource-constrained mode, whether to process an input through a first machine learning model or a second machine learning model, wherein:
 the first and second machine learning models are implemented by a computing device; 
 the first machine learning model is trained to generate, from the input, a first output with an average accuracy meeting a minimum accuracy threshold; and 
 the second machine learning model is trained to generate, from the input, a second output with an average accuracy exceeding the minimum accuracy threshold. 
   
     
     
         13 . The method of  claim 12 , further comprising:
 processing a secondary input through the second machine learning model when the first machine learning model predicts that a primary input recorded an activity; and   generating the second output predicting whether the secondary input recorded the activity.   
     
     
         14 . The method of  claim 12 , wherein the first machine learning model requires fewer operations to process the input than the second machine learning model requires to process the input. 
     
     
         15 . The method of  claim 12 , wherein the determination of whether to use the first machine learning model or the second machine learning model is based at least in part on how each machine learning model affects a resource of a computing device. 
     
     
         16 . The method of  claim 13 , wherein:
 obtaining the primary input comprises obtaining sensor input from a first sensor of one or more sensors operating at a first power level; and   the secondary input is at least partially obtained by the first sensor operating at a second power level higher than the first power level.   
     
     
         17 . The method of  claim 12 , wherein the method further comprises determining, from the second output, that the second machine learning model predicted that the input recorded an activity, and in response generating and transmitting a third output to a display of a computing device. 
     
     
         18 . The method of  claim 12 , wherein determining to process the input through the first machine learning model comprises:
 receiving a current battery-life of a battery of a computing device, the current battery-life indicating a change in available energy stored in the battery; and   determining to process the input through the first machine learning model:
 at a first frequency in response to the current battery-life indicating a change in more available energy stored in the battery; and 
 at a second frequency in response to the current battery-life indicating a change in less available energy stored in the battery, wherein the first frequency is greater than the second frequency. 
   
     
     
         19 . The method of  claim 12 , wherein determining to process the input through the first machine learning model comprises:
 receiving one or more performance thresholds corresponding to an average measure of performance for the first machine learning model under respective one or more conditions and that are equal to or greater than a minimum average threshold;   receiving a current battery-life of a battery of a computing device, the current battery-life indicating a change in available energy stored in the battery;   determining, based at least on the current battery-life, whether to cause the first machine learning model to perform at a level of performance meeting a performance threshold of the one or more performance thresholds; and   in response to determining to cause the first machine learning model to perform at the level of performance, causing the first machine learning model to operate under respective one or more conditions for the performance threshold.   
     
     
         20 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing device, causes the one or more processors to perform operations comprising:
 determining whether the computing device is operating in a resource-constrained mode; and   determining, based on at least the determination of whether the computing device is operating in a resource-constrained mode, whether to process an input through a first machine learning model or a second machine learning model, wherein:
 the first and second machine learning models are implemented by the computing device; 
 first machine learning model is trained to generate a first output with an average accuracy meeting a minimum accuracy threshold; and 
 the second machine learning model is trained to generate a second output with an average accuracy exceeding the minimum accuracy threshold.