Machine-learning based gesture recognition

Abstract

The subject technology receives, from a first sensor of a device, first sensor output of a first type. The subject technology receives, from a second sensor of the device, second sensor output of a second type, the first and second sensors being non-touch sensors. The subject technology provides the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted touch-based gesture based on sensor output of the first type and sensor output of the second type. The subject technology provides a predicted touch-based gesture based on output from the machine learning model. Further, the subject technology adjusts an audio output level of the device based on the predicted gesture, and where the device is an audio output device.

1 - 20 . (canceled) 
     
     
         21 . A method, comprising:
 providing sensor data as input to a machine learning model, the machine learning model having been trained to predict, based on the sensor data, multiple features of a gesture, and   obtaining a predicted gesture based on an output from the machine learning model, the output generated in response to the providing the sensor data as input to the machine learning model, wherein the output comprises at least one of the multiple features of the gesture; and   adjusting a function of a device based on the predicted gesture.   
     
     
         22 . The method of  claim 21 , wherein the gesture comprises a touch-based gesture on a surface of the device, and wherein the sensor data comprises sensor data from a non-touch sensor of the device. 
     
     
         23 . The method of  claim 21 , wherein the multiple features of the gesture comprise at least a start portion of the gesture, a middle portion of the gesture, and an end portion of the gesture. 
     
     
         24 . The method of  claim 23 , wherein the multiple features of the gesture further comprise a non-gesture. 
     
     
         25 . The method of  claim 24 , wherein the gesture comprises a swipe up gesture, wherein the non-gesture comprises a non-swipe, and wherein the multiple features of the gesture comprise at least: a start swipe up, a middle swipe up, an end swipe up, and the non-swipe. 
     
     
         26 . The method of  claim 25 , wherein the machine learning model is further configured to predict multiple features of another gesture. 
     
     
         27 . The method of  claim 26 , wherein the other gesture comprises a swipe down gesture, and wherein the multiple features of the other gesture comprise a start swipe down, a middle swipe down, an end swipe down, and the non-swipe. 
     
     
         28 . The method of  claim 27 , wherein the predicted gesture is based on a combination of multiple predictions from the machine learning model. 
     
     
         29 . The method of  claim 28 , wherein the multiple predictions include at least the middle swipe up or the middle swipe down, and wherein adjusting the function of the device comprises adjusting an audio output level of the device by a particular increment. 
     
     
         30 . The method of  claim 28 , wherein the multiple predictions include at least the start swipe up, the middle swipe up, and the end swipe up, and wherein adjusting the function of the device comprises adjusting an audio output level based on a distance determined using the start swipe up, the middle swipe up, and the end swipe up. 
     
     
         31 . The method of  claim 30 , wherein adjusting the audio output level based on the distance determined using the start swipe up, the middle swipe up, and the end swipe up comprises increasing or decreasing the audio output level in proportion to the distance. 
     
     
         32 . The method of  claim 21 , wherein the function of the device comprises an audio streaming function of the device. 
     
     
         33 . The method of  claim 32 , wherein the audio streaming function comprises streaming of audio corresponding to a phone call. 
     
     
         34 . A non-transitory computer readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to:
 provide sensor data as input to a machine learning model, the machine learning model having been trained to predict, based on the sensor data, multiple features of a gesture, and   obtain a predicted gesture based on an output from the machine learning model, the output generated in response to receiving the sensor data as input to the machine learning model, wherein the output comprises at least one of the multiple features of the gesture; and   adjust a function of a device based on the predicted gesture.   
     
     
         35 . The non-transitory computer readable medium of  claim 34 , wherein the gesture comprises a touch-based gesture on a surface of the device, and wherein the sensor data comprises sensor data from a non-touch sensor of the device. 
     
     
         36 . The non-transitory computer readable medium of  claim 34 , wherein the multiple features of the gesture comprise at least a start portion of the gesture, a middle portion of the gesture, and an end portion of the gesture. 
     
     
         37 . The non-transitory computer readable medium of  claim 36 , wherein the machine learning model has been further trained to predict a non-gesture and wherein the gesture comprises a swipe up gesture, wherein the non-gesture comprises a non-swipe, and wherein the multiple features of the gesture comprise at least: a start swipe up, a middle swipe up, and an end swipe up. 
     
     
         38 . The non-transitory computer readable medium of  claim 37 , wherein the machine learning model has been further trained to predict multiple features of another gesture different from the gesture. 
     
     
         39 . The non-transitory computer readable medium of  claim 38 , wherein the other gesture comprises a swipe down gesture, and wherein the multiple features of the other gesture comprise a start swipe down, a middle swipe down, and an end swipe down. 
     
     
         40 . An electronic device, comprising:
 a memory; and   one or more processors configured to:
 provide sensor data as input to a machine learning model, the machine learning model having been trained to predict, based on the sensor data, multiple features of a gesture, and 
 obtain a predicted gesture based on an output from the machine learning model, the output generated in response to receiving the sensor data as input to the machine learning model, wherein the output comprises at least one of the multiple features of the gesture; and 
 adjust a function of the electronic device based on the predicted gesture.