Expanding physical motion gesture lexicon for an automated assistant

Abstract

Implementations provided herein relate to correlating available input gestures to recently created application functions, and adapting available input gestures, and/or user-created input gestures, to be correlated with existing application functions. Available input gestures (e.g., a hand wave) can be those that can be readily performed upon setup of a computing device. When a user installs an application that is not initially configured to handle the available input gestures, the available input gestures can be correlated to certain functions of the application. Furthermore, a user can create new gestures for application actions and/or modify existing gestures according to their own preferences and/or physical capabilities. When multiple users elect to modify an existing gesture in the same way, the modification can be made universal, with permission from the users, in order to eliminate latency when subsequently adapting to preferences of other users.

1 . A method implemented by one or more processors, the method comprising:
 determining, based on data generated in response to a gesture input to a computing device, that a user has performed a particular gesture that the user has not previously provided as an input to the computing device;   determining, based on the data, that a separate gesture input was previously provided a separate computing device by a different user, and that the separate gesture input has one or more characteristics in common with the gesture input provided by the user to the computing device,
 wherein the separate gesture input was previously provided to the separate computing device to control an application function of the separate computing device; 
   causing, in response to determining that the separate gesture input has one or more characteristics in common with the gesture input, an interface of the computing device or another computing device to provide a request to the user to confirm whether the user intended the gesture input to imitate the separate gesture input;   in response receiving a user input that indicates the user intended the gesture input to imitate the separate gesture input responsive to the request:
 determining that another application function is controllable via the computing device and that the other application function includes a common feature to the application function, and 
 causing the application function to be controllable via an instance of the gesture input. 
   
     
     
         2 . The method of  claim 1 , further comprising:
 in response receiving a user input that indicates the user did not intend the gesture input to imitate the separate gesture input responsive to the request:
 identifying one or more applications functions that are controllable via the computing device, and 
 prompting the user to select a particular application function of the one or more application functions to be controllable via the gesture input. 
   
     
     
         3 . The method of  claim 2 , further comprising:
 in response receiving a user input that indicates the user did not intend the gesture input to imitate the separate gesture input responsive to the request:
 determining, that the user has selected the particular application function of the one or more application functions to be controllable via the gesture input, and 
 causing the interface of the computing device or the other computing device to provide another request for the user to perform another instance of the gesture input. 
   
     
     
         4 . The method of  claim 1 , wherein the common feature is a common parameter, and causing the application function to be controllable via the instance of the gesture input includes causing a parameter value to be assigned to the common parameter when the user subsequently performs the instance of the gesture input. 
     
     
         5 . The method of  claim 1 , wherein determining that the another application function is controllable via the computing device comprises:
 identifying an application, from multiple different applications, that the user has previously controlled without providing a physical motion gesture to a camera interface of the computing device, and   determining one or more different application functions that are controllable via the application, wherein the one or more different application functions include the application function.   
     
     
         6 . A method implemented by one or more processors, the method comprising:
 determining, based on processing image data captured via a camera interface of a computing device from which one or more applications are accessible, that a user has performed a physical motion gesture one or more times, wherein the one or more applications are responsive to other physical motion gestures and the one or more applications are currently unresponsive to the physical motion gesture;   determining that the user has caused an action to be performed, via an application of the one or more applications, and within a period of time of the user performing the physical motion gesture one or more times;   causing, in response to determining that the user caused the action to be performed via the application, an interface of the computing device or another computing device to provide a request to the user to confirm whether the user intended the physical motion gesture to control the action performed via the application;   in response receiving a user input that indicates the user did not intended the physical motion gesture to control the action performed by the application:
 generating control data that indicates the action performed by the application is controllable via an instance of the physical motion gesture. 
   
     
     
         7 . The method of  claim 6 , wherein determining that the user has performed the physical motion gesture one or more times includes:
 determining that the image data, and/or gesture data characterizing the performance of the physical motion gesture by the user, is associated with other data that corresponds to a particular physical motion gesture of one or more other physical motion gestures.   
     
     
         8 . The method of  claim 7 , wherein the particular physical motion gesture is associated with another user that configured another computing device and/or another application to be responsive to the particular physical motion gesture. 
     
     
         9 . The method of  claim 8 , wherein the other data, which corresponds to the particular physical motion gesture, is generated based on each user, of multiple different users, performing an instance of the particular physical motion gesture. 
     
     
         10 . The method of  claim 6 , wherein the period of time is based on an action type corresponding to the action performed via the application and/or is based on an application type corresponding to the application. 
     
     
         11 . The method of  claim 6 , wherein the request provided to the user to confirm whether the user intended the physical motion gesture to control the action performed via the application includes providing a request for the user to perform another instance of the physical motion gesture in order to indicate confirmation. 
     
     
         12 . The method of  claim 6 , further comprising:
 identifying another function of the application that is not currently responsive to another physical motion gesture captured via the camera interface of the computing device; and   when the user confirms they intended the physical motion gesture to control the action performed by the application:
 causing the interface of the computing device to provide another request for permission from the user to allow the other function to be controllable via the other physical motion gesture. 
   
     
     
         13 . A system comprising:
 at least one processor; and   memory storing instructions that, when executed, cause the at least one processor to perform operations, the operations comprising:
 determining, based on data generated in response to a gesture input to a computing device, that a user has performed a particular gesture that the user has not previously provided as an input to the computing device; 
 determining, based on the data, that a separate gesture input was previously provided a separate computing device by a different user, and that the separate gesture input has one or more characteristics in common with the gesture input provided by the user to the computing device,
 wherein the separate gesture input was previously provided to the separate computing device to control an application function of the separate computing device; 
 
 causing, in response to determining that the separate gesture input has one or more characteristics in common with the gesture input, an interface of the computing device or another computing device to provide a request to the user to confirm whether the user intended the gesture input to imitate the separate gesture input; 
 in response receiving a user input that indicates the user intended the gesture input to imitate the separate gesture input responsive to the request:
 determining that another application function is controllable via the computing device and that the other application function includes a common feature to the application function, and 
 causing the application function to be controllable via an instance of the gesture input. 
 
   
     
     
         14 . The system of  claim 13 , the operations further comprising:
 in response receiving a user input that indicates the user did not intend the gesture input to imitate the separate gesture input responsive to the request:
 identifying one or more applications functions that are controllable via the computing device, and 
 prompting the user to select a particular application function of the one or more application functions to be controllable via the gesture input. 
   
     
     
         15 . The system of  claim 14 , the operations further comprising:
 in response receiving a user input that indicates the user did not intend the gesture input to imitate the separate gesture input responsive to the request:
 determining, that the user has selected the particular application function of the one or more application functions to be controllable via the gesture input, and 
 causing the interface of the computing device or the other computing device to provide another request for the user to perform another instance of the gesture input. 
   
     
     
         16 . The system of  claim 13 , wherein the common feature is a common parameter, and causing the application function to be controllable via the instance of the gesture input includes causing a parameter value to be assigned to the common parameter when the user subsequently performs the instance of the gesture input. 
     
     
         17 . The system of  claim 13 , wherein determining that the another application function is controllable via the computing device comprises:
 identifying an application, from multiple different applications, that the user has previously controlled without providing a physical motion gesture to a camera interface of the computing device, and   determining one or more different application functions that are controllable via the application, wherein the one or more different application functions include the application function.