Sequential two-handed touch typing on a mobile device

Abstract

Examples of the present disclosure describe systems and methods of providing sequential two-handed touch typing. In aspects, a client device may provide a touch-based input receiving application. The client device may include one or more sensors operable to detect the approach of a user's hand or touch-based tool. In response to a detection, the client device may determine input interaction information, such as aspects of the hand (e.g., right or left) and/or finger(s) being used. The input interaction information may be used to detect candidate keys and to provide a keyboard (or a portion of a keyboard) based thereon. In aspects, the provided keyboard or keyboard portion may be altered in response to a subsequent detection.

1 - 20 . (canceled) 
     
     
         21 . A system for providing sequential two-handed typing comprising:
 at least one processor; and   memory coupled to the at least one processor, the memory comprising computer executable instructions that, when executed by the at least one processor, cause the processor to:
 generate an image of an object proximate to a touchscreen of a mobile device, the object comprising at least a portion of a hand, without receiving direct contact with the touchscreen of the mobile device; 
 determine, by the mobile device, input interaction information by comparing the image to a set of labeled images corresponding to input actions, wherein the comparing is performed using a machine learning model for image recognition; 
 when a match is found between the image and a first input action in the set of labeled images, correlate the image to the first input action; and 
 present output associated with the first input action via the touchscreen of the mobile device. 
   
     
     
         22 . The system of  claim 21 , wherein the first object is associated with a typing action. 
     
     
         23 . The system of  claim 21 , further comprising collecting sensor data from one or more sensors to generate the image of the object proximate to the touchscreen, wherein the one or more sensors comprise at least one of: a proximity sensor, an accelerometer sensor, a gyroscopic sensor, a force sensor, an acoustic sensor, an optical sensor, and a localization sensor. 
     
     
         24 . The system of  claim 21 , further comprising computer executable instructions that, when executed by the at least one processor, cause the processor to:
 train a model using at least the input interaction information.   
     
     
         25 . The system of  claim 21 , wherein the machine learning model for image recognition is operable to correlate the first object to one or more user intents. 
     
     
         26 . The system of  claim 21 , further comprising computer executable instructions that, when executed by the at least one processor, cause the processor to:
 receive user input via the touchscreen;   provide the user input as input to the machine learning model for image recognition; and   receive output from the machine learning model for image recognition.   
     
     
         27 . The system of  claim 26 , wherein the user input is incomplete, and wherein the machine learning model for image recognition is operable to use the incomplete user input to determine at least one of a character, a complete word, and a sentence. 
     
     
         28 . The system of  claim 21 , wherein the input interaction information corresponds to at least one of physical characteristics of the first object and motion data for the first object. 
     
     
         29 . The system of  claim 21 , wherein determining the input interaction information includes using at least one of: pattern-matching techniques, image recognition, sound wave analysis, a rule set, fuzzy logic, machine-learned models and/or one or more weighting algorithms. 
     
     
         30 . The system of  claim 21 , wherein the input interaction information comprises a label, the label describing a physical characteristic of the first object. 
     
     
         31 . The system of  claim 21 , further comprising computer executable instructions that, when executed by the at least one processor, cause the processor to:
 determine input options using the input interaction information, wherein the input options include one or more portions of a keyboard.   
     
     
         32 . The system of  claim 31 , wherein a left side of the keyboard is displayed when the input interaction information indicates the first object is approaching from the left of an interface, and a right side of the keyboard is displayed when the input interaction information indicates the second object is approaching from the right of the interface. 
     
     
         33 . A method for providing sequential two-handed typing, the method comprising:
 generating an image of an object proximate to a touchscreen of a mobile device, the object comprising at least a portion of a hand, without receiving direct contact with the touchscreen of the mobile device;   determining, by the mobile device, input interaction information by comparing the image of the object to a set of labeled images corresponding to input actions, wherein the comparing is performed using a machine learning model for image recognition;   when a match is found between the image and a first input action in the set of labeled images, correlating the image to the first input action; and   presenting output associated with the first input action via the touchscreen of the mobile device.   
     
     
         34 . The method of  claim 33 , further comprising collecting one or more sound waves to determine that the object is approaching the touchscreen of the mobile device. 
     
     
         35 . The method of  claim 33 , wherein the input interaction information comprises a context, the context describing one or more action categories. 
     
     
         36 . The method of  claim 33 , further comprising:
 determining an input option using the input interaction information, wherein the input option includes one or more portions of an onscreen keyboard.   
     
     
         37 . The method of  claim 36 , wherein the onscreen keyboard is presented as an overlay on an interface, and wherein the overlay swoops in from a side of the interface. 
     
     
         38 . The method of  claim 33 , wherein training the model comprises providing sensor data to the machine learning model for image recognition to correlate the sensor data to at least input interaction information. 
     
     
         39 . The method of  claim 33 , further comprising:
 receiving user input via the touchscreen;   providing the user input as input to the machine learning model for image recognition; and   receiving output from the model, wherein the output correlates the user input to one or more user intents.   
     
     
         40 . A computer storage media storing computer executable instructions that when executed cause a computing system to perform a method providing sequential two-handed touch typing, the method comprising:
 generating an image of an object proximate to a touchscreen of a mobile device without receiving direct contact with the touchscreen of the mobile device;   determining, by the mobile device, input interaction information by comparing the image to a set of labeled images corresponding to input actions, wherein the comparing is performed using a machine learning model for image recognition;   when a match is found between the image and a first input action in the set of labeled images, correlating the image to the first input action; and   presenting output via the touchscreen of the mobile device.