Handwriting-based predictive population of partial virtual keyboards

Abstract

A “Stroke Untangler” composes handwritten messages from handwritten strokes representing overlapping letters or partial letter segments are drawn on a touchscreen device or touch-sensitive surface. These overlapping strokes are automatically untangled and then segmented and combined into one or more letters, words, or phrases. Advantageously, segmentation and composition is performed without requiring user gestures, timeouts, or other inputs to delimit characters within words, and without using handwriting recognition-based techniques to guide untangling and composing of the overlapping strokes to form characters. In other words, the user draws multiple overlapping strokes. Those strokes are then automatically segmented and combined into one or more corresponding characters. Text recognition of the resulting characters is then performed. Further, the segmentation and combination is performed in real-time, thereby enabling real-time rendering of the resulting characters in a user interface window. A related drawing mode enables entry of drawings in combination with the handwritten characters.

1 . A computer-implemented process for receiving text input, comprising:
 using a computer to perform process actions for:   receiving multiple overlapping strokes;   automatically segmenting and compositing the overlapping strokes to form one or more human-readable characters prior to performing handwriting-based recognition of any of the strokes or human-readable characters; and   after one or more human-readable characters have been formed, performing handwriting-based recognition of those human-readable characters to generate corresponding machine-readable text.   
     
     
         2 . The computer-implemented process of  claim 1  wherein the handwriting-based recognition is performed on a character-by-character basis. 
     
     
         3 . The computer-implemented process of  claim 1  wherein the handwriting-based recognition is performed on a full word basis. 
     
     
         4 . The computer-implemented process of  claim 1  wherein a user interface displays a scaled representation of the automatic segmentation and compositing of overlapping strokes in real-time as the strokes are being entered. 
     
     
         5 . The computer-implemented process of  claim 1  wherein a user interface displays a fading trail of each stroke in real-time as the stroke is being entered. 
     
     
         6 . The computer-implemented process of  claim 1  wherein a user interface displays one or more selectable recognition candidates corresponding to the machine-readable text. 
     
     
         7 . The computer-implemented process of  claim 6  wherein the recognition candidates include any combination of dictionary-based suggestions and search-based suggestions. 
     
     
         8 . The computer-implemented process of  claim 1  wherein a user interface includes a drawing mode that allows the entry of sketches formed from strokes that are not segmented and composited to form human-readable characters. 
     
     
         9 . The computer-implemented process of  claim 1  wherein one or more segmentation models used to automatically segment and composite the overlapping strokes are automatically re-trained following handwriting-based recognition of one or more human-readable characters using samples of recognized human-readable characters as positive examples for re-training the segmentation models. 
     
     
         10 . The computer-implemented process of  claim 1  wherein a user interface displays:
 a scaled representation of the automatic segmentation and compositing of overlapping strokes in real-time as the strokes are being entered; 
 one or more selectable recognition candidates corresponding to the machine-readable text; and 
 one or more selectable search suggestions based on a most likely one of the selectable recognition candidates. 
 
     
     
         11 . A system for receiving text input, comprising:
 a writing panel for receiving an input of multiple overlapping strokes;   a device for segmenting and compositing the overlapping strokes to form one or more human-readable characters prior to performing handwriting-based recognition of any of the strokes or human-readable characters; and   
       a device for performing handwriting-based recognition of the human-readable characters to generate corresponding machine-readable text. 
     
     
         12 . The system of  claim 11  wherein the writing panel is a touch-sensitive surface for tracking and receiving the overlapping stroke inputs. 
     
     
         13 . The system of  claim 11  wherein the writing panel is a virtual surface associated with a tracking device that tracks stroke motions relative to the virtual surface and provides those stroke motions as the overlapping stroke inputs. 
     
     
         14 . The system of  claim 11  wherein a user interface on a display device displays a scaled representation of the automatic segmentation and compositing of overlapping strokes in real-time as the strokes are being entered. 
     
     
         15 . The system of  claim 11  a user interface on a display device displays:
 a scaled representation of the automatic segmentation and compositing of overlapping strokes in real-time as the strokes are being entered; and 
 one or more selectable recognition candidates corresponding to the machine-readable text. 
 
     
     
         16 . The system of  claim 15  wherein the user interface displays one or more selectable search suggestions based on a most likely one of the selectable recognition candidates. 
     
     
         17 . A computer-readable medium having computer executable instructions stored therein for receiving a text input, said instructions causing a computing device to execute a method comprising:
 receiving a plurality of overlapping strokes on a touchscreen;   segmenting and compositing the overlapping strokes to form one or more human-readable characters prior to performing any handwriting-based recognition of any of the strokes or human-readable characters; and   
       performing handwriting-based recognition of the human-readable characters to generate corresponding machine-readable text. 
     
     
         18 . The computer-readable medium of  claim 17  wherein one or more segmentation models used to automatically segment and composite the overlapping strokes are automatically re-trained following handwriting-based recognition of one or more human-readable characters using samples of recognized human-readable characters as positive examples for re-training the segmentation models. 
     
     
         19 . The computer-readable medium of  claim 17  wherein a library of recognized human-readable characters constructed by storing one or more of those human-readable characters in combination with an identification of corresponding machine-readable text following the handwriting-based recognition of the human-readable characters. 
     
     
         20 . The computer-readable medium of  claim 17  wherein a user interface on a display device displays any combination of:
 a fading trail of each stroke in real-time as the stroke is being entered; 
 a scaled representation of the automatic segmentation and compositing of overlapping strokes in real-time as the strokes are being entered; 
 one or more selectable recognition candidates corresponding to the machine-readable text; and 
 one or more selectable search suggestions based on a most likely one of the selectable recognition candidates.