Extracting structured data from handwritten and audio notes

Abstract

This application is directed to recognizing unstructured information based on hints provided by structured information. A computer system obtains unstructured information collected from a handwritten or audio source, and identifies one or more terms from the unstructured information. The one or more terms includes a first term that is ambiguous. The computer system performs a recognition operation on the first term to derive a first plurality of candidate terms for the first term, and obtains first contextual information from an information template associated with the unstructured information. In accordance with the first contextual information, the computer system selects a first answer term from the first plurality of candidate terms, such that the first term is recognized as the first answer term.

1 . A computer-implemented method, comprising:
 at a computer system having one or more processors and memory storing one or more programs executed by the one or more processors:
 obtaining unstructured information collected from a handwritten or audio source; 
 determining (i) a first ambiguous term from the unstructured information and (ii) a first plurality of candidate terms for the first ambiguous term, wherein each candidate of the first plurality of candidate terms is a potential interpretation of the first ambiguous term; 
 identifying a form field name in an information template; and 
 identifying a first answer term from the first plurality of candidate terms based on the form field name. 
   
     
     
         2 . The method of  claim 1 , wherein the information template is associated with the unstructured information and stored in the computer system, and obtaining the form field name further comprises:
 obtaining contextual information from the information template, wherein the contextual information includes at least the form field name.   
     
     
         3 . The method of  claim 2 , wherein the contextual information includes a plurality of contextual options, wherein the first answer term at least partially matches one of the plurality of contextual options, and other candidate terms of the first plurality of candidate terms do not match any of the plurality of contextual options. 
     
     
         4 . The method of  claim 2 , wherein:
 the information template further comprises a plurality of data units, each data unit having a field name and a form field associated with the field name, the plurality of data units further including a first data unit;   the contextual information relates to the form field of the first data unit; and   the first answer term is associated with the form field of the first data unit.   
     
     
         5 . The method of  claim 4 , wherein the form field name corresponds to a plurality of predefined menu entries, and the first answer term at least partially matches one of the plurality of predefined menu entries. 
     
     
         6 . The method of  claim 1 , further comprising:
 presenting the first answer term in association with the form field name of the information template.   
     
     
         7 . The method of  claim 1 , further comprising:
 determining a second term to derive a second plurality of candidate terms, wherein the second term is adjacent to the first ambiguous term in the unstructured information;   obtaining contextual information from the information template;   selecting a second answer term from the second plurality of candidate terms based on the contextual information and at least the second term; and   presenting the second answer term in association with a corresponding portion of the information template.   
     
     
         8 . The method of  claim 7 , further comprising:
 updating the first answer term based on the second answer term.   
     
     
         9 . The method of  claim 8 , wherein updating the first answer term based on the second answer term includes expanding the first answer term to include the second answer term. 
     
     
         10 . The method of  claim 7 , further comprising:
 removing the first answer term based on the second answer term.   
     
     
         11 . A computer system, comprising:
 one or more processors; and   memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for:
 obtaining unstructured information collected from a handwritten or audio source; 
 determining (i) a first ambiguous term from the unstructured information and (ii) a first plurality of candidate terms for the first ambiguous term, wherein each candidate of the first plurality of candidate terms is a potential interpretation of the first ambiguous term; 
 identifying a form field name in an information template; and 
 identifying a first answer term from the first plurality of candidate terms based on the form field name. 
   
     
     
         12 . The computer system of  claim 11 , wherein the information template is associated with the unstructured information and stored in the computer system, and obtaining the form field name further comprises:
 obtaining contextual information from the information template, wherein the contextual information includes at least the form field name, wherein the contextual information includes a plurality of contextual options, wherein the first answer term at least partially matches one of the plurality of contextual options, and other candidate terms of the first plurality of candidate terms do not match any of the plurality of contextual options.   
     
     
         13 . The computer system of  claim 11 , wherein:
 obtaining the form field name further includes obtaining contextual information from the information template;   the information template further includes a plurality of data units, each data unit having a field name and a form field associated with the field name, the plurality of data units further including a first data unit;   the contextual information relates to the form field of the first data unit; and   the first answer term is associated with the form field of the first data unit.   
     
     
         14 . The computer system of  claim 11 , wherein the first plurality of candidate terms corresponds to different segmentations or recognitions of the first answer term. 
     
     
         15 . The computer system of  claim 11 , the one or more programs further comprising instructions for:
 building one or more attachment tokens and hints based on at least the first answer term; and   associating the one or more attachment tokens and hints with the information template.   
     
     
         16 . A non-transitory, computer readable storage medium storing one or more programs configured for execution by a computer system, the one or more programs comprising instructions for:
 obtaining unstructured information collected from a handwritten or audio source;   determining (i) a first ambiguous term from the unstructured information and (ii) a first plurality of candidate terms for the first ambiguous term, wherein each candidate of the first plurality of candidate terms is a potential interpretation of the first ambiguous term;   identifying a form field name in an information template; and   identifying a first answer term from the first plurality of candidate terms based on the form field name.   
     
     
         17 . The non-transitory, computer readable storage medium of  claim 16 , the one or more programs further comprising instructions for:
 selecting a third answer term from the first plurality of candidate terms based on the form field name of the information template; and   presenting the third answer term in association with the form field name of the information template;   determining whether the first answer term and the third answer term are conflicting; and   in accordance with a determination that the first answer term and the third answer term are conflicting, resolving a conflict between the first answer term and the third answer term.   
     
     
         18 . The non-transitory, computer readable storage medium of  claim 17 , wherein resolving the conflict includes:
 validating the first answer term and the third answer term based on the information template and/or an answer term of a term adjacent to the first answer term.   
     
     
         19 . The non-transitory, computer readable storage medium of  claim 17 , wherein resolving the conflict includes:
 obtaining second contextual information from a parallel source that is independent from the handwritten or audio source and the information template; and   validating the first answer term and the third answer term based on the second contextual information.   
     
     
         20 . The non-transitory, computer readable storage medium of  claim 17 , the one or more programs further comprising instructions for:
 determining whether the conflict was resolved;   in accordance with a determination that the conflict is not resolved, displaying a user interface to present the first and third answer terms; and   receiving a user selection of the first answer term, such that the first answer term is associated with the form field name of the information template based on the user selection.