System and method for format-agnostic document ingestion

Abstract

A system for format-agnostic document ingestion including a document ingestion server and a database is disclosed. The server is configured to receive an image of a document comprising text in an unknown format, convert the image, using OCR, into a plurality of text elements a content, a size, and an absolute position. The server is also configured to retrieve data detectors from the database, each associated with a data type anticipated to be in the document, and comprising at least one identifier and direction, and at least one validation criteria. The server is also configured to identify a potential descriptor by comparing the content of each text element with the at least one identifier, and then determine if the text element pointed to by the data detector meets the validation criteria. Finally, the server is configured to associate the validated text element with the data detector, and store the content.

1 - 20 . (canceled) 
     
     
         21 . A system for format-agnostic document ingestion, comprising:
 a document ingestion server comprising a processor and a memory, the document ingestion server communicatively coupled to a database, the processor configured to:
 receive an image of a document, the document comprising text arranged in an unknown format; 
 convert, using optical character recognition, the image of the document into a plurality of text elements, each text element comprising a content and an absolute position within the document; 
 retrieve a plurality of data detectors from the database, each data detector associated with a data type that is anticipated to be in the document, each data detector comprising at least one identifier that is one of a potential label and a potential format, and at least one validation criteria; 
 identify a potential descriptor by comparing the content of each text element with the at least one identifier of at least one data detector; 
 associate the text element meeting the validation criteria of the at least one data detector with the at least one data detector; 
 store, for each text element associated with one data detector of the plurality of data detector, the content of the text element, in the database; 
 identify a table within the document by calculating for each text element of the plurality of text elements a relative position of at least one neighboring text element relative to the text element using the absolute position of the text element, and comparing the relative positions of the plurality of text elements; 
 locate a header for the table by comparing the content of the text elements within the table with the identifiers of the plurality of data detectors and then identifying the data type of the matching text elements, the header being one of a row and a column; 
 validate, for each identified text element in the header, at least one text element within the other of a row and a column described by the identified text element in the header with the validation criteria of the data detector that identified the identified text element in the header; 
 associate, for each identified text element in the header, at least one validated text element within the other of the row and the column described by the identified text element in the header with the data detector that identified the identified text element in the header; and 
 identify a potential descriptor by comparing the content of each text element not part of the table with the at least one identifier of at least one data detector. 
   
     
     
         22 . The system of  claim 21 , wherein each text element further comprises a size, and wherein the potential format of each data detector further comprises a potential size. 
     
     
         23 . The system of  claim 21 , wherein each validation criteria describes at least one of a valid format and a valid range. 
     
     
         24 . The system of  claim 21 :
 wherein each data detector further comprises at least one direction describing a potential relative direction of a text element having a label associated with the data detector; and   wherein the processor is further configured to:
 determine a source of the document by comparing the at least one identifier of a data detector associated with a data type that is unique among potential document sources with the content of each text element of the plurality of text elements; and 
 for each data detector, order at least one of the identifiers and the directions according to a history stored in the database and associated with the source; and 
 update, for each data detector, the history associated with the source, according to which identifier of the at least one identifier and which direction of the at least one direction matched the most text elements of the data type described by the data detector in the document. 
   
     
     
         25 . The system of  claim 24 , wherein determining the source of the document comprises identifying all postal addresses in the document based upon an observed format, validating each postal address, placing each postal address in a standard format, and comparing each address with a list of addresses unique to each of a plurality of known document sources. 
     
     
         26 . The system of  claim 21 , wherein the processor is further configured to:
 train a machine learning model correlating text elements with the data detectors they have been associated with;   determine whether the machine learning model performs better than one or more data detectors; and   automatically employ the machine learning model in place of the one or more data detectors once the machine leaming model outperforms the one or more data detectors.   
     
     
         27 . A system for format-agnostic document ingestion, comprising:
 a document ingestion server comprising a processor and a memory, the document ingestion server communicatively coupled to a database, the processor configured to:
 receive an image of a document, the document comprising text arranged in an unknown format; 
 convert, using optical character recognition, the image of the document into a plurality of text elements, each text element comprising a content and an absolute position within the document; 
 retrieve a plurality of data detectors from the database, each data detector associated with a data type that is anticipated to be in the document, each data detector comprising at least one identifier that is a potential label, at least one direction describing a potential relative direction of a text element having a label associated with the data detector, and at least one validation criteria; 
 identify a potential descriptor by comparing the content of each text element with the at least one identifier of at least one data detector; 
 determine if the text element pointed to by one of the at least one direction of the data detector used to identify the potential descriptor meets the validation criteria of the data detector; 
 associate the validated text element with the data detector; and 
 store, for each text element associated with one data detector of the plurality of data detector, the content of the text element, in the database. 
   
     
     
         28 . The system of  claim 27 , wherein the processor is further configured to:
 identify a table within the document by calculating for each text element of the plurality of text elements a relative position of at least one neighboring text element relative to the text element using the absolute position of the text element, and comparing the relative positions of the plurality of text elements;   locate a header for the table by comparing the content of the text elements within the table with the identifiers of the plurality of data detectors and then identifying the data type of the matching text elements, the header being one of a row and a column;   validate, for each identified text element in the header, at least one text element within the other of a row and a column described by the identified text element in the header with the validation criteria of the data detector that identified the identified text element in the header; and   associate, for each identified text element in the header, at least one validated text element within the other of the row and the column described by the identified text element in the header with the data detector that identified the identified text element in the header.   
     
     
         29 . The system of  claim 27 :
 wherein the processor is further configured to identify a document type by searching the content of each text element for a plurality of distinguishing strings, each distinguishing string being unique to one document type; and   wherein the plurality of data detectors retrieved from the database is selected based on the document type.   
     
     
         30 . The system of  claim 27 , wherein each identifier is at least one of a potential label and a potential format, and wherein each validation criteria describes a valid format. 
     
     
         31 . The system of  claim 30 , wherein each text element further comprises a size, and wherein the potential format of each data detector further comprises a potential size. 
     
     
         32 . The system of  claim 27 , wherein each validation criteria describes at least one of a valid format and a valid range. 
     
     
         33 . A method for format-agnostic document ingestion, comprising:
 receiving, by a processor, an image of a document, the document comprising text arranged in an unknown format;   converting, using optical character recognition performed by the processor, the image of the document into a plurality of text elements, each text element comprising a content and an absolute position within the document;   retrieving a plurality of data detectors, each data detector associated with a data type that is anticipated to be in the document, each data detector comprising at least one identifier that is a potential label, at least one direction describing a potential relative direction of a text element having a label associated with the data detector, and at least one validation criteria;   identifying a potential descriptor by comparing the content of each text element with the at least one identifier of at least one data detector;   determining if the text element pointed to by one of the at least one direction of the data detector used to identify the potential descriptor meets the validation criteria of the data detector;   associating the validated text element with the data detector; and   storing, for each text element associated with one data detector of the plurality of data detector, the content of the text element.   
     
     
         34 . The method of  claim 33 , further comprising:
 identifying a document type by searching the content of each text element for a plurality of distinguishing strings, each distinguishing string being unique to one document type;   wherein the plurality of data detectors retrieved is selected based on the document type.   
     
     
         35 . The method of  claim 33 , wherein each identifier is one of a potential label and a potential format, and wherein each validation criteria describes a valid format. 
     
     
         36 . The method of  claim 35 , wherein each text element further comprises a size, and wherein the potential format of each data detector further comprises a potential size. 
     
     
         37 . The method of  claim 33 , wherein each validation criteria describes one of a valid format and a valid range. 
     
     
         38 . The method of  claim 33 , further comprising:
 determining a source of the document by comparing the at least one identifier of a data detector associated with a data type that is unique among potential document sources with the content of each text element of the plurality of text elements; and   ordering, for each data detector, at least one of the identifiers and the directions according to a history associated with the source; and   updating, for each data detector, the history associated with the source, according to which identifier of the at least one identifier and which direction of the at least one direction matched the most text elements of the data type described by the data detector in the document.   
     
     
         39 . The method of  claim 38 , wherein determining the source of the document further comprises identifying all postal addresses in the document based upon an observed format, validating each postal address, placing each postal address in a standard format, and comparing each address with a list of addresses unique to each of a plurality of known document sources. 
     
     
         40 . The method of  claim 33 , further comprising:
 training a machine leaming model correlating text elements with the data detectors they have been associated with;   determining whether the machine leaming model performs better than one or more data detectors; and   automatically employ the machine learning model in place of the one or more data detectors once the machine leaming model outperforms the one or more data detectors.