Multi-modal electronic document classification

Abstract

A method comprising operating at least one hardware processor for: receiving, as input, a plurality of electronic documents, training a machine learning classifier based, at least on part, on a training set comprising: (i) labels associated with the electronic documents, (ii) raw text from each of said plurality of electronic documents, and (iii) a rasterized version of each of said plurality of electronic documents, and applying said machine learning classifier to classify one or more new electronic documents.

1 . A method comprising:
 receiving, at a computer, an electronic document on which to train a machine learning classifier;   applying, by the computer, a first neural network to raw text extracted from the electronic document to determine a textual data representation of the electronic document;   applying, by the computer, a second neural network to a raster image extracted from the electronic document to determine a visual data representation of the electronic document;   generating, by the computer, a fusion representation based on the textual data representation and the visual data representation of the electronic document; and   applying, by the computer, the machine learning classifier based on the fusion representation to classify one or more new electronic documents.   
     
     
         2 . The method of  claim 1 , wherein the generating is further based on a label associated with the electronic document, the label denoting a document category. 
     
     
         3 . The method of  claim 1 , wherein the applying the first neural network further comprises:
 generating, by the computer, the textual data representation of said extracted text as a fixed length vector.   
     
     
         4 . The method of  claim 1 , wherein the generating further comprises:
 generating, by the computer, the fusion representation based on a correlation between the textual data representation and the visual data representation, the textual data representation, and the visual data representation.   
     
     
         5 . The method of  claim 1 , wherein the first neural network is different from the second neural network. 
     
     
         6 - 7 . (canceled) 
     
     
         8 . The method of  claim 1 , wherein the generating is based, at least in part, on a cost function which
 maximizes a correlation between the textual data representation and the visual data representation.   
     
     
         9 . The method of  claim 1 , wherein the applying the machine learning classifier further comprises:
 classifying, by the computer, the one or more new electronic documents with the machine learning classifier based on one of textual content and raster image of the one or more new electronic documents.   
     
     
         10 . A computing device comprising:
 a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of multi-modal electronic document classification;   a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:   receive, as input, an electronic document on which to train a machine learning classifier for the multi-modal electronic document classification;   apply a first neural network to raw text extracted from the electronic document to determine a textual data representation;   apply a second neural network to an image extracted from the electronic document to determine a visual data representation;   calculate a correlation between the textual data representation and the visual data representation;   generate a fusion representation based on the correlation, the textual data representation, and the visual data representation; and
 (iii) 
   apply the machine learning classifier based on the fusion representation to classify a new electronic document.   
     
     
         11 . The computing device of  claim 10 , wherein the generation of the fusion representation is further based on a label denoting a document category. 
     
     
         12 . The computing device of  claim 10 , wherein the processor is further configured to execute the machine executable code to cause the processor, as part of the application of the first neural network to:
 generate the textual data representation of the raw text as a fixed length vector.   
     
     
         13 . The computing device of  claim 10 , wherein the first neural network is different from the second neural network. 
     
     
         14 . The computing device of  claim 10 , wherein the first neural network is the same as the second neural network. 
     
     
         15 - 16 . (canceled) 
     
     
         17 . The computing device of  claim 10 , wherein the generation of the fusion representation is based, at least in part, on a cost function which
 maximizes the correlation between the textual data representation and the visual data representation.   
     
     
         18 . (canceled) 
     
     
         19 . A non-transitory machine readable medium having stored thereon instructions for performing a method comprising machine executable code which when executed by at least one machine, causes the machine to:
 extract raw text and an image from an electronic document on which to train a machine learning classifier;   apply a first neural network to the raw text to determine a textual data representation of the electronic document;   apply a second neural network to the image to determine a visual data representation of the electronic document;   generate a fusion representation based on the textual data representation, the visual data representation, and a correlation between the textual data representation and the visual data representation; and   apply the machine learning classifier based on the fusion representation to classify one or more new electronic documents.   
     
     
         20 . (canceled) 
     
     
         21 . The non-transitory machine readable medium of  claim 19 , further comprising machine executable code which when executed by the at least one machine causes the machine to:
 generate the textual data representation of the raw text as a fixed length vector.   
     
     
         22 . The non-transitory machine readable medium of  claim 19 , wherein the first neural network is different from the second neural network. 
     
     
         23 . (canceled) 
     
     
         24 . The non-transitory machine readable medium of  claim 19 , further comprising machine executable code which when executed by the at least one machine causes the machine to:
 calculate the correlation between textual data representation and visual data representation.   
     
     
         25 . (canceled) 
     
     
         26 . The non-transitory machine readable medium of  claim 19 , further comprising machine executable code which when executed by the at least one machine causes the machine to:
 generate the fusion representation based, at least in part, on a cost function which maximizes the correlation.   
     
     
         27 . The non-transitory machine readable medium of  claim 19 , further comprising machine executable code which when executed by the at least one machine causes the machine to:
 classify the one or more new electronic documents with the machine learning classifier based on one of textual content and raster image of the one or more new electronic documents.