Determining experiments represented by images in documents

Abstract

A method may include acquiring one or more image texts from an image of a document, segmenting the image into one or more sub-images using the one or more image texts, determining, by applying a machine learning model, one or more experimental techniques of one or more experiments for the one or more sub-images, and adding, to a knowledge base, one or more mappings of the one or more sub-images to the one or more experiments.

1 . A computer-implemented method, comprising:
 acquiring one or more image texts from an image of a document;   segmenting the image into one or more sub-images using the one or more image texts;   determining, by applying a machine learning model, one or more experimental techniques of one or more experiments for the one or more sub-images; and   adding, to a knowledge base, one or more mappings of the one or more sub-images to the one or more experiments.   
     
     
         2 . The computer-implemented method of  claim 1 , further comprising:
 determining one or more bounding boxes within the image for the one or more sub-images.   
     
     
         3 . The computer-implemented method of  claim 2 , further comprising:
 recognizing one or more sub-image labels in the one or more image texts;   obtaining one or more bounding boxes within the image for the one or more image texts; and   matching the one or more sub-images with the one or more sub-image labels using the one or more bounding boxes for the one or more sub-images and the one or more bounding boxes for the one or more sub-image labels.   
     
     
         4 . The computer-implemented method of  claim 3 , further comprising:
 determining macromolecules and experimental contexts for the one or more experiments using the one or more image texts, the one or more bounding boxes for the one or more sub-images, and the one or more bounding boxes for the one or more image texts.   
     
     
         5 . The computer-implemented method of  claim 3 , wherein recognizing the one or more sub-image labels in the one or more image texts comprises:
 identifying one or more sub-legend texts in legend text associated with the image; and   associating the one or more sub-image labels with the one or more sub-legend texts.   
     
     
         6 . The computer-implemented method of  claim 1 , further comprising:
 determining, using body text of the document, materials for the one or more experiments.   
     
     
         7 . The computer-implemented method of  claim 1 , further comprising:
 classifying, by the machine learning model, a sub-image of the one or more sub-images as a filter class, and   in response to classifying the sub-image as the filter class, determining that the sub-image has no recognized experimental technique.   
     
     
         8 . A system, comprising:
 a memory coupled to a computer processor;   a repository configured to store:
 a document comprising an image comprising one or more image texts, 
 a machine learning model, and 
 a knowledge base; and 
   an image analyzer, executing on the computer processor and using the memory, configured to:
 acquire the one or more image texts from the image, 
 segment the image into one or more sub-images using the one or more image texts, 
 determine, by applying the machine learning model, one or more experimental techniques of one or more experiments for the one or more sub-images, and 
 add, to the knowledge base, one or more mappings of the one or more sub-images to the one or more experiments. 
   
     
     
         9 . The system of  claim 8 , wherein the image analyzer is further configured to: determine one or more bounding boxes within the image for the one or more sub-images. 
     
     
         10 . The system of  claim 9 , wherein the image analyzer is further configured to:
 recognize one or more sub-image labels in the one or more image texts;   obtain one or more bounding boxes within the image for the one or more image texts; and   match the one or more sub-images with the one or more sub-image labels using the one or more bounding boxes for the one or more sub-images and the one or more bounding boxes for the one or more sub-image labels.   
     
     
         11 . The system of  claim 10 , wherein the image analyzer is further configured to:
 determine macromolecules and experimental contexts for the one or more experiments using the one or more image texts, the one or more bounding boxes for the one or more sub-images, and the one or more bounding boxes for the one or more image texts.   
     
     
         12 . The system of  claim 10 , wherein the image analyzer is further configured to recognize the one or more sub-image labels in the one or more image texts by:
 identifying one or more sub-legend texts in legend text associated with the image; and   associating the one or more sub-image labels with the one or more sub-legend texts.   
     
     
         13 . The system of  claim 8 , wherein the image analyzer is further configured to:
 determine, using body text of the document, materials for the one or more experiments.   
     
     
         14 . The system of  claim 8 , wherein the image analyzer is further configured to:
 classify, by the machine learning model, a sub-image of the one or more sub-images as a filter class, and   in response to classifying the sub-image as the filter class, determine that the sub-image has no recognized experimental technique.   
     
     
         15 . A non-transitory computer readable medium comprising instructions that, when executed by a computer processor, perform the method of any of  claims 1  to  7 .