System and method for textual analysis of images

Abstract

Segmentation first breaks the images into segments or regions, with the segments of the region having text or symbols. The segmented image is separately applied to two different CNN-based models. Each model produces text boxes where potential text might exist. Then, a selective NMS algorithm is applied to the output of each model to produce a final group of text regions. These text regions are analyzed and actions taken.

What is claimed is: 
     
         1 . A system, comprising:
 a data storage unit including a trained first mathematical model and a trained second mathematical model, wherein the first mathematical model is different and distinct from the second mathematical model;   an electronic communication network;   an electronic server coupled to the electronic communication network that hosts a web-based catalog ordering system that receives electronic orders from customers;   a control circuit that is coupled to the electronic communication network and the data storage unit, wherein the control circuit is configured to:   receive an image of a product from a vendor via the electronic communication network, the product proposed by the vendor to be sold to retail customers;   perform segmentation on the image to divide the image into individual regions of homogeneous pixels, wherein the segmentation is effective to create a segmented image;   apply the segmented image to the first mathematical model to produce a first group of text regions and apply the segmented image to the second mathematical model to obtain a second group of text regions, wherein each of the text regions are regions includes potential text or symbols;   apply a selective non-maximal suppression (sNMS) algorithm to the first group of text regions and the second group of text regions to obtain a final group of text regions, the selective NMS algorithm being effective to remove overlapping regions at the same location or general location in the image, the selective NMS algorithm selecting text regions most likely to include text;   analyze informational content of the text regions and perform an action that utilizes the informational content of the text regions, the action being one or more of:   applying the informational content to the web-based ordering catalog, receiving a customer order from a customer as a result of the informational content, and physically fulfilling the received customer orders using an automated order fulfillment system to ship items in the order to the customer;   scanning the informational content for offensive content, and sending a message to a vendor via the electronic network to remove the offensive content or removing the item from a retail store or warehouse when an item including the offensive content exists in the retail store or warehouse.   
     
     
         2 . The system of  claim 1 , wherein the item that is removed from the retail store or warehouse is removed using an automated vehicle to navigate to the item and remove the item from a display unit or storage unit. 
     
     
         3 . The system of  claim 3 , wherein the automated vehicle is an automated ground vehicle or an aerial drone. 
     
     
         4 . The system of  claim 1 , wherein the first group of text regions, the second group of text regions, and the final group of text regions comprise text boxes. 
     
     
         5 . The system of  claim 1 , wherein the first mathematical model and the second mathematical model are convolutional neural networks (CNNs). 
     
     
         6 . The system of  claim 1 , wherein the first mathematical model and the second mathematical model are trained using training images. 
     
     
         7 . The system of  claim 1 , further comprising a camera, the camera coupled to the electronic communication network, the camera configured to obtain the image. 
     
     
         8 . A method, the method comprising:
 providing a data storage unit that includes a trained first mathematical model and a trained second mathematical model, wherein the first mathematical model is different and distinct from the second mathematical model;   providing an electronic communication network and an electronic server that is coupled to the electronic communication network, the server hosting a web-based catalog ordering system that receives electronic orders from customers;   providing a control circuit that is coupled to the electronic communication network and the data storage unit;   at the control circuit, receiving an image of a product from a vendor via the electronic communication network, the product proposed by the vendor to be sold to retail customers;   at the control circuit, performing segmentation on the image to divide the image into individual regions of homogeneous pixels, wherein the segmentation is effective to create a segmented image;   at the control circuit, applying the segmented image to the first mathematical model to produce a first group of text regions and apply the segmented image to the second mathematical model to obtain a second group of text regions, wherein each of the text regions are regions includes potential text or symbols;   at the control circuit, applying a selective non-maximal suppression (sNMS) algorithm to the first group of text regions and the second group of text regions to obtain a final group of text regions, the selective NMS algorithm being effective to remove overlapping regions at the same location or general location in the image, the selective NMS algorithm selecting text regions most likely to include text;   at the control circuit, analyzing informational content of the text regions and perform an action that utilizes the informational content of the text regions;   wherein the action being one or more of:   applying the informational content to the web-based ordering catalog, receiving a customer order from a customer as a result of the informational content, and physically fulfilling the received customer orders using an automated order fulfilment system to ship items in the order to the customer;   scanning the informational content for offensive content, and sending a message to a vendor via the electronic network to remove the offensive content or removing the item from a retail store or warehouse when an item including the offensive content exists in the retail store or warehouse.   
     
     
         9 . The method of  claim 8 , wherein the item that is removed from the retail store or warehouse is removed using an automated vehicle to navigate to the item and remove the item from a display unit or storage unit. 
     
     
         10 . The method of  claim 9 , wherein the automated vehicle is an automated ground vehicle or an aerial drone. 
     
     
         11 . The method of  claim 8 , wherein the first group of text regions, the second group of text regions, and the final group of text regions comprise text boxes. 
     
     
         12 . The method of  claim 8 , wherein the first mathematical model and the second mathematical model are convolutional neural networks (CNNs). 
     
     
         13 . The method of  claim 8 , wherein the first mathematical model and the second mathematical model are trained using training images. 
     
     
         14 . The method of  claim 8 , further comprising a camera, the camera coupled to the electronic communication network, the camera configured to obtain the image.