Zero-shot object detection

Abstract

A method, apparatus and system for zero shot object detection includes, in a semantic embedding space having embedded object class labels, training the space by embedding extracted features of bounding boxes and object class labels of labeled bounding boxes of known object classes into the space, determining regions in an image having unknown object classes on which to perform object detection as proposed bounding boxes, extracting features of the proposed bounding boxes, projecting the extracted features of the proposed bounding boxes into the space, computing a similarity measure between the projected features of the proposed bounding boxes and the embedded, extracted features of the bounding boxes of the known object classes in the space, and predicting an object class label for proposed bounding boxes by determining a nearest embedded object class label to the projected features of the proposed bounding boxes in the space based on the similarity measures.

The invention claimed is: 
     
       1. A method of creating a semantic embedding space for improved zero-shot object detection, the method comprising:
 for each of a plurality of known object class labels, creating a respective, first word vector representative of each of the object class labels; 
 for each of a plurality of object bounding boxes, extracting features of an object bounding box; 
 for the extracted features of each of the object bounding boxes, creating a respective feature vector representative of the extracted features; 
 for each of the plurality of object bounding boxes, creating a respective, second word vector representative of an object class label associated with each of the plurality of object bounding boxes; and 
 semantically embedding the respective, first word vectors, the respective, second word vectors, and the respective feature vectors in a common semantic embedding space such that distances between the feature vectors and the word vectors embedded in the semantic embedding space follow semantic relationships between words representative of extracted features of object bounding boxes and words representative of object class labels; 
 wherein embedded vectors that are related are closer together in the semantic embedding space than unrelated vectors. 
 
     
     
       2. The method of  claim 1 , further comprising:
 determining a region in an image having an unknown object class on which to perform an object detection as a proposed object bounding box; 
 extracting features of the proposed object bounding box; 
 projecting the extracted features of the proposed object bounding box of the image into the semantic embedding space; 
 computing a similarity measure between the projected features of the proposed object bounding box and the embedded features of at least one of the object bounding boxes in the semantic embedding space; and 
 predicting an object class label to be associated with the features of the proposed object bounding box by determining a nearest embedded object class label to the projected features of the proposed object bounding box in the semantic embedding space based on the similarity measures computed for the projected features of the proposed object bounding box. 
 
     
     
       3. The method of  claim 2 , wherein the proposed object bounding box is determined using an arbitrary region proposal method. 
     
     
       4. The method of  claim 1 , wherein the object bounding boxes comprises at least one of a foreground object bounding box and a background object bounding box and the respective object class labels comprise at least one of a respective foreground object class label and a respective background object class labels. 
     
     
       5. The method of  claim 1 , wherein the features of an object bounding box are extracted using a neural network. 
     
     
       6. The method of  claim 1 , further comprising:
 determining a loss for an extracted feature of an object bounding box used to create the semantic embedding space. 
 
     
     
       7. The method of  claim 6 , further comprising:
 determining an additional reconstruction loss to be added to the loss determined for the extracted feature of the object bounding box used to create the semantic embedding space. 
 
     
     
       8. The method of  claim 1 , further comprising:
 determining a region of interest in an image as a proposed object bounding box. 
 
     
     
       9. The method of  claim 1 , wherein at least one of the plurality of object bounding boxes is at least one of a known object bounding box of a dataset used to create the semantic embedding space and a proposed object bounding box of an image used to create the semantic embedding space or an image on which to perform an object detection. 
     
     
       10. The method of  claim 1 , wherein the respective, first word vectors are determined using at least one of Glove and fastText. 
     
     
       11. An apparatus to create a semantic embedding space for improved zero-shot object detection, the apparatus comprising:
 a feature extraction module configured to, for each of a plurality of object bounding boxes, extract features of an object bounding box; and 
 a semantic space generator module configured to:
 for each of a plurality of known object class labels, create a respective, first word vector representative of each of the object class labels; 
 for each of the extracted features of each of the object bounding boxes, create a respective feature vector representative of the extracted features; 
 for each of the plurality of object bounding boxes, create a respective, second word vector representative of an object class label associated with each of the plurality of object bounding boxes; and 
 semantically embed the respective, first word vectors, the respective, second word vectors, and the respective feature vectors in a common semantic embedding space such that distances between the features vectors and the word vectors embedded in the semantic embedding space follow semantic relationships between words representative of extracted features of object bounding boxes and words representative of object class labels. 
 
 
     
     
       12. The apparatus of  claim 11 , further comprising a bounding box proposal module configured to:
 determine a region of interest in an image as a proposed object bounding box. 
 
     
     
       13. The apparatus of  claim 12 , wherein the proposed object bounding box is determined using an arbitrary region proposal process. 
     
     
       14. The apparatus of  claim 12 , wherein for performing zero-shot object detection:
 the bounding box proposal module is further configured to determine a region in an image having an unknown object class on which to perform an object detection as a proposed object bounding box; 
 the feature extraction module is further configured to extract features of the proposed object bounding box; and 
 the semantic space generator module is further configured to:
 project the extracted features of the proposed object bounding box of the image into the semantic embedding space; 
 compute a similarity measure between the projected features of the proposed object bounding box and the embedded features of at least one of the object bounding boxes in the semantic embedding space; and 
 predict an object class label to be associated with the features of the proposed object bounding box by determining a nearest embedded object class label to the projected features of the proposed object bounding box in the semantic embedding space based on the similarity measures computed for the projected features of the proposed object bounding box. 
 
 
     
     
       15. The apparatus of  claim 11 , wherein the object bounding boxes comprises at least one of foreground object bounding boxes and background object bounding boxes and the respective object class labels comprise at least one of respective foreground object class labels and respective background object class labels. 
     
     
       16. The apparatus of  claim 11 , wherein the feature extraction module is configured to extract the features of an object bounding box using a neural network. 
     
     
       17. The apparatus of  claim 11 , wherein the semantic space generator module is further configured to determine a loss for an extracted feature of an object bounding box used to create the semantic embedding space. 
     
     
       18. The apparatus of  claim 17 , wherein the semantic space generator module is further configured to determine an additional reconstruction loss to be added to the loss determined for the extracted feature of the object bounding box used to create the semantic embedding space. 
     
     
       19. The apparatus of  claim 11 , wherein at least one of the plurality of object bounding boxes is at least one of a known object bounding box of a dataset used to create the semantic embedding space and a proposed object bounding box of an image used to create the semantic embedding space or an image on which to perform an object detection. 
     
     
       20. The apparatus of  claim 11 , wherein the respective, first word vectors are determined using at least one of Glove and fastText.