Extracting attributes from arbitrary digital images utilizing a multi-attribute contrastive classification neural network

Abstract

This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that extract multiple attributes from an object portrayed in a digital image utilizing a multi-attribute contrastive classification neural network. For example, the disclosed systems utilize a multi-attribute contrastive classification neural network that includes an embedding neural network, a localizer neural network, a multi-attention neural network, and a classifier neural network. In some cases, the disclosed systems train the multi-attribute contrastive classification neural network utilizing a multi-attribute, supervised-contrastive loss. In some embodiments, the disclosed systems generate negative attribute training labels for labeled digital images utilizing positive attribute labels that correspond to the labeled digital images.

What is claimed is: 
     
         1 . A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to perform operations comprising:
 generating high-level attribute feature map from a digital image portraying an object utilizing a high-level embedding layer of an embedding neural network;   generating a low-level attribute feature map from the digital image utilizing a low-level embedding layer of the embedding neural network; and   determining a set of attributes for the object portrayed within the digital image from a combination of the high-level attribute feature map and the low-level attribute feature map utilizing a classifier neural network.   
     
     
         2 . The non-transitory computer-readable medium of  claim 1 , wherein the operations further comprise:
 determining the set of attributes for the object portrayed within the digital image by utilizing the classifier neural network with:   a localized image-object feature vector based on the high-level attribute feature map; and   a localized low-level attribute feature vector based on the low-level attribute feature map.   
     
     
         3 . The non-transitory computer-readable medium of  claim 2 , wherein the operations further comprise:
 generating an image-object feature map by combining the high-level attribute feature map and an object-label embedding vector generated from a label corresponding to the object; and   generating a localized object attention feature vector from the image-object feature map reflecting a segmentation prediction of the object portrayed in the digital image utilizing a localizer neural network.   
     
     
         4 . The non-transitory computer-readable medium of  claim 3 , wherein the operations further comprise:
 generating the localized image-object feature vector by combining the localized object attention feature vector with the image-object feature map; and   generating a localized low-level attribute feature vector by combining the low-level attribute feature map with the localized object attention feature vector.   
     
     
         5 . The non-transitory computer-readable medium of  claim 3 , wherein the operations further comprise:
 generating a multi-attention feature vector from the image-object feature map using a plurality of attention layers of a multi-attention neural network; and   determining the set of attributes for the object portrayed within the digital image by utilizing the classifier neural network with the multi-attention feature vector.   
     
     
         6 . A system comprising:
 one or more memory devices comprising:
 a digital image portraying an object; and 
 a multi-attribute contrastive classification neural network comprising: an embedding neural network, a localizer neural network, a multi-attention neural network, and a classifier neural network; and 
   one or more processors configured to cause the system to:   determine an image-object feature map from the digital image utilizing the embedding neural network;   generate a multi-attention feature vector from the image-object feature map using a plurality of attention layers of the multi-attention neural network;   generate a localized object attention feature vector from the image-object feature map reflecting a segmentation prediction of the object portrayed in the digital image utilizing the localizer neural network;   generate a localized image-object feature vector by combining the image-object feature map and the localized object attention feature vector; and   determine a plurality of attributes for the object portrayed within the digital image from the localized image-object feature vector and the multi-attention feature vector utilizing a classifier neural network.   
     
     
         7 . The system of  claim 6 , wherein the one or more processors are configured to cause the system to:
 generate a low-level attribute feature map for the portrayed object utilizing a low-level embedding layer of the embedding neural network;   generate a localized low-level attribute feature vector by combining the low-level attribute feature map with the localized object attention feature vector; and   determine the plurality of attributes for the object portrayed within the digital image utilizing a classifier neural network from a concatenation of the localized image-object feature vector, the multi-attention feature vector, and the localized low-level attribute feature vector.   
     
     
         8 . The system of  claim 6 , wherein the one or more processors are configured to cause the system to:
 generate a high-level attribute feature map for the portrayed object utilizing a high-level embedding layer of the embedding neural network;   generate an object-label embedding vector from a label corresponding to the object; and   generate the image-object feature map by combining the high-level attribute feature map and the object-label embedding vector.   
     
     
         9 . The system of  claim 6 , wherein the one or more processors are configured to cause the system to:
 detect the object from multiple objects depicted within the digital image utilizing an object detector model; and   determine the plurality of attributes for the detected object portrayed within the digital image.   
     
     
         10 . The system of  claim 6 , wherein the one or more processors are configured to cause the system to generate a negative attribute label within a set of labeled images by determining exclusive relationships or overlapping relationships between the negative attribute label and a positive attribute label corresponding to a labeled image within the set of labeled images. 
     
     
         11 . A computer-implemented method comprising:
 generating a high-level attribute feature map from a digital image portraying an object utilizing a high-level embedding layer of an embedding neural network;   generating a low-level attribute feature map from the digital image utilizing a low-level embedding layer of the embedding neural network; and   determining a set of attributes for the object portrayed within the digital image from a combination of the high-level attribute feature map and the low-level attribute feature map utilizing a classifier neural network.   
     
     
         12 . The computer-implemented method of  claim 11 , further comprising determining the set of attributes for the object portrayed within the digital image by utilizing the classifier neural network with:
 a localized image-object feature vector based on the high-level attribute feature map; and   a localized low-level attribute feature vector based on the low-level attribute feature map.   
     
     
         13 . The computer-implemented method of  claim 12 , further comprising:
 generating an image-object feature map by combining the high-level attribute feature map and an object-label embedding vector generated from a label corresponding to the object; and   generating a localized object attention feature vector from the image-object feature map reflecting a segmentation prediction of the object portrayed in the digital image utilizing a localizer neural network.   
     
     
         14 . The computer-implemented method of  claim 13 , further comprising:
 generating the localized image-object feature vector by combining the localized object attention feature vector with the image-object feature map; and   generating a localized low-level attribute feature vector by combining the low-level attribute feature map with the localized object attention feature vector.   
     
     
         15 . The computer-implemented method of  claim 13 , further comprising:
 generating a multi-attention feature vector from the image-object feature map using a plurality of attention layers of a multi-attention neural network; and   determining the set of attributes for the object portrayed within the digital image by utilizing the classifier neural network with the multi-attention feature vector.   
     
     
         16 . The computer-implemented method of  claim 11 , wherein the embedding neural network comprises a convolutional neural network and the low-level embedding layer proceeds the high-level embedding layer in the convolutional neural network. 
     
     
         17 . The computer-implemented method of  claim 11 , wherein the high-level attribute feature map and the low-level attribute feature map encode attributes that span across a wide range of semantic levels. 
     
     
         18 . The computer-implemented method of  claim 11 , further comprising associating tags with the digital image, wherein the tags indicate the attributes of the set of attributes. 
     
     
         19 . The computer-implemented method of  claim 18 , further comprising:
 receiving a search query for an image search via a graphical user interface, the search query comprising one or more object attributes;   determining that the digital image includes the one or more object attributes based on the tags associated with the digital image; and   returning the digital image as a result to the image search via the graphical user interface.   
     
     
         20 . The computer-implemented method of  19 , further comprising returning a plurality of digital image depicting the one or more object attributes together with the digital image in response to the image search.