Joint-based item recognition

Abstract

For an input image of a person, a set of object proposals are generated in the form of bounding boxes. A pose detector identifies coordinates in the image corresponding to locations on the person's body, such as the waist, head, hands, and feet of the person. A convolutional neural network receives the portions of the input image defined by the bounding boxes and generates a feature vector for each image portion. The feature vectors are input to one or more support vector machine classifiers, which generate an output representing a probability of a match with an item. The distance between the bounding box and a joint associated with the item is used to modify the probability. The modified probabilities for the support vector machine are then compared with a threshold and each other to identify the item.

What is claimed is: 
     
         1 . A system comprising:
 a memory having instructions embodied thereon; and   one or more processors configured by the instructions to perform operations comprising:
 analyzing an image of a person to determine a set of joints of the person; 
 generating a set of candidate bounding boxes, each bounding box of the set of candidate bounding boxes defining a portion of the image; and 
 analyzing the portion of the image defined by a bounding box of the set of candidate bounding boxes to identify an item of apparel, the identification being based on a distance between the bounding box and a joint of the set of joints. 
   
     
     
         2 . The system of  claim 1 , wherein the operations further comprise:
 receiving the image of the person;   selecting an advertisement based on the item of apparel; and   causing the display of the image of the person and the advertisement.   
     
     
         3 . The system of  claim 2 , wherein:
 the image of the person is owned by a first user account; and   the display of the image of the person and the advertisement is to a second user account.   
     
     
         4 . The system of  claim 2 , wherein:
 the causing of the display of the image of the person and the advertisement comprises:
 causing the display of the person with an annotation superimposed over the item of apparel; and 
 in response to a user interaction with the annotation, causing the display of the advertisement. 
   
     
     
         5 . The system of  claim 1 , wherein:
 the identification based on the distance between the bounding box and the joint of the set of joints is based on a distance between a center of the bounding box and the joint of the set of joints.   
     
     
         6 . The system of  claim 1 , wherein:
 the analyzing of the portion of the image to identify the item of apparel comprises:
 resizing the portion of the image to a predetermined size; 
 providing the resized image to a plurality of support vector machine (SVM) classifiers as input, each SVM corresponding to a particular item of apparel; and 
 receiving output from each SVM classifier, the outputs indicative of a probability that the resized image contains the particular item of apparel corresponding to the SVM classifier; and 
   the identification of the item is further based on the outputs provided by the SVM classifiers.   
     
     
         7 . The system of  claim 1 , wherein:
 the identification of the item is further based on a ratio of height and width of the portion of the image.   
     
     
         8 . The system of  claim 1 , wherein:
 the identification of the item is further based on a logarithm of a ratio of height and width of the portion of the image.   
     
     
         9 . The system of  claim 1 , wherein:
 the analyzing of the image of the person to determine the set of joints of the person comprises identifying a pose of the person; and   the identification of the item is further based on the identified pose of the person.   
     
     
         10 . The system of  claim 1 , wherein:
 the identification of the item is further based on a distance between the bounding box and a second joint of the set of joints.   
     
     
         11 . The system of  claim 1 , wherein the operations further comprise:
 receiving the image of the person from a client device;   identifying a set of images based on the identified item; and   causing the set of images to be displayed on the client device.   
     
     
         12 . The system of  claim 11 , wherein:
 the identifying of the set of images based on the identified item identifies images containing the identified item.   
     
     
         13 . The system of  claim 11 , wherein:
 the identifying of the set of images based on the identified item identifies images of items for sale in an online marketplace.   
     
     
         14 . A method comprising:
 analyzing an image of a person to determine a pose of the person;   generating, by a hardware processor, a set of candidate bounding boxes, each bounding box of the set of candidate bounding boxes defining a portion of the image; and   analyzing the portion of the image defined by a bounding box of the set of candidate bounding boxes to identify an item of apparel, the identification being based on a distance between the bounding box and a joint of the set of joints.   
     
     
         15 . The method of  claim 14 , further comprising:
 receiving the image of the person;   selecting an advertisement based on the item of apparel; and   causing the display of the image of the person and the advertisement.   
     
     
         16 . The method of  claim 15 , wherein:
 the image of the person is owned by a first user account; and   the display of the image of the person and the advertisement is to a second user account.   
     
     
         17 . The method of  claim 14 , wherein:
 the identification based on the distance between the bounding box and the joint of the set of joints is based on a distance between a center of the bounding box and the joint of the set of joints.   
     
     
         18 . The method of  claim 14 , wherein:
 the analyzing of the portion of the image to identify the item of apparel comprises:
 resizing the portion of the image to a predetermined size; 
 providing the resized image to a plurality of support vector machine (SVM) classifiers as input, each SVM corresponding to a particular item of apparel; and 
 receiving output from each SVM classifier, the outputs indicative of a probability that the resized image contains the particular item of apparel corresponding to the SVM classifier; and 
   the identification of the item is further based on the outputs provided by the SVM classifiers.   
     
     
         19 . The method of  claim 14 , wherein:
 the identification of the item is further based on a ratio of height and width of the portion of the image.   
     
     
         20 . A machine-readable medium having instructions embodied thereon, the instructions executable by a processor of a machine to perform operations comprising:
 analyzing an image of a person to determine a pose of the person;   generating a set of candidate bounding boxes, each bounding box of the set of candidate bounding boxes defining a portion of the image; and   analyzing the portion of the image defined by a bounding box of the set of candidate bounding boxes to identify an item of apparel, the identification being based on a distance between the bounding box and a joint of the set of joints.