Locating and Augmenting Object Features in Images

Abstract

A computer-implemented method and system are described for augmenting image data of an object in an image, the method comprising receiving captured image data defining a respective plurality of augmentation values to be applied to the captured image data, storing a plurality of augmentation representations, each representation identifying a respective portion of augmentation image data, selecting one of said augmentation image data and one of said augmentation representations based on at least one colourisation parameter, determining a portion of the augmentation image data to be applied based on the selected augmentation representation, augmenting the captured image data by applying said determined portion of the augmentation image data to the corresponding portion of the captured image data, and outputting the augmented captured image data.

1 . A computer-implemented method of generating augmented image data, the method comprising the computer-implemented steps of:
 storing, in a memory, data defining a plurality of virtual makeup products, each associated with colourisation parameters;   receiving, via a user interface, user input selection of at least one of said virtual makeup products;   retrieving, from the memory, the colourisation parameters associated with the or each selected virtual makeup product;   receiving data of an image captured by a camera; and   augmenting the captured image data based on the retrieved colourisation parameters.   
     
     
         2 . The method of  claim 1 , further comprising providing a plurality of shader modules, each configured to augmenting image data based on one or more colourisation parameters, wherein one or more of said shader modules is used to augment the captured image data. 
     
     
         3 . The method of  claim 2 , wherein each virtual makeup product is further associated with one or more of said shader modules. 
     
     
         4 . The method of  claim 1 , further comprising generating at least one augmentation representation based on respective predefined mask data identifying coordinates of a plurality of masked pixels, wherein the captured image data is augmented further based on the generated at least one augmentation representation. 
     
     
         5 . The method of  claim 4 , wherein generating the at least one augmentation representation comprises:
 receiving data identifying coordinates of a plurality of labelled feature points defining a detected object in the captured image;   generating at least one augmentation representation based on respective predefined mask data identifying coordinates of a plurality of masked pixels, by:   retrieving data defining a plurality of polygonal regions of augmentation image data determined for the detected object, wherein the augmentation image data defines a plurality of augmentation values to be applied to the captured image data, and wherein each polygonal region is defined by three or more vertices, each vertex of the three or more vertices being associated with a corresponding labelled feature point;   placing the retrieved plurality of polygonal regions over the respective mask data;   identifying polygonal regions that include at least one masked pixel; and   storing data representing the identified subset of polygonal regions.   
     
     
         6 . The method of  claim 1 , further comprising locating an object in the captured image, by:
 storing a representation of the object, the representation including data defining a first object model and a corresponding trained function to fit the first object model to the captured image data, and data defining at least one second object model comprising a subset of the data defining the first object model, and at least one corresponding trained function to fit the respective second object model to the captured image data, wherein the first object model defines a shape of the whole object and the at least one second object model defines a shape of a portion of the object;
 determining an approximate location of the object in the captured image, by generating a candidate global shape of the object based on the first object model, and using the corresponding trained function to update the candidate object global shape based on the captured image data; and 
 refining the location of the object in the captured image by splitting the candidate global shape into one or more candidate object sub-shapes based on the at least one second object models, and determining a location of the one or more candidate object sub-shapes based on the second object model and its corresponding trained function. 
   
     
     
         7 . A system comprising one or more processors configured to perform processing generate augmented image data by:
 storing, in a memory, data defining a plurality of virtual makeup products, each associated with colourisation parameters;   receiving, via a user interface, user input selection of at least one of said virtual makeup products;   retrieving, from the memory, the colourisation parameters associated with the or each selected virtual makeup product;   receiving data of an image captured by a camera; and   augmenting the captured image data based on the retrieved colourisation parameters.   
     
     
         8 . The system of  claim 7 , wherein the one or more processors are further configured to provide a plurality of shader modules, each configured to augmenting image data based on one or more colourisation parameters, wherein one or more of said shader modules is used to augment the captured image data. 
     
     
         9 . The system of  claim 8 , wherein each virtual makeup product is further associated with one or more of said shader modules. 
     
     
         10 . The system of  claim 7 , wherein the one or more processors are further configured to generate at least one augmentation representation based on respective predefined mask data identifying coordinates of a plurality of masked pixels, wherein the captured image data is augmented further based on the generated at least one augmentation representation. 
     
     
         11 . The system of  claim 10 , wherein the one or more processors are further configured to generate the at least one augmentation representation by:
 receiving data identifying coordinates of a plurality of labelled feature points defining a detected object in the captured image;   generating at least one augmentation representation based on respective predefined mask data identifying coordinates of a plurality of masked pixels, by:   retrieving data defining a plurality of polygonal regions of augmentation image data determined for the detected object, wherein the augmentation image data defines a plurality of augmentation values to be applied to the captured image data, and wherein each polygonal region is defined by three or more vertices, each vertex of the three or more vertices being associated with a corresponding labelled feature point;   placing the retrieved plurality of polygonal regions over the respective mask data;   identifying polygonal regions that include at least one masked pixel; and   storing data representing the identified subset of polygonal regions.   
     
     
         12 . The system of  claim 7 , wherein the one or more processors are further configured to locate an object in the captured image by:
 storing a representation of the object, the representation including data defining a first object model and a corresponding trained function to fit the first object model to the captured image data, and data defining at least one second object model comprising a subset of the data defining the first object model, and at least one corresponding trained function to fit the respective second object model to the captured image data, wherein the first object model defines a shape of the whole object and the at least one second object model defines a shape of a portion of the object;
 determining an approximate location of the object in the captured image, by generating a candidate global shape of the object based on the first object model, and using the corresponding trained function to update the candidate object global shape based on the captured image data; and 
 refining the location of the object in the captured image by splitting the candidate global shape into one or more candidate object sub-shapes based on the at least one second object models, and determining a location of the one or more candidate object sub-shapes based on the second object model and its corresponding trained function. 
   
     
     
         13 . A non-transitory computer-readable medium comprising computer-executable instructions, that when executed perform the method of generating augmented image data by:
 storing, in a memory, data defining a plurality of virtual makeup products, each associated with colourisation parameters;   receiving, via a user interface, user input selection of at least one of said virtual makeup products;   retrieving, from the memory, the colourisation parameters associated with the or each selected virtual makeup product;   receiving data of an image captured by a camera; and   augmenting the captured image data based on the retrieved colourisation parameters.