Apparatus and method with object posture estimating

Abstract

An apparatus and method of estimating an object posture are provided. A method includes determining key point information in an image to be processed, determining modified key point information based on a key point feature map corresponding to the key point information, and estimating an object posture of an object in the image to be processed, based on the modified key point information.

What is claimed is: 
     
         1 . A method of estimating an object posture, the method comprising:
 determining key point information in an image;   determining modified key point information based on a key point feature map corresponding to the key point information; and   estimating an object posture of an object in the image, based on the modified key point information.   
     
     
         2 . The method of  claim 1 , further comprising:
 prior to the determining the key point information in the image,   receiving an input image; and   obtaining the image by converting the input image to an image having a preset image style.   
     
     
         3 . The method of  claim 2 , wherein
 the converting the input image to the image having the preset image style comprises:
 extracting an image-content feature of the input image by a content neural network; 
 obtaining a preset image-style feature; 
 generating an integrated feature by integrating the image-content feature of the input image with the preset image-style feature; and 
   obtaining the image by rendering the input image with a renderer, wherein the rendering is based on the integrated feature.   
     
     
         4 . The method of  claim 3 , wherein
 the image-content feature of the input image comprises a low-resolution image-content feature, and   the preset image-style feature comprises a low-resolution image-style feature.   
     
     
         5 . The method of  claim 1 , wherein
 the determining the modified key point information comprises:
 obtaining key point offset residual information by performing offset regression on the key point feature map; and 
 obtaining the modified key point information based on the key point offset residual information and the key point information. 
   
     
     
         6 . The method of  claim 1 , further comprising:
 performing a refinement process on the object posture to increase an accuracy thereof by using the estimated object posture.   
     
     
         7 . The method of  claim 6 , wherein
 the key point information comprises two-dimensional (2D) key point information, and   the performing the refinement process comprises:
 determining 3D key point mapping information of the object in the based on the estimated object posture; 
 obtaining updated key point information based on the 3D key point mapping information; 
 redetermining the modified key point information based on the updated key point information; and 
 estimating the object posture based on the redetermined modified key point information. 
   
     
     
         8 . The method of  claim 7 , wherein
 the determining the 3D key point mapping information is based on the estimated object posture, a preset 3D model set corresponding to the object, and a camera eigen matrix.   
     
     
         9 . The method of  claim 7 , wherein
 the performing the refinement process comprises:   iteratively performing the refinement process on the object posture until a termination condition is satisfied, wherein the termination condition comprises either:
 a difference of a key point before and after modification being less than a threshold, or 
 a number of modifications of the key point reaching a predetermined number of times. 
   
     
     
         10 . The method of  claim 9 , wherein
 the difference of a key point before and after modification being less than a threshold comprises:   either a sum of differences of at least one key point before and after modification being less than the threshold, or a difference of each key point of the at least one key point before and after modification being less than the threshold.   
     
     
         11 . The method of  claim 1 , wherein
 the determining the key point information comprises:
 determining key point coordinate information and key point visible information through a detector network that processes the image; and 
 determining the key point information based on the key point coordinate information and the key point visible information. 
   
     
     
         12 . The method of  claim 11 , wherein
 the determining the key point information further comprises:
 obtaining a region of interest (Rol) feature map in the image through the detector network, and 
 prior to the determining the modified key point information: determining the key point feature map corresponding the key point information in the Rol feature map, based on the key point information. 
   
     
     
         13 . The method of  claim 11 , further comprising:
 determining an object category of the object in the image through the detector network, and   prior to the determining the modified key point information: determining the key point feature map based on the object category and the key point information.   
     
     
         14 . A method of estimating an object posture, the method comprising:
 converting an input image to an image having a preset image style;   determining key point information in the image having the preset image style; and   estimating an object posture in the input image having the preset style based on the key point information.   
     
     
         15 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of  claim 1 . 
     
     
         16 . An apparatus for estimating an object posture, the apparatus comprising:
 one or more processors;   storage storing instructions configured to, when executed by the one or more processors, cause the one or more processors to:
 determine key point information in an image; 
 determine modified key point information based on a key point feature map corresponding to the key point information; and 
 estimate an object posture of an object in the image based on the modified key point information. 
   
     
     
         17 . The apparatus of  claim 16 , wherein
 the image is converted from an input image to be a version of the input image having an image style that the input image does not have.   
     
     
         18 . The apparatus of  claim 17 , wherein the determining the key point information comprises:
 extracting an image-content feature of the input image through a content network,   obtaining an image-style feature,   generating an integrated feature by integrating the image-content feature with the image-style feature, and   rendering the image based on the input image and based on the integrated feature.   
     
     
         19 . The apparatus of  claim 16 , wherein the modifying comprises:
 obtaining key point offset residual information by performing offset regression on the key point feature map, and   obtaining the modified key point information based on the key point offset residual information and the key point information.   
     
     
         20 . The apparatus of  claim 16 , wherein
 the estimating comprises: perform a refinement process on the object posture to increase an accuracy thereof by using the estimated object posture.