Image processing apparatus, image processing method, and storage medium

Abstract

The image quality of a virtual viewpoint image is improved. The image processing apparatus according to the present disclosure obtains data of a plurality of captured images obtained by capturing an object from a plurality of directions and virtual viewpoint information relating to a position of a virtual viewpoint and a viewing direction at the virtual viewpoint, sets a generation scheme of a virtual viewpoint image based on features of the object, which are obtained based on the virtual viewpoint information, and generates the virtual viewpoint image based on the virtual viewpoint information and the set generation scheme of the virtual viewpoint image.

What is claimed is: 
     
         1 . An image processing apparatus comprising:
 one or more hardware processors; and   one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions for:
 obtaining data of a plurality of captured images obtained by capturing an object from a plurality of directions; 
 obtaining virtual viewpoint information relating to a position of a virtual viewpoint and a viewing direction at the virtual viewpoint; 
 setting a generation scheme of a virtual viewpoint image based on features of the object, which are obtained based on the virtual viewpoint information; and 
 generating the virtual viewpoint image based on the virtual viewpoint information and the set generation scheme of the virtual viewpoint image. 
   
     
     
         2 . The image processing apparatus according to  claim 1 , wherein
 the one or more programs further include instructions for:
 setting either a rendering scheme based on shape or a rendering scheme based on radiance fields as the generation scheme of the virtual viewpoint image. 
   
     
     
         3 . The image processing apparatus according to  claim 2 , wherein
 the rendering scheme based on shape is a scheme in which rendering is performed based on information indicating a three-dimensional shape of the object, which is generated by using the plurality of captured images by a visual hull method, and data of the plurality of captured images.   
     
     
         4 . The image processing apparatus according to  claim 2 , wherein
 the radiance fields are a function that is estimated as results of performing repetitive processing so that a difference between a pixel value in the plurality of captured images and a pixel value obtained by volume rendering based on the radiance fields becomes small and   the rendering scheme based on radiance fields is a scheme in which rendering is performed by inputting the virtual viewpoint information to the function indicating the radiance fields.   
     
     
         5 . The image processing apparatus according to  claim 2 , wherein
 the features of the object are complicatedness of the shape of the object in a case where the object is viewed from the virtual viewpoint and   the one or more programs further include instructions for:
 setting, the more complicated the object shape, the more preferentially the rendering schemed based on radiance fields as the generation scheme of the virtual viewpoint image. 
   
     
     
         6 . The image processing apparatus according to  claim 2 , wherein
 the features of the object are a size of the object per pixel in the virtual viewpoint image in a case where the object is viewed from the virtual viewpoint and   the one or more programs further include instructions for:
 setting, the smaller the object size per pixel, the more preferentially the rendering scheme based on radiance fields as the generation scheme of the virtual viewpoint image. 
   
     
     
         7 . The image processing apparatus according to  claim 2 , wherein
 the features of the object are complicatedness of the object shape and the size of the object per pixel in the virtual viewpoint image in a case where the object is viewed from the virtual viewpoint and   the one or more programs further include instructions for:
 setting, the more complicated the object shape and the smaller the object size per pixel, the more preferentially the rendering scheme based on radiance fields as the generation scheme of the virtual viewpoint image. 
   
     
     
         8 . The image processing apparatus according to  claim 1 , wherein
 the one or more programs further include instructions for:
 selecting, in a case where a plurality of the objects included as an image in the plurality of captured images exists, one of the plurality of the objects that exist as a representative object and 
 setting the generation scheme of the virtual viewpoint image based on the features of the representative object in a case where the representative object is viewed from the virtual viewpoint. 
   
     
     
         9 . The image processing apparatus according to  claim 1 , wherein
 the one or more programs further include instructions for:
 setting, in a case where a plurality of the objects included as an image in the plurality of captured images exists, the generation scheme of the virtual viewpoint image based on the features of the object in a case where the object is viewed from the virtual viewpoint for each of the objects; 
 generating a plurality of provisional virtual viewpoint images by a plurality of generation schemes different from one another based on the generation scheme set for each of the objects; and 
 generating the virtual viewpoint image by compositing the plurality of the generated provisional virtual viewpoint images. 
   
     
     
         10 . The image processing apparatus according to  claim 9 , wherein
 the one or more programs further include instructions for:
 compositing, in generating the virtual viewpoint image, the plurality of the generated provisional virtual viewpoint images based on overlapping in a case of being viewed from the virtual viewpoint of the plurality of the objects that exist. 
   
     
     
         11 . An image processing method comprising the steps of:
 obtaining data of a plurality of captured images obtained by capturing an object from a plurality of directions;   obtaining virtual viewpoint information relating to a position of a virtual viewpoint and a viewing direction at the virtual viewpoint;   setting a generation scheme of a virtual viewpoint image based on features of the object, which are obtained based on the virtual viewpoint information; and   generating the virtual viewpoint image based on the virtual viewpoint information and the set generation scheme of the virtual viewpoint image.   
     
     
         12 . A non-transitory computer readable storage medium storing a program for causing a computer to perform a control method of an image processing apparatus, the control method comprising the steps of:
 obtaining data of a plurality of captured images obtained by capturing an object from a plurality of directions;   obtaining virtual viewpoint information relating to a position of a virtual viewpoint and a viewing direction at the virtual viewpoint;   setting a generation scheme of a virtual viewpoint image based on features of the object, which are obtained based on the virtual viewpoint information; and   generating the virtual viewpoint image based on the virtual viewpoint information and the set generation scheme of the virtual viewpoint image.