Image processing apparatus and image processing method

Abstract

An image processing apparatus for generating a virtual viewpoint image corresponding to a position and a direction of a virtual viewpoint obtains information related to the position and the direction of the virtual viewpoint, obtains a captured image from one or more cameras, determines whether the obtained camera image is to be used for generating the virtual viewpoint image, and generates, based on the captured image corresponding to the determination result, the virtual viewpoint image corresponding to the position and the direction of the virtual viewpoint.

What is claimed is: 
     
         1 . An image processing apparatus comprising:
 an obtainment unit configured to obtain a captured image from one or more cameras;   a determination unit configured to determine whether the captured image obtained by the obtainment unit is not to be used for generating a virtual viewpoint image corresponding to a position and a direction of a virtual viewpoint; and   a notification unit configured to notify a generation device, which is configured to generate the virtual viewpoint image, of information indicating a determination result by the determination unit.   
     
     
         2 . The apparatus according to  claim 1 , wherein the determination unit executes detection processing for detecting a specific object from the captured image, and the captured image in which the specific object has been detected is determined not to be used for generating the virtual viewpoint image. 
     
     
         3 . The apparatus according to  claim 2 , wherein the specific object is at least an object that satisfies at least one of conditions of an object of a size not less than a threshold and an object which does not have a predetermined image pattern. 
     
     
         4 . The apparatus according to  claim 2 , wherein the specific object is an object that moves at a speed not less than a threshold. 
     
     
         5 . The apparatus according to  claim 1 , wherein the determination unit determines, based on a result comparing captured images from two or more cameras, whether the captured image from the one or more cameras is not to be used for generating the virtual viewpoint image. 
     
     
         6 . The apparatus according to  claim 5 , wherein the determination unit determines whether the captured image from the one or more cameras is not to be used for generating the virtual viewpoint image based on at least one or a combination of
 (1) a difference between a size of an object detected from a captured image of a first camera and a size of an object detected from a captured image of a second camera,   (2) a pixel count in which a difference between pixel values of the captured image of the first camera and pixel values of the captured image of the second camera is not less than a threshold, and   (3) a comparison of statistical information related to the pixel values of the captured image of the first camera and the pixel values of the captured image of the second camera.   
     
     
         7 . The apparatus according to  claim 1 , wherein the virtual viewpoint image is generated based on three-dimensional shape model information that is generated by using captured images other than a captured image determined, by the determination unit, not to be used for generating the virtual viewpoint image. 
     
     
         8 . The apparatus according to  claim 1 , wherein the virtual viewpoint image is generated based on composition processing of one or a plurality of captured images specified based on the position and direction of the virtual viewpoint among captured images other than a captured image determined, by the determination unit, not to be used for generating the virtual viewpoint image. 
     
     
         9 . The apparatus according to  claim 1 , wherein the notification unit transmits, to the generation device of generating the virtual viewpoint image, added image data obtained by adding predetermined information of inadequacy to image data corresponding to a captured image determined, by the determination unit, not to be used for generating the virtual viewpoint image. 
     
     
         10 . The apparatus according to  claim 9 , further comprising:
 a compression unit configured to compress the image data corresponding to the captured image determined, by the determination unit, not to be used for generating the virtual viewpoint image at a compression rate higher than that for image data corresponding to a captured image that has not been determined not to be used for generating the virtual viewpoint image,   wherein the notification unit transmits the image data compressed by the compression unit to the generation device.   
     
     
         11 . The apparatus according to  claim 9 , wherein the notification unit transmits image data corresponding to a captured image of a period determined, by the determination unit, not to be used for generating the virtual viewpoint image at a lower frame rate than image data corresponding to a captured image of a period that has not been determined not to be used for generating the virtual viewpoint image. 
     
     
         12 . The apparatus according to  claim 1 , wherein the notification unit does not transmit, to the generation device for generating the virtual viewpoint image, image data corresponding to a captured image that has not been determined, by the determination unit, not to be used for generating the virtual viewpoint image data, and the notification unit transmits, to the generation device, a captured image determined not to be used for generating the virtual viewpoint image. 
     
     
         13 . An image processing method for generating a virtual viewpoint image corresponding to a position and a direction of a virtual viewpoint, the method comprising:
 obtaining information related to the position and the direction of the virtual viewpoint;   obtaining a captured image from one or more cameras;   determining whether the obtained captured image is to be used for generating the virtual viewpoint image; and   generating, based on the captured image corresponding to the determination result, a virtual viewpoint image corresponding to the position and the direction of the virtual viewpoint.   
     
     
         14 . A non-transitory computer-readable storage medium storing a program configured to cause a computer to execute an image processing method for generating a virtual viewpoint image corresponding to a position and a direction of a virtual viewpoint, the method comprising:
 obtaining information related to the position and the direction of the virtual viewpoint;   obtaining a captured image from one or more cameras;   determining whether the obtained captured image is to be used for generating the virtual viewpoint image; and   generating, based on the captured image corresponding to the determination result, a virtual viewpoint image corresponding to the position and the direction of the virtual viewpoint.