Image processing apparatus, image processing system, image processing method, and storage medium

Abstract

An image processing system includes an image obtaining unit that obtains images based on capturing from plural directions by plural cameras, an information obtaining unit that obtains viewpoint information indicating a virtual viewpoint, and a generation unit configured to generate virtual viewpoint images on a basis of the obtained images and viewpoint information. The generation unit generates a first virtual viewpoint image outputted to a display apparatus that displays an image for a user to specify a virtual viewpoint and a second virtual viewpoint image outputted to an output destination different from the display apparatus by using at least one of data generated in a process for generating the first virtual viewpoint image by image processing using the plural images obtained by the image obtaining unit and the first virtual viewpoint image, the second virtual viewpoint image having a higher image quality than that of the first virtual viewpoint image.

1 . An image processing system comprising:
 one or more hardware processors; and   one or more memories which store instructions executable by the one or more hardware processors to cause the image processing system to perform at least:
 obtaining a plurality of images captured from respectively different positions by a plurality of image capturing apparatuses; 
   accepting an input corresponding to an operation for specifying a virtual viewpoint; and   generating a plurality of virtual viewpoint images based on the obtained plurality of images and the accepted input, the plurality of virtual viewpoint images including a first virtual viewpoint image and a second virtual viewpoint image,   
       wherein a number of image capturing apparatuses corresponding to an image to be used for generating the second virtual viewpoint image is larger than a number of image capturing apparatuses corresponding to an image to be used for generating the first virtual viewpoint image, and 
       wherein a plurality of image capturing apparatuses corresponding to images to be used for generating the second virtual viewpoint image comprises a plurality of image capturing apparatuses corresponding to images to be used for generating the first virtual viewpoint image. 
     
     
         2 . The image processing system according to  claim 1 , wherein the second virtual viewpoint image has a larger image data size per frame of a moving image than that of the first virtual viewpoint image. 
     
     
         3 . The image processing system according to  claim 1 ,
 wherein the first virtual viewpoint image is a virtual viewpoint image to be broadcast live, and   wherein the second virtual viewpoint image is a virtual viewpoint image to be broadcast after recording.   
     
     
         4 . The image processing system according to  claim 1 , wherein the second virtual viewpoint image is generated based on a three dimensional model of an object which is generated based on the obtained plurality of images and whose accuracy is higher than that of a three dimensional model of the object used for generating the first virtual viewpoint image. 
     
     
         5 . The image processing system according to  claim 1 , wherein the second virtual viewpoint image is an image to be displayed for a viewer to see an image generated in response to the operation. 
     
     
         6 . The image processing system according to  claim 1 ,
 wherein the instructions further cause the image processing system to perform outputting the first virtual viewpoint image and the second virtual viewpoint image,   wherein timing when the first virtual viewpoint image is output is earlier than timing when the second virtual viewpoint image is output.   
     
     
         7 . The image processing system according to  claim 1 , wherein the instructions further cause the image processing system to perform controlling output of the first virtual viewpoint image and the second virtual viewpoint image so that the first virtual viewpoint image is displayed in a first display area and the second virtual viewpoint image is displayed in a second display area different from the first display area. 
     
     
         8 . The image processing system according to  claim 1 , wherein the second virtual viewpoint image is generated by performing image processing using at least one of image data to be generated in a process of generating the first virtual viewpoint image from the obtained image and the first virtual viewpoint image. 
     
     
         9 . The image processing system according to  claim 1 , wherein the second virtual viewpoint image is generated by performing image processing of increasing an image data size per frame of a moving image on another virtual viewpoint image generated based on the plurality of images and the input, and the first virtual viewpoint image is generated by performing processing that is a part of processing of generating the second virtual viewpoint image from said another virtual viewpoint image and that is executed in a processing time smaller than or equal to a predetermined value. 
     
     
         10 . The image processing system according to  claim 1 ,
 wherein the first virtual viewpoint image is an image indicating a shape of an object to be captured by at least one of the plurality of image capturing apparatuses, and   wherein the second virtual viewpoint image is an image indicating a color of the object, the color not appearing in the first virtual viewpoint image, in addition to the shape of the object.   
     
     
         11 . The image processing system according to  claim 1 , wherein a number of gradations of colors included in the second virtual viewpoint image is larger than a number of gradations of colors included in the first virtual viewpoint image. 
     
     
         12 . The image processing system according to  claim 1 , wherein a resolution of the second virtual viewpoint image is higher than a resolution of the first virtual viewpoint image. 
     
     
         13 . The image processing system according to  claim 1 ,
 wherein the instructions further cause the image processing system to perform outputting a parameter related to image quality of a virtual viewpoint image to the generation means,   wherein the generation means generates the first virtual viewpoint image and the second virtual viewpoint image based on the parameter related to image quality to be output from the output means.   
     
     
         14 . The image processing system according to  claim 13 , wherein the parameter related to image quality includes at least one of a number of image capturing apparatuses corresponding to a captured image to be used to generate a virtual viewpoint image, a resolution of a virtual viewpoint image, and a permissible time as a processing time for generating a virtual viewpoint image. 
     
     
         15 . The image processing system according to  claim 1 , wherein a processing amount of image processing for generating the second virtual viewpoint image is larger than a processing amount of image processing for generating the first virtual viewpoint image. 
     
     
         16 . An image processing method comprising:
 obtaining a plurality of images captured from respectively different positions by a plurality of image capturing apparatuses;   accepting an input corresponding to an operation for specifying a virtual viewpoint; and   generating a plurality of virtual viewpoint images based on the obtained plurality of images and the accepted input, the plurality of virtual viewpoint images including a first virtual viewpoint image and a second virtual viewpoint image,   wherein a number of image capturing apparatuses corresponding to an image to be used for generating the second virtual viewpoint image is larger than a number of image capturing apparatuses corresponding to an image to be used for generating the first virtual viewpoint image, and   wherein a plurality of image capturing apparatuses corresponding to images to be used for generating the second virtual viewpoint image comprises a plurality of image capturing apparatuses corresponding to images to be used for generating the first virtual viewpoint image.   
     
     
         17 . The image processing method according to  claim 16 , wherein the second virtual viewpoint image has a larger image data size per frame of a moving image than that of the first virtual viewpoint image. 
     
     
         18 . The image processing method according to  claim 16 ,
 wherein the first virtual viewpoint image is a virtual viewpoint image to be broadcast live, and   wherein the second virtual viewpoint image is a virtual viewpoint image to be broadcast after recording.   
     
     
         19 . The image processing method according to  claim 16 ,
 wherein the first virtual viewpoint image is an image indicating a shape of an object to be captured by at least one of the plurality of image capturing apparatuses, and   wherein the second virtual viewpoint image is an image indicating a color of the object, the color not appearing in the first virtual viewpoint image, in addition to the shape of the object.   
     
     
         20 . A non-transitory storage medium that stores a program for causing a computer to execute an image processing method, the image processing method comprising:
 obtaining a plurality of images captured from respectively different positions by a plurality of image capturing apparatuses;   accepting an input corresponding to an operation for specifying a virtual viewpoint; and   generating a plurality of virtual viewpoint images based on the obtained plurality of images and the accepted input, the plurality of virtual viewpoint images including a first virtual viewpoint image and a second virtual viewpoint image,   wherein a number of image capturing apparatuses corresponding to an image to be used for generating the second virtual viewpoint image is larger than a number of image capturing apparatuses corresponding to an image to be used for generating the first virtual viewpoint image, and   wherein a plurality of image capturing apparatuses corresponding to images to be used for generating the second virtual viewpoint image comprises a plurality of image capturing apparatuses corresponding to images to be used for generating the first virtual viewpoint image.