File generation apparatus, image generation apparatus based on file, file generation method and storage medium

Abstract

A file generation apparatus generates a file which includes material data used for generation of a virtual viewpoint image that is based on a multi-viewpoint image and type information for specifying a type of the material data, and outputs the generated file.

1 - 24 . (canceled) 
     
     
         25 . An image processing system comprising:
 one or more memories storing instructions; and   one or more processors executing the instructions to: 
 obtain, of a plurality of 3D models used for generation of a virtual viewpoint image that is based on a plurality of captured images obtained by capturing using a plurality of capturing devices, the 3D model specified based on virtual viewpoint information obtained by a user operation and indicating a position and an orientation of a virtual camera, the 3D model being included in a capturing range of the virtual camera; and 
 generate the virtual viewpoint image based on an obtained 3D model and the virtual viewpoint information. 
   
     
     
         26 . The image processing system according to  claim 25 , wherein 
 in the obtaining, of the plurality of 3D models, a 3D model surrounded by a bounding box included in a capturing range is obtained.   
     
     
         27 . The image processing system according to  claim 26 , wherein 
 in the obtaining, of the plurality of 3D models, a 3D model surrounded by a bounding box whose vertex is included in a capturing range is obtained.   
     
     
         28 . The image processing system according to  claim 25 , wherein the one or more processors further execute the instructions to 
 specify, of the plurality of 3D models, the 3D model included in a capturing range of the virtual camera; and   output a specified 3D model.   
     
     
         29 . The image processing system according to  claim 25 , wherein 
 the plurality of 3D models correspond to information indicating a frame of the virtual viewpoint image,   the virtual viewpoint information includes information indicating a frame of the virtual viewpoint image, and   in the obtaining, the 3D model corresponding to information indicating a frame included in the virtual viewpoint information is obtained.   
     
     
         30 . The image processing system according to  claim 25 , wherein 
 the plurality of 3D models are included in a file in which a plurality of kinds of material data used for generation of a virtual viewpoint image are integrated, and   in the obtaining, from the file, the 3D model specified based on the virtual viewpoint information is obtained.   
     
     
         31 . The image processing system according to  claim 30 , wherein 
 the file includes the plurality of kinds of material data and information indicating a kind of the material data .   
     
     
         32 . An image processing system comprising:
 one or more memories storing instructions; and   one or more processors executing the instructions to: 
 obtain, of a plurality of foreground images, a plurality of background images, or both used for generation of a virtual viewpoint image based on a plurality of captured images obtained by capturing using a plurality of capturing devices, at least either of the foreground image and the background image specified based on virtual viewpoint information obtained by a user operation and indicating a position and an orientation of a virtual camera; and 
 generate the virtual viewpoint image based on at least either of an obtained foreground image and an obtained background image and the virtual viewpoint information. 
   
     
     
         33 . The image processing system according to  claim 32 , wherein 
 in the obtaining, the at least either of the foreground image and the background image is obtained with the capturing device specified based on a position of the virtual camera.   
     
     
         34 . The image processing system according to  claim 33 , wherein 
 in the obtaining, at least either of the foreground image and the background image corresponding to a captured image obtained by capturing using the capturing device close to a position of the virtual camera.   
     
     
         35 . An image processing method comprising:
 obtaining, of a plurality of 3D models used for generation of a virtual viewpoint image that is based on a plurality of captured images obtained by capturing using a plurality of capturing devices, the 3D model specified based on virtual viewpoint information obtained by a user operation and indicating a position and an orientation of a virtual camera, the 3D model being included in a capturing range of the virtual camera; and   generating the virtual viewpoint image based on an obtained 3D model and the virtual viewpoint information.   
     
     
         36 . An image processing method comprising:
 obtaining, of a plurality of foreground images, a plurality of background images, or both used for generation of a virtual viewpoint image based on a plurality of captured images obtained by capturing using a plurality of capturing devices, at least either of the foreground image and the background image specified based on virtual viewpoint information obtained by a user operation and indicating a position and an orientation of a virtual camera; and   generating the virtual viewpoint image based on at least either of an obtained foreground image and an obtained background image and the virtual viewpoint information.   
     
     
         37 . A non-transitory computer-readable storage medium storing a program for causing a computer to function as an image processing system comprising:
 one or more memories storing instructions:   one or more processors executing the instructions to: 
 obtain, of a plurality of 3D models used for generation of a virtual viewpoint image that is based on a plurality of captured images obtained by capturing using a plurality of capturing devices, the 3D model specified based on virtual viewpoint information obtained by a user operation and indicating a position and an orientation of a virtual camera, the 3D model being included in a capturing range of the virtual camera; and 
 generate the virtual viewpoint image based on an obtained 3D model and the virtual viewpoint information. 
   
     
     
         38 . A non-transitory computer-readable storage medium storing a program for causing a computer to function as an image processing system comprising: 
 one or more memories storing instructions:   one or more processors executing the instructions to: 
 obtain, of a plurality of foreground images, a plurality of background images, or both used for generation of a virtual viewpoint image based on a plurality of captured images obtained by capturing using a plurality of capturing devices, at least either of the foreground image and the background image specified based on virtual viewpoint information obtained by a user operation and indicating a position and an orientation of a virtual camera; and 
 generate the virtual viewpoint image based on at least either of an obtained foreground image and an obtained background image and the virtual viewpoint information.