Volumetric video-based augmentation with user-generated content

Abstract

A processing system having at least one processor may obtain a two-dimensional source video, select a volumetric video associated with at least one feature of the source video from a library of volumetric videos, identify a first object in the source video, and determine a location of the first object within a space of the volumetric video. The processing system may further obtain a three-dimensional object model of the first object, texture map the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object, and modify the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video.

1 .- 20 . (canceled) 
     
     
         21 . A method comprising:
 obtaining, by a processing system including at least one processor, a source video, wherein the source video is a two-dimensional video;   selecting, by the processing system, a volumetric video from a library of volumetric videos;   identifying, by the processing system, a first object in the source video and an audio portion in the source video associated with the first object;   determining, by the processing system, a position of the first object within a space of the volumetric video;   obtaining, by the processing system, a three-dimensional object model of the first object;   texture mapping, by the processing system, the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and   modifying, by the processing system, the volumetric video to include the enhanced three-dimensional object model of the first object in the position of the first object within the space of the volumetric video, wherein the modifying further includes modifying the volumetric video to include the audio portion associated with the first object.   
     
     
         22 . The method of  claim 21 , further comprising:
 performing an alignment of the source video to the volumetric video, wherein the determining the position of the first object within the space of the volumetric video is in accordance with the alignment.   
     
     
         23 . The method of  claim 22 , wherein the alignment further comprises a time alignment of the source video and the volumetric video. 
     
     
         24 . The method of  claim 21 , wherein the obtaining the source video further comprises:
 obtaining at least one feature of the source video, wherein the at least one feature comprises at least one of:
 location information; 
 time information; 
 an event tag; or 
 a keyword. 
   
     
     
         25 . The method of  claim 24 , wherein the selecting comprises:
 determining that the volumetric video shares the at least one feature of the source video.   
     
     
         26 . The method of  claim 24 , wherein the selecting is in accordance with a ranked list of volumetric videos from the library of volumetric videos based upon a level of matching to the at least one feature of the source video. 
     
     
         27 . The method of  claim 21 , wherein the obtaining the source video further comprises:
 detecting a second object that appears in the source video, wherein at least one feature of the source video comprises the second object.   
     
     
         28 . The method of  claim 27 , wherein the selecting comprises:
 detecting the second object appearing in the volumetric video.   
     
     
         29 . The method of  claim 21 , further comprising:
 presenting, via an endpoint device, the volumetric video that is modified.   
     
     
         30 . The method of  claim 21 , further comprising:
 generating an output video comprising a two dimensional traversal of the volumetric video that is modified.   
     
     
         31 . The method of  claim 30 , wherein the output video is generated in accordance with a selection by a user of at least one viewing perspective within the space of the volumetric video. 
     
     
         32 . The method of  claim 31 , wherein the at least one viewing perspective is one of a plurality of viewing perspectives within the space of the volumetric video for which a two dimensional traversal of the volumetric video is available. 
     
     
         33 . The method of  claim 30 , further comprising:
 presenting, via an endpoint device, the output video.   
     
     
         34 . The method of  claim 33 , wherein the source video is obtained from the endpoint device. 
     
     
         35 . The method of  claim 21 , wherein the obtaining the three-dimensional object model of the first object is in accordance with a catalog of two-dimensional objects and three-dimensional object models that are matched to the two-dimensional objects. 
     
     
         36 . The method of  claim 35 , wherein the obtaining the three-dimensional object model comprises:
 matching the first object to one of the two-dimensional objects in the catalog; and   obtaining, from the catalog, the three-dimensional object model that is matched to the one two-dimensional object.   
     
     
         37 . The method of  claim 36 , wherein the matching is in accordance with a machine learning-based image detection model. 
     
     
         38 . The method of  claim 21 , wherein the modifying comprises changing voxel data of the volumetric video that corresponds to the first object so that the voxel data corresponds to the enhanced three-dimensional object model. 
     
     
         39 . A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor, cause the processing system to perform operations, the operations comprising:
 obtaining a source video, wherein the source video is a two-dimensional video;   selecting a volumetric video from a library of volumetric videos;   identifying a first object in the source video and an audio portion in the source video associated with the first object;   determining a position of the first object within a space of the volumetric video;   obtaining a three-dimensional object model of the first object;   texture mapping the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and   modifying the volumetric video to include the enhanced three-dimensional object model of the first object in the position of the first object within the space of the volumetric video, wherein the modifying further includes modifying the volumetric video to include the audio portion associated with the first object.   
     
     
         40 . A device comprising:
 a processing system including at least one processor; and   a computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising:
 obtaining a source video, wherein the source video is a two-dimensional video; 
 selecting a volumetric video from a library of volumetric videos; 
 identifying a first object in the source video and an audio portion in the source video associated with the first object; 
 determining a position of the first object within a space of the volumetric video; 
 obtaining a three-dimensional object model of the first object; 
 texture mapping the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and 
 modifying the volumetric video to include the enhanced three-dimensional object model of the first object in the position of the first object within the space of the volumetric video, wherein the modifying further includes modifying the volumetric video to include the audio portion associated with the first object.