Viewport-based transcoding for immersive visual streams

Abstract

In one example, a processing system including at least one processor may obtain a predicted viewport of a mobile computing device for an immersive visual stream, identify a first plurality of blocks of a frame of the immersive visual stream that are associated with the predicted viewport, encode the first plurality of blocks at a first encoding quality level, and encode a second plurality of blocks of the frame at a second encoding quality level, where the second encoding quality level is associated with a lesser visual quality as compared to the first encoding quality level and where the second plurality of blocks are outside of the predicted viewport. The processing system may then transmit the frame having the first plurality of blocks encoded at the first encoding quality level and the second plurality of blocks encoded at the second encoding quality level to the mobile computing device.

What is claimed is: 
     
         1 . A method comprising:
 generating, by a processor of a mobile computing device, a predicted viewport for an immersive visual stream, wherein the predicted viewport is for a first future time period for which the predicted viewport is predicted;   sending, by the processor, the predicted viewport to a server for identifying a first plurality of blocks of a frame of the immersive visual stream that is associated with the predicted viewport, and for encoding the first plurality of blocks at a first encoding quality level; and   receiving from the server, by the processor, the frame having the first plurality of blocks encoded at the first encoding quality level, a second plurality of blocks encoded at a second encoding quality level, and a third plurality of blocks encoded at a third encoding quality level, wherein the second encoding quality level is associated with a lesser visual quality as compared to the first encoding quality level, wherein the second plurality of blocks is outside of the predicted viewport, wherein the third encoding quality level is associated with a lesser visual quality as compared to the second encoding quality level, and wherein the third plurality of blocks is further outside of the predicted viewport as compared to the second plurality of blocks.   
     
     
         2 . The method of  claim 1 , wherein the immersive visual stream comprises:
 a video having dimensions that exceed display capabilities of the mobile computing device;   a 360 degree video; or   a volumetric video.   
     
     
         3 . The method of  claim 1 , wherein a block of the frame comprises a spatially adjacent group of pixels or voxels within the frame. 
     
     
         4 . The method of  claim 3 , wherein the block comprises a macroblock. 
     
     
         5 . The method of  claim 3 , wherein the block comprises a micro-block. 
     
     
         6 . The method of  claim 1 , wherein the second plurality of blocks comprises blocks of the frame other than the first plurality of blocks. 
     
     
         7 . The method of  claim 1 , wherein the first encoding quality level and the second encoding quality level are selected in accordance with a channel quality experienced by the mobile computing device. 
     
     
         8 . The method of  claim 1 , wherein the first encoding quality level, the second encoding quality level, and the third encoding quality level are selected in accordance with a channel quality experienced by the mobile computing device. 
     
     
         9 . The method of  claim 1 , wherein positions of the second plurality of blocks within the frame are weighted towards a direction of a trajectory of the predicted viewport, and wherein positions of the third plurality of blocks within the frame are weighted towards a direction opposite the trajectory of the predicted viewport. 
     
     
         10 . The method of  claim 9 , wherein the trajectory is determined in accordance with a plurality of previous viewports of the mobile computing device for a plurality prior frames of the immersive visual stream. 
     
     
         11 . The method of  claim 10 , wherein the trajectory is further determined based upon at least one of:
 historical viewport information regarding previous viewports with respect to the frame for other mobile computing devices that have experienced the immersive visual stream; or   salient portions of the frame as determined in accordance with an image saliency detection algorithm.   
     
     
         12 . The method of  claim 1 , wherein the predicted viewport is further based upon at least one of:
 historical viewport information regarding previous viewports with respect to the frame for other mobile computing devices that have experienced the immersive visual stream; or   salient portions of the frame as determined in accordance with an image saliency detection algorithm.   
     
     
         13 . The method of  claim 1 , wherein the second plurality of blocks of the frame of the immersive visual stream is encoded at the second encoding quality level in response to determining that an accuracy associated with the predicted viewport is below a threshold, and wherein the third plurality of blocks of the frame of the immersive visual stream is encoded at the third encoding quality level in response to determining that the accuracy associated with the predicted viewport is below the threshold. 
     
     
         14 . The method of  claim 1 , wherein the generating the predicted viewport comprises:
 calculating the predicted viewport based upon a plurality of previous viewports of the mobile computing device for a plurality prior frames of the immersive visual stream.   
     
     
         15 . The method of  claim 14 , wherein the calculating the predicted viewport is further based upon at least one of:
 historical viewport information regarding previous viewports with respect to the frame for other mobile computing devices that have experienced the immersive visual stream; or   salient portions of the frame as determined in accordance with an image saliency detection algorithm.   
     
     
         16 . A mobile computing device comprising:
 a processing system including at least one processor; and   a computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising:
 generating a predicted viewport for an immersive visual stream, wherein the predicted viewport is for a first future time period for which the predicted viewport is predicted; 
 sending the predicted viewport to a server for identifying a first plurality of blocks of a frame of the immersive visual stream that is associated with the predicted viewport, and for encoding the first plurality of blocks at a first encoding quality level; and 
 receiving from the server the frame having the first plurality of blocks encoded at the first encoding quality level, a second plurality of blocks encoded at a second encoding quality level, and a third plurality of blocks encoded at a third encoding quality level, wherein the second encoding quality level is associated with a lesser visual quality as compared to the first encoding quality level, wherein the second plurality of blocks is outside of the predicted viewport, wherein the third encoding quality level is associated with a lesser visual quality as compared to the second encoding quality level, and wherein the third plurality of blocks is further outside of the predicted viewport as compared to the second plurality of blocks. 
   
     
     
         17 . A non-transitory computer-readable medium storing instructions which, when executed by a processor of a mobile computing device, cause the processor to perform operations, the operations comprising:
 generating a predicted viewport for an immersive visual stream, wherein the predicted viewport is for a first future time period for which the predicted viewport is predicted;   sending the predicted viewport to a server for identifying a first plurality of blocks of a frame of the immersive visual stream that is associated with the predicted viewport, and for encoding the first plurality of blocks at a first encoding quality level; and   receiving from the server the frame having the first plurality of blocks encoded at the first encoding quality level, a second plurality of blocks encoded at a second encoding quality level, and a third plurality of blocks encoded at a third encoding quality level, wherein the second encoding quality level is associated with a lesser visual quality as compared to the first encoding quality level, wherein the second plurality of blocks is outside of the predicted viewport, wherein the third encoding quality level is associated with a lesser visual quality as compared to the second encoding quality level, and wherein the third plurality of blocks is further outside of the predicted viewport as compared to the second plurality of blocks.   
     
     
         18 . The non-transitory computer-readable medium of  claim 17 , wherein the immersive visual stream comprises:
 a video having dimensions that exceed display capabilities of the mobile computing device;   a 360 degree video; or   a volumetric video.   
     
     
         19 . The non-transitory computer-readable medium of  claim 17 , wherein a block of the frame comprises a spatially adjacent group of pixels or voxels within the frame. 
     
     
         20 . The non-transitory computer-readable medium of  claim 19 , wherein the block comprises a macroblock.