Partitioning, rendering, and blending of extended reality application instances

Abstract

An extended reality (XR) streaming method of streaming XR images to an XR device is described. The XR streaming method comprising, generating, virtual content to be rendered with an XR application instance; partitioning the virtual content to be rendered into a plurality of rendering jobs with the XR application instance; forwarding the plurality of rendering jobs to at least a first and second rendering instance of a plurality of rendering instances; generating first XR images based on the corresponding rendering job received by the first render instance from the XR application instance; generating second XR images based on the corresponding rendering job received by the second render instance from the XR application instance; and blending the first XR images with the second XR images, with a merging instance to obtain blended XR images to be displayed on the XR device.

The invention claimed is: 
     
         1 . A method of streaming extended reality (XR) images from a first computer device of a plurality of computer devices that are external to an XR device comprising:
 generating virtual content to be rendered with an XR application instance running on the first computer device;   partitioning the virtual content to be rendered into a plurality of rendering jobs with the XR application instance;   forwarding the plurality of rendering jobs to at least two rendering instances that are configured to render the respective rendering jobs, wherein the at least two rendering instances comprise a first rendering instance running on the first computer device and a second rendering instance running on a second computer device of the plurality of computer devices;   generating first XR images with the first rendering instance based on a first rendering job received from the XR application instance;   generating second XR images with the second rendering instance based on a second rendering job received from the XR application instance, wherein the first XR images and the second XR images are rendered by the first rendering instance and the second rendering instance in parallel;   streaming the second XR images, first depth data corresponding to the first XR images, and second depth data corresponding to the second XR images to a merging instance running on the first computer device, wherein the first depth data comprise depth information associated with pixels of the first XR images and the second depth data comprise depth information associated with pixels of the second XR images; and   blending the first XR images with the second XR images with the merging instance based on the first depth data and second depth data to obtain blended XR images to be displayed on the XR device.   
     
     
         2 . The method of  claim 1 , wherein blending the first XR images with the second XR images comprises positionally blending the first XR images with the second XR images based on at least one of an orientation of the XR device and a location of the XR device. 
     
     
         3 . The method of  claim 1 , further comprising assigning priorities to the first XR images and to the second XR images, and wherein the first XR images are blended with the second XR images based on the priorities assigned to the first XR images and to the second XR images. 
     
     
         4 . The method of  claim 3 , further comprising cutting pixels having a predefined cutout color from the first XR images and the second XR images with the merging instance. 
     
     
         5 . The method of  claim 1 , further comprising streaming alpha data to the merging instance with the first rendering instance and the second rendering instance, wherein the alpha data comprise transparency information associated with pixels of the first XR images and the second XR images, and wherein blending the first XR images with the second XR images further comprises blending the first XR images with the second XR images based on the alpha data. 
     
     
         6 . The method of  claim 1 , wherein different rendering jobs correspond to different portions of a virtual object. 
     
     
         7 . The method of  claim 1 , wherein different rendering jobs correspond to different virtual objects. 
     
     
         8 . The method of  claim 1  further comprising: generating a list of rendering jobs with the XR application instance; and
 forwarding the first rendering job to the first rendering instance and the second rendering job to the second rendering instance based on the list of rendering jobs. 
 
     
     
         9 . The method of  claim 1 , wherein the first rendering instance implements a first rendering engine, wherein the second rendering instance implements a second rendering engine, and wherein the first rendering engine is different from the second rendering engine. 
     
     
         10 . The method of  claim 1  further comprising:
 determining momentary position data, wherein the momentary position data is associated with a momentary position of the XR device; and 
 forwarding the momentary position data to the at least two rendering instances, wherein generating the first XR images and the second XR images with the first rendering instance and the second rendering instance, respectively, comprises generating the first XR images and the second XR images based on the momentary position data. 
 
     
     
         11 . The method of  claim 10 , further comprising forwarding the momentary position data to the first rendering instance and to the second rendering instance in a synchronized manner. 
     
     
         12 . The method of  claim 1 , wherein the at least two rendering instances comprise dedicated rendering hardware. 
     
     
         13 . The method of  claim 1 , wherein blending the first XR images with the second XR images with the merging instance based on the first depth data and second depth data comprises, for each pixel of the blended XR images, determining the closest of a corresponding first pixel of the first XR images and a corresponding second pixel of the second XR images based on a first distance of the XR device to a portion of a virtual object rendered for the first pixel and a second distance of the XR device to a portion of the same or another virtual object rendered for the second pixel. 
     
     
         14 . A system comprising:
 a first computer device of a plurality of computer devices communicatively coupled to and external to an extended reality (XR) device, wherein the first computer device is configured to,
 generate virtual content to be rendered with an XR application instance; 
 partition the virtual content with the XR application instance into a plurality of rendering jobs; 
 forward the plurality of rendering jobs to at least two rendering instances that are configured to render the respective rendering jobs, wherein the first computer device is configured to run a first rendering instance of the at least two rendering instances; 
 generate first XR images with the first rendering instance based on a first rendering job received from the XR application instance; and 
 stream first depth data corresponding to the first XR images to a merging instance that runs on the first computer device, wherein the first depth data comprise depth information associated with pixels of the first XR images; and 
   a second computer device of the plurality of computer devices, wherein the second computer device is configured to,
 generate second XR images with a second rendering instance of the at least two rendering instances based on a second rendering job received from the XR application instance, wherein the second XR images are rendered in parallel with the first XR images; and 
 stream the second XR images and second depth data corresponding to the second XR images to the merging instance, wherein the second depth data comprise depth information associated with pixels of the second XR images; and 
   wherein the first computer device is further configured to blend the first XR images with the second XR images with the merging instance based on the first and second depth data to obtain blended XR images to be displayed on the XR device.   
     
     
         15 . The system of  claim 14 , wherein the first and second computer devices comprise servers. 
     
     
         16 . The system of  claim 14 , wherein blending the first XR images with the second XR images with the merging instance further comprises determining the closest of a corresponding first pixel of the first XR images and a corresponding second pixel of the second XR images based on a first distance of the XR device to a portion of a virtual object rendered for the first pixel and a second distance of the XR device to a portion of the same or another virtual object rendered for the second pixel. 
     
     
         17 . The system of  claim 14 , wherein the first computer device is further configured with a first rendering engine that is utilized by the first rendering instance to render the first XR images and the second computer device is configured with a second rendering engine that is utilized by the second rendering instance to render the second XR images, wherein the first rendering engine and the second rendering engine comprise different rendering engines. 
     
     
         18 . A non-transitory machine-readable medium having program code stored thereon, the program code comprising instructions to:
 generate virtual content to be rendered with an extended reality (XR) application instance that runs on a first computer device of a plurality of computer devices communicatively coupled to and external to an XR device;   partition the virtual content to be rendered into a plurality of rendering jobs with the XR application instance;   forward the plurality of rendering jobs to at least two rendering instances that are configured to render the respective rendering jobs, wherein the at least two rendering instances comprise a first rendering instance that runs on the first computer device and a second rendering instance that runs on a second computer device of the plurality of computer devices;   generate first XR images with the first rendering instance based on a first rendering job received from the XR application instance;   generate second XR images with the second rendering instance based on a second rendering job received from the XR application instance, wherein the first XR images and the second XR images are rendered by the first rendering instance and the second rendering instance in parallel;   stream the second XR images, first depth data corresponding to the first XR images, and second depth data corresponding to the second XR images to a merging instance running on the first computer device, wherein the first depth data comprises depth information associated with pixels of the first XR images and the second depth data comprise depth information associated with pixels of the second XR images; and   blend the first XR images with the second XR images with the merging instance based on the first depth data and second depth data to obtain blended XR images to be displayed on an XR device.   
     
     
         19 . The non-transitory machine-readable medium of  claim 18 , wherein the instructions to blend the first XR images with the second XR images comprise program code to positionally blend the first XR images with the second XR images based on at least one of an orientation of the XR device and a location of the XR device. 
     
     
         20 . The non-transitory machine-readable medium of  claim 18 , wherein the instructions to blend the first XR images with the second XR images with the merging instance based on the first depth data and second depth data comprise instructions to determine the closest of a corresponding first pixel of the first XR images and a corresponding second pixel of the second XR images based on a first distance of the XR device to a portion of a virtual object rendered for the first pixel and a second distance of the XR device to a portion of the same or another virtual object rendered for the second pixel.