Separation of graphics from natural video in streaming video content

Abstract

Aspects of the subject disclosure may include, for example, a method that includes obtaining, by a processing system including a processor, video frames over a network; the processing system uses a machine learning algorithm to identify in each frame a first region comprising a natural image and a second region comprising a synthetic graphic image. The processing system separates the natural image from the synthetic graphic image to generate a natural video and a graphics video, encodes the natural video, and processes the graphics video to generate instructions for rendering graphic images at a client system. The client system performs a decoding procedure for the encoded video, a rendering procedure for client-side graphics in accordance with the instructions, and a compositing procedure to obtain a presentable video stream including the natural image and a client-side graphic corresponding to the synthetic graphic image. Other embodiments are disclosed.

What is claimed is: 
     
         1 . A device, comprising:
 a processing system including a processor; and   a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
 obtaining a video stream including a plurality of frames; 
 separating, in each of the plurality of frames, a natural image from a synthetic graphic image to generate respectively a natural video and a graphics video; 
 encoding the natural video to obtain an encoded video; 
 processing the graphics video to generate instructions for use by a client system; 
 transmitting the encoded video and the instructions to the client system, wherein the client system generates a presentable video stream including the natural image and a client-side graphic corresponding to the synthetic graphic image; and 
 determining a visual similarity of the client-side graphic to the corresponding synthetic graphic image by comparing the client-side graphic to the corresponding synthetic graphic image using a comparison metric. 
   
     
     
         2 . The device of  claim 1 , wherein the separating further comprises identifying, in each of the plurality of frames, a first region comprising the natural image and a second region comprising the synthetic graphic image. 
     
     
         3 . The device of  claim 2 , wherein the identifying comprises identifying, in at least one of the plurality of frames, a third region comprising a synthetic graphic image that is partially transparent and overlies the natural image. 
     
     
         4 . The device of  claim 2 , wherein the identifying is performed using a machine learning algorithm. 
     
     
         5 . The device of  claim 4 , wherein the machine learning algorithm is trained using an automated process comprising inputting to the machine learning algorithm a first feed including video frames having both natural video images and synthetic graphics and a second feed having the natural video images without the synthetic graphics. 
     
     
         6 . The device of  claim 5 , wherein the machine learning algorithm is further trained using annotations provided via a client-side user interface. 
     
     
         7 . The device of  claim 1 , wherein the transmitting is performed via a network. 
     
     
         8 . The device of  claim 1 , wherein the client system generates the presentable video stream by performing a decoding procedure for the encoded video, a rendering procedure in accordance with the instructions, and a compositing procedure. 
     
     
         9 . The device of  claim 1 , wherein the instructions comprise instructions for hiding artifacts at an edge of the synthetic graphic image. 
     
     
         10 . The device of  claim 1 , wherein the encoding comprises a synthetic natural hybrid coding (SNHC) process. 
     
     
         11 . A method comprising:
 obtaining, by a processing system including a processor, a video stream over a network, the video stream including a plurality of frames;   separating, by the processing system in each of the plurality of frames, a natural image from a synthetic graphic image to generate respectively a natural video and a graphics video;   encoding, by the processing system, the natural video to obtain an encoded video;   processing, by the processing system, the graphics video to generate instructions for use by a client system;   transmitting, by the processing system, the encoded video and the instructions to the client system, wherein the client system generates a presentable video stream including the natural image and a client-side graphic corresponding to the synthetic graphic image; and   determining, by the processing system, a visual similarity of the client-side graphic to the corresponding synthetic graphic image by comparing the client-side graphic to the corresponding synthetic graphic image using a comparison metric.   
     
     
         12 . The method of  claim 11 , wherein the separating further comprises identifying, in each of the plurality of frames, a first region comprising the natural image and a second region comprising the synthetic graphic image. 
     
     
         13 . The method of  claim 12 , wherein the identifying is performed using a machine learning algorithm. 
     
     
         14 . The method of  claim 11 , wherein the transmitting is performed via the network. 
     
     
         15 . The method of  claim 11 , wherein the client system generates the presentable video stream by performing a decoding procedure for the encoded video, a rendering procedure in accordance with the instructions, and a compositing procedure. 
     
     
         16 . A non-transitory machine-readable medium comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
 obtaining a video stream over a network, the video stream including a plurality of frames;   separating, in each of the plurality of frames, a natural image from a synthetic graphic image to generate respectively a natural video and a graphics video;   encoding the natural video to obtain an encoded video;   processing the graphics video to generate instructions for use by a client system;   transmitting the encoded video and the instructions to the client system via the network, wherein the client system generates a presentable video stream including the natural image and a client-side graphic corresponding to the synthetic graphic image; and   determining a visual similarity of the client-side graphic to the corresponding synthetic graphic image by comparing the client-side graphic to the corresponding synthetic graphic image using a comparison metric.   
     
     
         17 . The non-transitory machine-readable medium of  claim 16 , wherein the separating further comprises identifying, in each of the plurality of frames, a first region comprising the natural image and a second region comprising the synthetic graphic image. 
     
     
         18 . The non-transitory machine-readable medium of  claim 17 , wherein the identifying is performed using a machine learning algorithm. 
     
     
         19 . The non-transitory machine-readable medium of  claim 18 , wherein the machine learning algorithm is trained using an automated process comprising inputting to the machine learning algorithm a first feed including video frames having both natural video images and synthetic graphics and a second feed having the natural video images without the synthetic graphics. 
     
     
         20 . The non-transitory machine-readable medium of  claim 16 , wherein the client system generates the presentable video stream by performing a decoding procedure for the encoded video, a rendering procedure in accordance with the instructions, and a compositing procedure.