Decoding and Synthesizing Frames for Incomplete Video Data

Abstract

Embodiments are directed to decoding incomplete video data. In one scenario, a computer system receives a piece of a frame (e.g. a video frame). The frame includes multiple frame pieces, where each frame piece is encoded so as to be individually decodable. The computer system decodes the received frame pieces while waiting for at least one additional frame piece to be received. Then, when a condition occurs indicating that the frame is incomplete, the computer system synthesizes frame pieces to replace the additional frame piece that has not yet been received.

We claim: 
     
         1 . At a computer system including at least one processor and a memory, a computer-implemented method for displaying incomplete video data, the method comprising:
 displaying a corrupted version of a first frame, the corrupted version of the first frame including the at least one first portion and a replacement portion;   sending a notification to a second computing system to trigger a refresh of the first frame to receive a second frame;   receiving the second frame from the second computing system; and   using the second frame in place of the corrupted version of the first frame in one or more later frames that use the first frame as a reference frame.   
     
     
         2 . The method of  claim 1 , further comprising synthesizing the replacement portion as a result of determining that at least one second portion of the first frame is lost or delayed. 
     
     
         3 . The method of  claim 1 , further comprising combining the at least one first portion of the first frame and one or more synthesized frame pieces to form a frame. 
     
     
         4 . The method of  claim 1 , further comprising:
 determining a portion of the first frame includes forward error correction data; and   implementing the forward error correction data to generate a frame.   
     
     
         5 . The method of  claim 1 , further comprising waiting for forward error correction information for the first frame, while continuing to decode frame portions that are complete. 
     
     
         6 . The method of  claim 5 , further comprising:
 receiving forward error correction information for the first frame; and   decoding a frame portion to which the forward error correction information corresponds.   
     
     
         9 . The method of  claim 1 , further comprising synthesizing the replacement portion as a result of an indication that a specified minimum threshold number of frame portions has been received. 
     
     
         10 . The method of  claim 1 , wherein the replacement portion of the first frame is synthesized using frame data associated with at least one previous frame. 
     
     
         11 . A computer system comprising the following:
 one or more processors;   system memory; and   one or more computer-readable storage media having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing system to perform a method for decoding incomplete video data, the method comprising the following:
 display a corrupted version of the first frame, the corrupted version of the first frame including the at least one first portion and the replacement portion; 
 send a notification to a second computing system to trigger a refresh of the first frame to receive a second frame; 
 receive the second frame from the second computing system; and 
 use the second frame in place of the corrupted version of the first frame in one or more later frames that use the first frame as a reference frame. 
   
     
     
         12 . The computer system of  claim 11 , wherein the one or more computer-readable storage media further have stored thereon instructions that are executable by the one or more processors to configure the computer system to synthesize replacement portion by using at least a portion of information indicating prior movement between frames. 
     
     
         13 . The computer system of  claim 11 , wherein the one or more computer-readable storage media further have stored thereon instructions that are executable by the one or more processors to configure the computer system to synthesize the replacement portion by determining which direction a camera is panning based on one or more previous frames, and to synthesize the replacement portion using the determined panning direction. 
     
     
         14 . The computer system of  claim 11 , wherein the one or more computer-readable storage media further have stored thereon instructions that are executable by the one or more processors to configure the computer system to synthesize the replacement portion by using frame data indicating motion vectors. 
     
     
         15 . The computer system of  claim 11 , wherein the one or more computer-readable storage media further have stored thereon instructions that are executable by the one or more processors to configure the computer system to synthesize the replacement portion as a result of receiving at least one portion of a subsequent frame. 
     
     
         16 . The computer system of  claim 15 , wherein the one or more computer-readable storage media further have stored thereon instructions that are executable by the one or more processors to configure the computer system to track frame portion data packets across a plurality of frames to determine when data packets are received for the subsequent frame. 
     
     
         17 . The computer system of  claim 15 , wherein the one or more computer-readable storage media further have stored thereon instructions that are executable by the one or more processors to configure the computer system to use timestamp information to associate frame portion data packets with frame portions and with frames. 
     
     
         18 . A computer program product for implementing a method for decoding incomplete video data, the computer program product comprising one or more computer-readable storage media having stored thereon computer-executable instructions that, when executed by one or more processors of the computing system, cause the computing system to perform the method, the method comprising:
 displaying a corrupted version of the first frame, the corrupted version of the first frame including the at least one first portion and the replacement portion;   sending a notification to a second computing system to trigger a refresh of the first frame to receive a second frame;   receiving the second frame from the second computing system; and   using the second frame in place of the corrupted version of the first frame in one or more later frames that use the first frame as a reference frame.   
     
     
         19 . The computer program product of  claim 18 , wherein frames and frame portions are tracked separately using at least one of a protocol-based sequence number and an incrementally-increasing timestamp. 
     
     
         20 . The computer program product of  claim 18 , wherein the frames are received as part of a streamed video game.