Error concealment for frame loss in multiple description coding

Abstract

Systems and methodologies for concealing frame loss in a video transmission environment are provided herein. Multiple Description Coding (MDC) can be used as an Error Resilience technique for video coding. In case of transmission errors, Error Concealment can be combined with MDC to reconstruct a lost frame, such that the propagated error to following frames can be reduced. Further, multi-hypothesis decoding can be employed to enhance reconstructed video quality of MDC over packet loss networks. For instance, one or more frames after the lost frame in the same stream can be reconstructed using multi-hypothesis decoding, which combines directly decoding and temporally interpolating these frames. Moreover, output obtained from directly decoding and temporally interpolating each frame can be combined by generating a weighted sum of these hypotheses. Constant weights and/or adaptive weights (e.g., determined based on the minimum mean square error criterion) can be used for yielding the weighted sum.

1 . A system, comprising:
 a decoder configured to perform a decoding of a plurality of streams, wherein the decoding includes a first reconstruction of a first frame in a first stream of the plurality of streams, corrupted during a transmission of the first stream, with error concealment, and perform a multi-hypothesis decoding including a direct decoding of at least one frame subsequent to the first frame, wherein the at least one frame subsequent to the first frame directly or indirectly references the first frame; and   a memory, communicatively coupled to the decoder, that stores an output from the decoding.   
     
     
         2 . The system of  claim 1 , wherein the multi-hypothesis decoding comprises a second reconstruction of the at least one frame subsequent to the first frame in the first stream, wherein the second reconstruction is based at least in part on a temporal interpolation of the first frame and as a function of frames included in a second stream of the plurality of streams. 
     
     
         3 . The system of  claim 1 , wherein the decoder is further configured to reduce error propagation by a determination of concealment of the at least one frame subsequent to the first frame as an additional hypothesis used for the multi-hypothesis decoding. 
     
     
         4 . The system of  claim 1 , further comprising a frame loss detection component configured to identify a transmission error associated with the first frame corrupted during the transmission. 
     
     
         5 . The system of  claim 1 , further comprising a lost frame interpolation component configured to perform the error concealment for the first frame based at least in part on temporal interpolation. 
     
     
         6 . The system of  claim 5 , wherein the lost frame interpolation component is further configured to employ at least one of unidirectional motion compensated temporal interpolation overlapped motion compensated temporal interpolation, modified fast motion compensated temporal interpolation, multiframe optical flow estimation, or three-loop temporal interpolation on the first frame. 
     
     
         7 . The system of  claim 2 , wherein the decoder is further configured to perform the multi-hypothesis decoding in part by generation of a weighted sum of outputs from the direct decoding and the second reconstruction. 
     
     
         8 . The system of  claim 7 , wherein the decoder is further configured to perform the multi-hypothesis decoding in part by generation of the weighted sum of outputs based on constant weights. 
     
     
         9 . The system of  claim 7 , wherein the decoder is further configured to perform the multi-hypothesis decoding in part by adaptive determination of weights to combine outputs from the direct decode component and outputs from the concealment component based on a minimum mean square error criterion. 
     
     
         10 . The system of  claim 1 , further comprising a render component configured to generate a display based upon video frames from the decoded plurality of streams. 
     
     
         11 . The system of  claim 1 , wherein the plurality of streams are encoded using multiple description coding with odd or even sub-sampling. 
     
     
         12 . The system of  claim 1 , wherein the decoder component is further configured to perform the multi-hypothesis decoding in part by reconstruction of four, five, or six frames subsequent to the first frame in the first stream. 
     
     
         13 . An apparatus, comprising:
 a memory storing computer-executable instructions; and   a processor, communicatively coupled to a memory, that facilitates execution of the computer-executable instructions to at least:
 reconstruct a first frame in a first stream of multiple video streams corrupted during transmission with a temporal interpolation using frames surrounding the first frame included in a second stream; and 
 reconstruct at least one frame subsequent to the first frame in the first stream of the multiple video streams with a multi-hypothesis decoding that is based at least in part on the temporal interpolation of the first frame and as a function of frames included in the second stream of the multiple video streams. 
   
     
     
         14 . The apparatus of  claim 13 , wherein the processor further facilitates the execution of the computer-executable instructions to directly decode the at least one frame subsequent to the first frame, wherein the at least one frame directly or indirectly references the first frame. 
     
     
         15 . The apparatus of  claim 14 , wherein the processor further facilitates the execution of the computer-executable instructions to generate a weighted sum of outputs from the direct decoding of the at least one frame subsequent to the first frame and the temporal interpolation of the first frame with the frames surrounding the first frame and included in the second stream. 
     
     
         16 . A method, comprising:
 reconstructing, by a system including a processor, a first frame in a first stream of video streams corrupted during transmission with error concealment; and   reconstructing one or more frames subsequent to the first frame in the first stream of video streams with multi-hypothesis decoding that is based at least in part on temporal interpolation of the first frame using frames surrounding the first frame and included in a second stream of the video frames.   
     
     
         17 . The method of  claim 16 , wherein the multi-hypothesis decoding comprises:
 directly decoding a second frame in the second stream that directly or indirectly references the first frame in the first stream;   interpolating the second frame based on at least one frame surrounding the second frame included in the second stream; and   reconstructing the second frame by generating a weighted sum of the directly decoded second frame and the interpolated second frame as a function of weights.   
     
     
         18 . The method of  claim 17 , wherein the reconstructing the second frame includes reconstructing the second frame by generating the weighted sum of the directly decoded second frame and the interpolated second frame using a constant weight. 
     
     
         19 . The method of  claim 17 , wherein the multi-hypothesis decoding further comprises:
 determining adaptive weights corresponding to the directly decoded second frame and the interpolated second frame based upon a minimum mean square error criterion,   wherein the reconstructing the second frame includes reconstructing the second frame by generating the weighted sum of the directly decoded second frame and the interpolated second frame based on the adaptive weights.   
     
     
         20 . A computer readable storage medium comprising computer executable instructions that, in response to execution, cause at least one computing device including at least one processor to perform operations, comprising:
 reconstructing a first frame of a first stream of at least two video streams corrupted during transmission with error concealment; and   reconstructing one or more frames subsequent to the first frame with multi-hypothesis decoding that is based at least in part on temporal interpolation and as a function of frames included in a second stream of the at least two video streams.   
     
     
         21 . The computer readable storage medium of  claim 20 , wherein the operations further comprise:
 identifying an error associated with the first frame,   wherein the multi-hypothesis decoding comprises directly decoding a second frame of the second stream that directly or indirectly depends from the first frame included in the first stream and interpolating the second frame based on frames surrounding the second frame.