Nonlinear, in-the-loop, denoising filter for quantization noise removal for hybrid video compression

Abstract

A method and apparatus is disclosed herein for using an in-the-loop denoising filter for quantization noise removal for video compression. In one embodiment, the video encoder comprises a transform coder to apply a transform to a residual frame representing a difference between a current frame and a first prediction, the transform coder outputting a coded differential frame as an output of the video encoder; a transform decoder to generate a reconstructed residual frame in response to the coded differential frame; a first adder to create a reconstructed frame by adding the reconstructed residual frame to the first prediction; a non-linear denoising filter to filter the reconstructed frame by deriving expectations and performing denoising operations based on the expectations; and a prediction module to generate predictions, including the first prediction, based on previously decoded frames.

1 . An image processing device comprising:
 a circuit component to supply a video frame; and   a filter coupled to the circuit component to filter the video frame by determining whether a coefficient of a linear transform of a version of the video frame should remain as is or be set to zero and performing operations, including a denoising operation, on the video frame based on the determination.   
     
     
         2 . The image processing device defined in  claim 1  further comprising a first unit to perform a first image processing operation, the output of the first unit being the video frame, and wherein the filter is a post processing filter. 
     
     
         3 . The image processing device defined in  claim 1  wherein the filter derives an expectation for each coefficient resulting from application of the linear transform to the version of the video frame. 
     
     
         4 . The image processing device defined in  claim 1  wherein the filter constructs an expectation according to a mode-based decision. 
     
     
         5 . The image processing device defined in  claim 1  wherein the filter uses one transform to obtain an estimate of the original frame. 
     
     
         6 . The image processing device defined in  claim 1  wherein the filter uses a plurality of linear transforms and combines the estimates produced from the plurality of denoising transforms to obtain an estimate of the original frame. 
     
     
         7 . The image processing device defined in  claim 6  wherein the filter combines the estimates by averaging the estimates. 
     
     
         8 . The image processing device defined in  claim 7  wherein averaging the estimates is performed using a per pixel weighted averaging. 
     
     
         9 . The image processing device defined in  claim 1  wherein the filter determines the filtering to perform on each coefficient based on a compression mode. 
     
     
         10 . The image processing device defined in  claim 1  wherein the filter uses mode-based, per coefficient threshold detection to determine whether to filter coefficients. 
     
     
         11 . The image processing device defined in  claim 1  wherein the filter estimates each of a plurality of coefficients of the original frame by selecting between two values, wherein the selecting is based on whether the means square error of using one value is lower than the mean square error of using the other value. 
     
     
         12 . The image processing device defined in  claim 11  wherein the mean square error associated with using a predetermined constant is less than the mean square error of using a corresponding coefficient of the reconstructed frame. 
     
     
         13 . The image processing device defined in  claim 1  wherein the filter performs denoising on a selected subset of pixels. 
     
     
         14 . The image processing device defined in  claim 13  wherein the subset of pixels is determined by defining a mask using compression mode parameters of the reconstructed frame. 
     
     
         15 . The image processing device defined in  claim 1  wherein the filter alters denoising parameters of each coefficient. 
     
     
         16 . The image processing device defined in  claim 15  wherein altering the denoising parameters for each coefficient is based on compression mode parameters. 
     
     
         17 . The image processing device defined in  claim 1  wherein the filter performs per coefficient weighting when performing denoising to generate an estimate of the original frame. 
     
     
         18 . The image processing device defined in  claim 1  wherein the filter performs weighted averaging in the transform domain to generate an estimate of the original frame. 
     
     
         19 . An image processing process comprising:
 receiving a video frame; and   filtering the video frame by determining whether a coefficient of a linear transform of a version of the video frame should remain as is or be set to zero performing operations, including a denoising operation, on the video frame based on the determination.   
     
     
         20 . The image processing process defined in  claim 19  wherein filtering the video frame comprises:
 creating an expectation for each coefficient resulting from application of the linear transform to a quantized video frame; and 
 performing denoising operations based on the expectation. 
 
     
     
         21 . The image processing process defined in  claim 20  further comprising refining expectation according to a mode-based decision. 
     
     
         22 . The image processing process defined in  claim 19  further comprising:
 applying a plurality of denoising transforms to video data to generate a plurality of estimates of the original frame; and 
 combining the estimates produced from the plurality of denoising transforms to obtain an overall estimate of the original frame. 
 
     
     
         23 . The image processing process defined in  claim 19  wherein filtering the video frame comprises determining the filtering to perform on each coefficient based on compression mode. 
     
     
         24 . The image processing process defined in  claim 19  wherein filtering the video frame comprises estimates each of a plurality of coefficients of the original frame by selecting between two values, wherein the selecting is based on whether the means square error of using one value being lower than the mean square error of using the other value. 
     
     
         25 . The image processing process defined in  claim 19  wherein filtering the video frame comprises determining whether the mean square error associated with using a predetermined constant is less than the mean square error of using a corresponding coefficient of the reconstructed frame. 
     
     
         26 . The image processing process defined in  claim 19  wherein filtering the video frame occurs on a selected subset of pixels. 
     
     
         27 . The image processing process defined in  claim 26  wherein the subset of pixels is determined by defining a mask using compression mode parameters of an quantized frame. 
     
     
         28 . The image processing process defined in  claim 19  further comprising altering denoising parameters for each coefficient based on compression mode parameters.