Temporal gradient calculations in bio

Abstract

Embodiments of the present disclosure relates to improvement for temporal gradient calculating in BIO. A method for video processing is provided, including: determining a reference block for a current block; performing a refinement process for prediction of the current block, based on the reference block; and performing a conversion between the current block and a bitstream representation of a video including the current block, based on a result of the refinement process; wherein, the refinement process comprises: modifying the reference block into a modified reference block, and calculating a temporal gradient based on the modified reference block, or calculating an original temporal gradient based on the reference block, and modifying the original temporal gradient into a modified temporal gradient.

What is claimed is: 
     
       1. A method of video processing, comprising:
 determining a reference block for a current block; 
 performing a refinement process for prediction of the current block, based on the reference block; and 
 performing a conversion between the current block and a bitstream representation of a video including the current block, based on a result of the refinement process; 
 wherein, the refinement process comprises: 
 modifying the reference block into a modified reference block, and calculating a temporal gradient based on the modified reference block, or 
 calculating an original temporal gradient based on the reference block, and modifying the original temporal gradient into a modified temporal gradient, 
 wherein the step of modifying the reference block into a modified reference block comprises:
 removing a mean value for the reference block, 
 wherein the mean value is removed for all reference blocks. 
 
 
     
     
       2. The method of  claim 1 , wherein the step of removing a mean value for the reference block comprises:
 calculating the mean value for each of the reference blocks; and 
 subtracting, for each pixel in the reference block, a pixel value by the mean value. 
 
     
     
       3. The method of  claim 1 , wherein the step of modifying the reference block into a modified reference block further comprises, in prior to the removing step:
 deciding, for each of reference blocks, whether to remove a mean value or not; 
 wherein the mean value for a reference block for which it is decided to remove the mean value is removed. 
 
     
     
       4. The method of  claim 1 , wherein the step of modifying the reference block into a modified reference block further comprises, in prior to the removing step:
 deciding, for each of different reference blocks, whether the reference block is to be modified or not. 
 
     
     
       5. The method of  claim 1 , wherein the mean value is defined as an average of selected samples in the reference block, all pixels in the reference block or a sub-block of the reference block are used to calculate the mean value and partial pixels in the reference block or a sub-block of the reference block are used to calculate the mean value, and
 wherein pixels in every second row or column are used to calculate the mean value, or pixels in every fourth row or column are used to calculate the mean value, or pixels at four corners are used to calculate the mean value, or pixels at four corners and at the center are used to calculate the mean value. 
 
     
     
       6. The method of  claim 1 , further comprising, in prior to the calculating step:
 filtering the reference block to generate the modified reference block, 
 wherein the step of filtering the reference block to generate the modified reference block comprises: 
 applying a smooth filtering method to the reference block, and 
 wherein pixels at block boundaries are filtered firstly. 
 
     
     
       7. The method of  claim 6 , wherein the step of filtering the reference block to generate the modified reference block comprises at least one of:
 applying an Overlapped Block Motion Compensation (OBMC) method to the reference block; 
 applying an Illumination Compensation (IC) method to the reference block; or 
 applying a Weighted prediction method to the reference block. 
 
     
     
       8. The method of  claim 1 , wherein the step of modifying the original temporal gradient into a modified temporal gradient comprises:
 subtracting the original temporal gradient by a difference between a first mean value for a first reference block and a second mean value for a second reference block, to generate the modified temporal gradient. 
 
     
     
       9. The method of  claim 1 , wherein the refinement process comprises a bi-directional optical flow (BIO) refinement process. 
     
     
       10. The method of  claim 1 , wherein the method is applied for each sub-block, and
 wherein the current block is split to multiple sub-blocks and for each sub-block, the corresponding reference block is identified. 
 
     
     
       11. The method of  claim 1 , wherein the method is not applied in case that the current block meets a certain condition. 
     
     
       12. The method of  claim 11 , wherein the certain condition specifies at least one of: a size of the current block, a slice type of the current block, a picture type of the current block and a tile type of the current block. 
     
     
       13. The method of  claim 11 , wherein the certain condition specifies at least one of:
 a sample number that the current block contains is smaller than a first threshold; 
 minimum size of a width and a height of the current block is smaller than or no larger than a second threshold; 
 a width of the current block is smaller than or no larger than a third threshold, and/or a height of the current block is smaller than or no larger than a fourth threshold; 
 a width of the current block is larger than or no smaller than a third threshold, and/or a height of the current block is larger than or no smaller than a fourth threshold. 
 
     
     
       14. The method of  claim 13 , wherein each of the first to fifth thresholds is pre-defined or signaled in sequence parameter set (SPS) level, or picture parameter set (PPS) level, or picture level, or slice level, or tile level. 
     
     
       15. The method of  claim 14 , wherein each of the first to fifth thresholds is defined depending on coded information including at least one of a block size, a picture type, and a temporal layer index. 
     
     
       16. The method of  claim 11 , wherein the method is applied in a sub-block level, in case that a width and/or a height of a block, to which the sub-block belongs, is equal to or larger than a fifth threshold, and
 wherein the current block is split into multiple sub-blocks and each of the multiple sub-blocks further undergoes a bidirectional optical flow (BIO) process, in a same way as a normal coding block with a size equal to the sub-block size. 
 
     
     
       17. The method of  claim 1 , wherein the conversion comprises encoding the video into the bitstream representation or decoding the bitstream representation to generate pixel values of the video. 
     
     
       18. A video processing apparatus comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:
 determine a reference block for a current block; 
 perform a refinement process for prediction of the current block, based on the reference block; and 
 perform a conversion between the current block and a bitstream representation of a video including the current block, based on a result of the refinement process; 
 wherein, the refinement process comprises: 
 modifying the reference block into a modified reference block, and calculating a temporal gradient based on the modified reference block, or 
 calculating an original temporal gradient based on the reference block, and modifying the original temporal gradient into a modified temporal gradient, 
 wherein the step of modifying the reference block into a modified reference block comprises:
 removing a mean value for the reference block, 
 wherein the mean value is removed for all reference blocks. 
 
 
     
     
       19. The video processing apparatus of  claim 18 , wherein the step of removing a mean value for the reference block comprises:
 calculating the mean value for each of the reference blocks; and 
 subtracting, for each pixel in the reference block, a pixel value by the mean value. 
 
     
     
       20. A non-transitory computer-readable storage medium storing instructions that cause a processor to:
 determine a reference block for a current block; 
 perform a refinement process for prediction of the current block, based on the reference block; and 
 perform a conversion between the current block and a bitstream representation of a video including the current block, based on a result of the refinement process; 
 wherein, the refinement process comprises: 
 modifying the reference block into a modified reference block, and calculating a temporal gradient based on the modified reference block, or 
 calculating an original temporal gradient based on the reference block, and modifying the original temporal gradient into a modified temporal gradient, 
 wherein the step of modifying the reference block into a modified reference block comprises:
 removing a mean value for the reference block, 
 wherein the mean value is removed for all reference blocks.