Methods and systems motion vector calculation and processing

Abstract

Systems and method are herein provided for motion vector calculation, processing, and filtering. In one example, a method comprises receiving a plurality of image frames as inputs; calculating one or more internal motion vector (MV) fields between a past frame (PF) and a current frame (CF) of the plurality of image frames; generating a foreground MV field and a background MV field of the one or more MV fields; processing the one or more internal MV fields and the foreground and background MV fields to generate one or more depth fields, wherein processing the one or more internal MV fields comprises generating virtual depths of the one or more internal MV fields to generate one or more MVD fields; and outputting the one or more MVD fields for image processing.

1 . A method, comprising:
 receiving a plurality of image frames as inputs;   calculating one or more internal motion vector (MV) fields between a past frame (PF) and a current frame (CF) of the plurality of image frames;   generating a foreground MV field and a background MV field of the one or more MV fields;   processing the one or more internal MV fields and the foreground and background MV fields to generate one or more depth fields, wherein processing the one or more internal MV fields comprises generating virtual depths of the one or more internal MV fields to generate one or more MVD fields; and   outputting the one or more MVD fields for image processing.   
     
     
         2 . The method of  claim 1 , wherein one or more internal MV fields between the PF and the CF comprise an internal phase 0 MV field of the PF and an internal phase 1 MV field of the CF. 
     
     
         3 . The method of  claim 1 , wherein the one or more internal MV fields are calculated at a block level wherein each block comprises a group of pixels. 
     
     
         4 . The method of  claim 3 , wherein processing the one or more internal MV fields comprises decomposing the one or more internal MV fields from block level to pixel level. 
     
     
         5 . The method of  claim 1 , wherein the plurality of image frames are luma image frames. 
     
     
         6 . The method of  claim 1 , wherein the plurality of image frames are modified according to pixel value, wherein pixel value indicates whether a given pixel belongs to object, special effect, or other. 
     
     
         7 . The method of  claim 1 , wherein the one or more MV fields are internally generated and are combined with one or more externally obtained MV fields to generate the one or more MVD fields. 
     
     
         8 . The method of  claim 1 , wherein detecting foreground and background MVs comprises:
 determining potential background MVs and foreground MVs via MV projection;   determining reliability of each of the potential background and foreground MVs; and   accumulating reliable background and foreground MVs.   
     
     
         9 . The method of  claim 1 , wherein generating virtual depths comprises determining a virtual depth for each respective frame based on an initial depth value, a first difference in MVs between a current MV field and a regional foreground MV field, and a second difference in MVs between a current MV field and a regional background MV field. 
     
     
         10 . A system, comprising:
 one or more processors and non-transitory memory communicably coupled to a game engine, wherein the memory stores instructions executable by the one or more processors that, when executed, cause the processors to:   generate input image frames including a past frame (PF) and a current frame (CF) each comprising a plurality of pixels divided into a plurality of blocks;   calculate block level motion vector (MV) fields between the PF and the CF;   generate a block level virtual depth for each of the block level MV fields;   decompose the block level MV fields and block level virtual depths into pixel level MV fields and pixel level virtual depths; and   output the pixel level MV fields and pixel level virtual depths for image processing.   
     
     
         11 . The system of  claim 10 , wherein calculating block level MVs between the PF and the CF comprises detecting foreground MVs and background MVs, wherein detecting foreground MVs and background MVs comprises using MVs related to a past of past frame (PPF), the PF, and the CF. 
     
     
         12 . The system of  claim 11 , wherein the imaging processing comprises one or more of frame interpolation, extrapolation, and reprojection. 
     
     
         13 . The system of  claim 10 , wherein the block level MVs between the PF and the CF comprise one or more block level phase 0 MVs of the PF and one or more block level phase 1 MVs of the CF. 
     
     
         14 . A method, comprising:
 calculating a motion vector (MV) of a first block between a past frame (PF) and a current frame (CF) of an input image, wherein the input image comprises a plurality of pixels partitioned into a plurality of blocks   determining potential foreground MVs and potential background MVs for the first block;   determining reliable foreground MVs and reliable background MVs of the potential foreground and background MVs for a corresponding region;   generating a virtual depth of the first block based at least in part on the foreground and background MVs of the corresponding region;   decomposing the MV of the first block into a plurality of MVs each of a second, smaller block; and   outputting the plurality of MVs for image processing.   
     
     
         15 . The method of  claim 14 , further comprising calculating a MV of one or more second, smaller blocks between the PF and the CF to generate an MV field. 
     
     
         16 . The method of  claim 15 , further comprising using regional and global foreground MVs and regional and global background MVs to replace unreliable MVs. 
     
     
         17 . The method of  claim 14 , wherein image processing comprises at least one of frame interpolation, extrapolation, and reprojection. 
     
     
         18 . The method of  claim 14 , wherein decomposing the MV of the first block comprises determining a weight of the first block and applying one of a bilateral filter and a guided filter based on the weight of the first block. 
     
     
         19 . The method of  claim 18 , wherein the guided filter finds a best fit between the MV of the first block and guide data to generate the MVs of the second, smaller blocks. 
     
     
         20 . The method of  claim 18 , wherein determining the weight of the first block comprises determining a product of 1) weight of a brightness of the first block, 2) weight of spatial of the first block, and 3) weight of object mask of the first block.