Efficient Motion-Compensated Spatiotemporal Sampling

Abstract

In one embodiment, a computing system may access a video including a first frame and a second frame. The computing system may determine first sampling locations for the first frame and determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame. The computing system may detect one or more invalid second sampling locations based on determining pixels in the first frame corresponding to the first sampling locations do not match pixels in the second frame corresponding to the second sampling locations. The computing system may reject the one or more invalid second sampling locations to determine third sampling locations for the second frame. The computing system may generate a sample of the video.

1 - 20 . (canceled) 
     
     
         21 . A method comprising, by a computing system:
 accessing a video comprising a first frame and a second frame;   determining first sampling locations for the first frame;   determining second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame;   detecting one or more invalid second sampling locations based on determining pixels in the first frame corresponding to the first sampling locations do not match pixels in the second frame corresponding to the second sampling locations;   rejecting the one or more invalid second sampling locations to determine third sampling locations for the second frame; and   generating a sample of the video based on the pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the third sampling locations.   
     
     
         22 . The method of  claim 21 , wherein determining the first sampling locations for the first frame comprises generating a sample of the first frame using a two-dimensional mask comprising a blue noise property. 
     
     
         23 . The method of  claim 22 , wherein generating the sample of the first frame using the two-dimensional mask comprises using a threshold percentage of sampling points of the two-dimensional mask. 
     
     
         24 . The method of  claim 21 , wherein determining the second sampling locations for the second frame by transforming the first sampling locations to the second frame further comprises using a color temporal gradient between the first frame and the second frame. 
     
     
         25 . The method of  claim 21 , further comprising:
 selecting a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations;   defining one or more rejection areas in the second frame based on the subset of the second sampling locations; and   determining fourth sampling locations for the second frame in areas outside of the one or more rejection areas, wherein generating the sample of the video is further based on the fourth sampling locations.   
     
     
         26 . The method of  claim 25 , wherein the one or more rejection areas in the second frame are areas of a fixed radius extending from the subset of the second sampling locations. 
     
     
         27 . The method of  claim 25 , wherein determining fourth sampling locations for the second frame comprises generating a sample of the second frame in areas outside of the one or more rejection areas using a two-dimensional mask comprising a blue noise property. 
     
     
         28 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
 access a video comprising a first frame and a second frame;   determine first sampling locations for the first frame;   determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame;   detect one or more invalid second sampling locations based on determining pixels in the first frame corresponding to the first sampling locations do not match pixels in the second frame corresponding to the second sampling locations;   reject the one or more invalid second sampling locations to determine third sampling locations for the second frame; and   generate a sample of the video based on the pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the third sampling locations.   
     
     
         29 . The media of  claim 28 , wherein determining the first sampling locations for the first frame comprises generating a sample of the first frame using a two-dimensional mask comprising a blue noise property. 
     
     
         30 . The media of  claim 29 , wherein generating the sample of the first frame using the two-dimensional mask comprises using a threshold percentage of sampling points of the two-dimensional mask. 
     
     
         31 . The media of  claim 28 , wherein determining the second sampling locations for the second frame by transforming the first sampling locations to the second frame further comprises using a color temporal gradient between the first frame and the second frame. 
     
     
         32 . The media of  claim 28 , wherein the software is further operable when executed to:
 select a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations;   define one or more rejection areas in the second frame based on the subset of the second sampling locations; and   determine fourth sampling locations for the second frame in areas outside of the one or more rejection areas, wherein generating the sample of the video is further based on the fourth sampling locations.   
     
     
         33 . The media of  claim 32 , wherein the one or more rejection areas in the second frame are areas of a fixed radius extending from the subset of the second sampling locations. 
     
     
         34 . The media of  claim 32 , wherein determining third sampling locations for the second frame comprises generating a sample of the second frame in areas outside of the one or more rejection areas using a two-dimensional mask comprising a blue noise property. 
     
     
         35 . A system comprising:
 one or more processors; and   one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
 access a video comprising a first frame and a second frame; 
 determine first sampling locations for the first frame; 
 determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame; 
 detect one or more invalid second sampling locations based on determining pixels in the first frame corresponding to the first sampling locations do not match pixels in the second frame corresponding to the second sampling locations; 
 reject the one or more invalid second sampling locations to determine third sampling locations for the second frame; and 
 generate a sample of the video based on the pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the third sampling locations. 
   
     
     
         36 . The system of  claim 35 , wherein determining the first sampling locations for the first frame comprises generating a sample of the first frame using a two-dimensional mask comprising a blue noise property. 
     
     
         37 . The system of  claim 36 , wherein generating the sample of the first frame using the two-dimensional mask comprises using a threshold percentage of sampling points of the two-dimensional mask. 
     
     
         38 . The system of  claim 35 , wherein determining the second sampling locations for the second frame by transforming the first sampling locations to the second frame further comprises using a color temporal gradient between the first frame and the second frame. 
     
     
         39 . The system of  claim 35 , wherein the processors are further operable when executing the instructions to:
 select a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations;   define one or more rejection areas in the second frame based on the subset of the second sampling locations; and   determine fourth sampling locations for the second frame in areas outside of the one or more rejection areas, wherein generating the sample of the video is further based on the fourth sampling locations.   
     
     
         40 . The system of  claim 39 , wherein the one or more rejection areas in the second frame are areas of a fixed radius extending from the subset of the second sampling locations.