Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling

Abstract

Embodiments relate to different methods for reducing computations used to estimate depth information. One aspect relates to using down-sampled blur kernels. Another aspect relates to processing of edges in the images. Yet another aspect relates to using partial blur kernels, such as single-sided blur kernels. Yet another aspect relates to frequency filtering to reduce energy and noise at frequencies that do not distinguish between different blur kernels.

1 . A method for processing blurred image data, comprising:
 downsampling first image data associated with a first image of an object, the first image captured using a first imaging system characterized by a first point spread function;   downsampling second image data associated with a second image of the object, the second image captured using a second imaging system characterized by a second point spread function that varies as a function of depth differently than the first point spread function;   for each blur kernel from a bank of down-sampled blur kernels, wherein each blur kernel corresponds to the first point spread function relative to the second point spread function at a different object depth, and the bank of blur kernels spans a range of object depths:
 blurring the down-sampled second image data with the down-sampled blur kernel; and 
 comparing the blurred down-sampled second image data and the down-sampled first image data; and 
   generating depth information for the object based on said comparisons.   
     
     
         2 . The method of  claim 1 , wherein:
 for each blur kernel, comparing the blurred down-sampled second image data and the down-sampled first image data comprises calculating an error between the blurred down-sampled second image data and the down-sampled first image data; and   generating depth information for the object based on said comparisons comprises generating depth information based on a depth that corresponds to the blur kernel with a lowest calculated error.   
     
     
         3 . The method of  claim 2 , wherein blurring the down-sampled second image data with the down-sampled blur kernel comprises:
 first deblurring the down-sampled second image data; and   then blurring the deblurred, down-sampled second image data with the down-sampled blur kernel.   
     
     
         4 . The method of  claim 2 , wherein blurring the down-sampled second image data with the down-sampled blur kernel comprises convolving the down-sampled second image data with the down-sampled blur kernel. 
     
     
         5 . The method of  claim 1 , wherein:
 the first image data and the second image data each contain a same edge;   for each blur kernel:
 blurring the down-sampled second image data with the down-sampled blur kernel comprises blurring the edge in the down-sampled second image data with the down-sampled blur kernel; and 
 comparing the blurred down-sampled second image data and the down-sampled first image data comprises comparing the blurred edge in the down-sampled second image data and the same edge in the down-sampled first image data. 
   
     
     
         6 . The method of  claim 5 , wherein blurring the edge in the down-sampled second image data comprises:
 binarizing the edge in the down-sampled second image data; and   blurring the binarized edge with the down-sampled blur kernel.   
     
     
         7 . The method of  claim 5 , wherein comparing the blurred edge in the down-sampled second image data and the same edge in the down-sampled first image data comprises phase matching the edges in the first and second image data. 
     
     
         8 . The method of  claim 5 , wherein comparing the blurred edge in the down-sampled second image data and the same edge in the down-sampled first image data comprises equating energy in the edges in the first and second image data. 
     
     
         9 . The method of  claim 1 , wherein said blurring and comparing for each blur kernel and said generating depth information is performed for each of a plurality of banks of down-sampled blur kernels, each bank down-sampled by a different downsampling factor. 
     
     
         10 . The method of  claim 9 , wherein the plurality of banks span a contiguous range of object depths. 
     
     
         11 . The method of  claim 9 , further comprising:
 classifying each bank as containing or not containing an extremum with respect to said comparison; and   generating depth information for the object based on said classifications for the bank that contains the extremum.   
     
     
         12 . The method of  claim 9 , further comprising:
 classifying each bank as monotonically increasing, monotonically decreasing or containing an extremum with respect to said comparison; and   generating depth information for the object based on said classifications for the banks.   
     
     
         13 . The method of  claim 12 , further comprising:
 if the classifications indicate an extremum occurs between two banks, then creating an additional bank that spans between the two banks.   
     
     
         14 . The method of  claim 9 , wherein each bank is down-sampled by a different integer downsampling factor. 
     
     
         15 . The method of  claim 9 , wherein a largest down-sampled blur kernel for each bank is a same size for all the banks. 
     
     
         16 . The method of  claim 9 , wherein all of the blur kernels are sufficiently down-sampled so that no down-sampled blur kernel is larger than 5×5. 
     
     
         17 . The method of  claim 1 , wherein the first imaging system has a first f-number and the second imaging system has a second f-number that is slower than the first f-number, wherein the f-number is defined as a ratio of a focal length and an effective diameter of an aperture and whereby a size of the second point spread function varies as a function of depth more slowly than a size of the first point spread function. 
     
     
         18 . The method of  claim 17 , further comprising:
 exposing an image sensor in a multi-aperture shared sensor imaging system to light from the object, using a first aperture with the first f-number to expose the first image and a second aperture with the second f-number to expose the second image.   
     
     
         19 . The method of  claim 18 , wherein the first aperture exposes the first image using light from a first spectral band, and the second aperture exposes the second image using light from a different second spectral band. 
     
     
         20 . The method of  claim 18 , wherein the first aperture exposes the first image using light from a visible spectrum, and the second aperture exposes the second image using light from an infrared spectrum. 
     
     
         21 . The method of  claim 1 , wherein the bank of down-sampled blur kernels comprises a bank of down-sampled single-sided blur kernels. 
     
     
         22 . The method of  claim 1 , further comprising:
 frequency filtering the second image data.   
     
     
         23 . A non-transitory computer-readable storage medium storing executable computer program instructions for processing blurred image data, the instructions executable by a processor and causing the processor to perform a method comprising:
 downsampling first image data associated with a first image of an object, the first image captured using a first imaging system characterized by a first point spread function;   downsampling second image data associated with a second image of the object, the second image captured using a second imaging system characterized by a second point spread function that varies as a function of depth differently than the first point spread function;   for each blur kernel from a bank of down-sampled blur kernels, wherein each blur kernel corresponds to the first point spread function of the first imaging system relative to the second point spread function at a different object depth, and the bank of blur kernels spans a range of object depths:
 blurring the down-sampled second image data with the down-sampled blur kernel; and 
 comparing the blurred down-sampled second image data and the down-sampled first image data; and 
   generating depth information for the object based on said comparisons.