Multi-Aperture Depth Map Using Frequency Filtering

Abstract

Frequency filtering is used to generate depth information from multiple images of the same object. For example, two images may be captured by imaging systems with blur characteristics that vary differently as a function of object depth. For example, a dual-aperture system may simultaneously capture a faster f-number visible image and a slower f-number infrared image. Depth information may be generated by comparing blurring of one side of the same edge in the two images. Frequency filtering may be used as part of this process, for example to reduce frequency content at frequencies that are not useful to distinguish between different blur kernels.

What is claimed is: 
     
         1 . A computer-implemented method for processing blurred image data, comprising:
 accessing first image data associated with a first image of an object, the first image captured using a first imaging system;   accessing second image data associated with a second image of the object, the second image captured using a second imaging system wherein a comparison of blurring by the first imaging system and blurring by the second imaging system varies as a function of object depth;   estimating the comparison of blurring by the first and second imaging systems, comprising: frequency filtering the first and/or second image data, the frequency filtering increasing a distinction between blurring by the first imaging system and blurring by the second imaging system; and   generating depth information for the object based on said estimated comparisons.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the comparison of blurring by the first imaging system and blurring by the second imaging system is a comparison of blur spot size of the first imaging system and blur spot size of the second imaging system. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein the frequency filtering reduces content at low frequencies passed by both the first and second imaging systems. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein the frequency filtering reduces content at high frequencies passed by neither the first or second imaging systems. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein:
 estimating the comparison of blurring by the first and second imaging systems further comprises determining a blur kernel that approximates blurring of the first and second imaging systems, wherein different blur kernels correspond to different object depths; and   generating depth information for the object comprises selected the object depth that corresponds to the determined blur kernel.   
     
     
         6 . The computer-implemented method of  claim 1 , wherein:
 estimating the comparison of blurring by the first and second imaging systems further comprises:   for each blur kernel from a bank of blur kernels, wherein each blur kernel corresponds to a different object depth and the bank of blur kernels spans a range of object depths:
 blurring the second image data with the blur kernel; and 
 comparing the blurred second image data and corresponding first image data; and 
   generating depth information for the object comprises generating depth information for the object based on said comparisons.   
     
     
         7 . The computer-implemented method of  claim 6 , wherein the frequency filtering reduces content at low frequencies passed by both of two adjacent blur kernels in the bank. 
     
     
         8 . The computer-implemented method of  claim 6 , wherein the frequency filtering reduces content at high frequencies passed by neither of two adjacent blur kernels in the bank. 
     
     
         9 . The computer-implemented method of  claim 6 , wherein the frequency filtering both reduces content at low frequencies passed by both of two adjacent blur kernels in the bank and also reduces content at high frequencies passed by neither of the two adjacent blur kernels. 
     
     
         10 . The computer-implemented method of  claim 6 , wherein the frequency filtering increases a distinction of content at frequencies passed by one of two adjacent blur kernels in the bank but not passed by the other of the two adjacent blur kernels. 
     
     
         11 . The computer-implemented method of  claim 6 , wherein the frequency filtering includes frequency filtering of the second image data. 
     
     
         12 . The computer-implemented method of  claim 6 , wherein the frequency filtering includes applying the blur kernels to the second image data, the blur kernels including the frequency filtering. 
     
     
         13 . The computer-implemented method of  claim 6 , wherein the frequency filtering includes down-sampling the second image data. 
     
     
         14 . The computer-implemented method of  claim 1 , further comprising:
 selecting a plurality of first windows from the first image data;   selecting a corresponding plurality of second windows from the second image data wherein corresponding first and second windows contain a same edge in the object;   wherein estimating the comparison of blurring by the first and second imaging systems comprises, for pairs of corresponding first and second windows, estimating the comparison of blurring by the first and second imaging systems based on blurring of the same edge in corresponding first and second windows.   
     
     
         15 . The computer-implemented method of  claim 1 , wherein the first imaging system is characterized by a first f-number and the second imaging system is characterized by a second f-number that is different than the first f-number. 
     
     
         16 . The computer-implemented method of  claim 1 , wherein the first and second imaging systems are different parts of a dual-aperture imaging system, the first imaging system using a visible spectral band and characterized by a first f-number, and the second imaging system using an infrared spectral band and characterized by a second f-number that is slower than the first f-number. 
     
     
         17 . A non-transitory computer-readable storage medium storing executable computer program instructions for processing blurred image data, the instructions executable by a processor and causing the processor to perform a method comprising:
 accessing first image data associated with a first image of an object, the first image captured using a first imaging system;   accessing second image data associated with a second image of the object, the second image captured using a second imaging system wherein a comparison of blurring by the first imaging system and blurring by the second imaging system varies as a function of object depth;   estimating the comparison of blurring by the first and second imaging systems, comprising: frequency filtering the first and/or second image data, the frequency filtering increasing a distinction between blurring by the first imaging system and blurring by the second imaging system; and   generating depth information for the object based on said estimated comparisons.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , wherein the comparison of blurring by the first imaging system and blurring by the second imaging system is a comparison of frequency response of the first imaging system and frequency response of the second imaging system. 
     
     
         19 . The non-transitory computer-readable storage medium of  claim 17 , wherein the frequency filtering reduces content at low frequencies passed by both the first and second imaging systems. 
     
     
         20 . The non-transitory computer-readable storage medium of  claim 17 , wherein the frequency filtering reduces content at high frequencies passed by neither the first or second imaging systems.