Reducing effects of light diffraction in under display camera (udc) systems

Abstract

A method includes obtaining, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern. The first diffraction pattern and the second diffraction pattern are related through a transformation. The method also includes generating a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames. The method further includes combining the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.

What is claimed is: 
     
         1 . A method comprising:
 obtaining, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation;   generating a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and   combining the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.   
     
     
         2 . The method of  claim 1 , wherein combining the first and second deblurred images comprises:
 performing an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and   merging the aligned versions of the first and second deblurred images.   
     
     
         3 . The method of  claim 2 , wherein performing the alignment operation comprises at least one of:
 performing a global registration operation in order to globally align the first deblurred image and the second deblurred image; and   performing a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.   
     
     
         4 . The method of  claim 3 , wherein at least one of:
 the global registration operation comprises a feature extraction operation and a feature matching operation; and   the local alignment operation comprises a non-rigid warping operation.   
     
     
         5 . The method of  claim 2 , wherein merging the aligned versions of the first and second deblurred images comprises:
 merging the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.   
     
     
         6 . The method of  claim 5 , wherein the at least one merging operation comprises at least one of:
 an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio;   a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and   a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.   
     
     
         7 . The method of  claim 1 , wherein:
 the first diffraction pattern is associated with a first pattern of a display under which the at least one under display camera is positioned;   the second diffraction pattern is associated with a second pattern of the display; and   the second diffraction pattern is obtained by at least one of:
 rotating the at least one under display camera; 
 rotating at least a portion of the display relative to the at least one under display camera; 
 integrating different grid patterns into different portions of the display; and 
 rotating an electronic device that includes the display and the at least one under display camera. 
   
     
     
         8 . An electronic device comprising:
 a display;   at least one under display camera positioned under the display; and   at least one processing device configured to:
 obtain, using the at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation; 
 generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and 
 combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene. 
   
     
     
         9 . The electronic device of  claim 8 , wherein, to combine the first and second deblurred images, the at least one processing device is configured to:
 perform an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and   merge the aligned versions of the first and second deblurred images.   
     
     
         10 . The electronic device of  claim 9 , wherein, to perform the alignment operation, the at least one processing device is configured to at least one of:
 perform a global registration operation in order to globally align the first deblurred image and the second deblurred image; and   perform a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.   
     
     
         11 . The electronic device of  claim 10 , wherein at least one of:
 the global registration operation comprises a feature extraction operation and a feature matching operation; and   the local alignment operation comprises a non-rigid warping operation.   
     
     
         12 . The electronic device of  claim 9 , wherein, to merge the aligned versions of the first and second deblurred images, the at least one processing device is configured to merge the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images. 
     
     
         13 . The electronic device of  claim 12 , wherein the at least one merging operation comprises at least one of:
 an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio;   a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and   a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.   
     
     
         14 . The electronic device of  claim 8 , wherein:
 the first diffraction pattern is associated with a first pattern of the display;   the second diffraction pattern is associated with a second pattern of the display; and   the second diffraction pattern is obtained by at least one of:
 rotation of the at least one under display camera; 
 rotation of at least a portion of the display relative to the at least one under display camera; 
 integration of different grid patterns into different portions of the display; and 
 rotation of the electronic device. 
   
     
     
         15 . A non-transitory machine readable medium containing instructions that when executed cause at least one processor of an electronic device to:
 obtain, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation;   generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and   combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.   
     
     
         16 . The non-transitory machine readable medium of  claim 15 , wherein the instructions that when executed cause the at least one processor to combine the first and second deblurred images comprise:
 instructions that when executed cause the at least one processor to:
 perform an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and 
 merge the aligned versions of the first and second deblurred images. 
   
     
     
         17 . The non-transitory machine readable medium of  claim 16 , wherein the instructions that when executed cause the at least one processor to perform the alignment operation comprise:
 instructions that when executed cause the at least one processor to at least one of:
 perform a global registration operation in order to globally align the first deblurred image and the second deblurred image; and 
 perform a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image. 
   
     
     
         18 . The non-transitory machine readable medium of  claim 17 , wherein at least one of:
 the global registration operation comprises a feature extraction operation and a feature matching operation; and   the local alignment operation comprises a non-rigid warping operation.   
     
     
         19 . The non-transitory machine readable medium of  claim 16 , wherein the instructions that when executed cause the at least one processor to merge the aligned versions of the first and second deblurred images comprise:
 instructions that when executed cause the at least one processor to merge the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.   
     
     
         20 . The non-transitory machine readable medium of  claim 19 , wherein the at least one merging operation comprises at least one of:
 an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio;   a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and   a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.