Image processing apparatus and method for generating a restoration image

Abstract

The present invention relates to an image processing apparatus which can restore, from a color and sensitivity mosaic image acquired using a CCD image sensor of the single plate type or the like, a color image signal of a wide dynamic range wherein the sensitivity characteristics of pixels are uniformized and each of the pixels has all of a plurality of color components. A sensitivity uniformization section uniformizes the sensitivities of pixels of a color and sensitivity mosaic image to produce a color mosaic image, and a color interpolation section interpolates color components of the pixels of the color mosaic image M to produce output images R, G and B. The present invention can be applied to a digital camera which converts a picked up optical image into a color image signal of a wide dynamic range.

The invention claimed is as follows: 
     
       1. An image processing apparatus comprising:
 restoration means for generating a restoration image based on a color-and-sensitivity mosaic image wherein   each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light,   in terms of the sensitivity characteristics of the pixels, the pixels disposed in a same line have a same sensitivity characteristic, and the pixels disposed in different lines, which are adjacent to each other, have different sensitivity characteristics,   in terms of the color components of the pixels, the pixels having the first color component are arranged in a checker pattern irrespective of the pixels' sensitivity characteristics, the pixels having the second color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction, and the pixels having the third color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction,   in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so that each of the pixels having different sensitivity characteristics are (i) scaled to the same light intensity as the pixels having the same sensitivity characteristics to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics and (ii) compared with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components.   
     
     
       2. An image processing method comprising:
 a restoration step of generating a restoration image based on a color-and-sensitivity mosaic image wherein,   each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light,   in terms of sensitivity characteristics of the pixels, the pixels disposed in a same line have a same sensitivity characteristic, and the pixels disposed in different lines, which are adjacent to each other, have different sensitivity characteristics,   in terms of color components of the pixels, the pixels having the first color component are arranged in a checker pattern irrespective of the pixels' sensitivity characteristics, the pixels having the second color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction, and the pixels having the third color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction,   in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so that each of the pixels having different sensitivity characteristics are (i) scaled to the same light intensity as the pixels having the same sensitivity characteristics to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics and (ii) compared with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components.   
     
     
       3. An image processing apparatus comprising:
 restoration means for generating a restoration image based on a color-and-sensitivity mosaic image wherein   each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light,   the pixels having the first to third color components are arranged in a Bayer pattern with their color components,   the pixels having the first color component have different sensitivity characteristics from each other in different lines,   the pixels having the second color component are arranged so as to form a checker pattern with their sensitivity characteristics,   the pixels having the third color component are arranged so as to form a checker pattern with their sensitivity characteristics, and   in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so that each of the pixels having different sensitivity characteristics are (i) scaled to the same light intensity as the pixels having the same sensitivity characteristics to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics and (ii) compared with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components.   
     
     
       4. An image processing method comprising:
 a restoration step of generating a restoration image based on a color-and-sensitivity mosaic image wherein   each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light,   the pixels having the first to third color components are arranged in a Bayer pattern with their color components,   the pixels having the first color component have different sensitivity characteristics from each other in different lines,   the pixels having the second color component are arranged so as to form a checker pattern with their sensitivity characteristics,   the pixels having, the third color component are arranged so as to form a checker pattern with their sensitivity characteristics, and   in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so that each of the pixels having different sensitivity characteristics are (i) scaled to the same light intensity as the pixels having the same sensitivity characteristics to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics and (ii) compared with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components.   
     
     
       5. An image pickup device comprising:
 a plurality of pixels configured to sense light, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   a photo-electric conversion unit configured to produce a color and sensitivity mosaic image based on the light sensed by the plurality of pixels; and   a restoration unit configured to (i) use the color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics, (ii) use the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and (iii) interpolate the sensitivity compensated pixel information based on the discrimination information.    
     
     
       6. The image pickup device according to claim 5, wherein the plurality of color components include three color components.  
     
     
       7. The image pickup device according to claim 5, wherein the plurality of color components include red, green, and blue color components.  
     
     
       8. The image pickup device according to claim 5, wherein the plurality of color components include green and blue color components.  
     
     
       9. The image pickup device according to claim 5, wherein the plurality of color components include red and blue color components.  
     
     
       10. The image pickup device according to claim 5, wherein the plurality of color components include red and green color components.  
     
     
       11. The image pickup device according to claim 5, wherein the plurality of color components include at least a color component besides red, green and blue.  
     
     
       12. The image pickup device according to claim 5, wherein the plurality of color components include red, green, and blue color components and a fourth color component.  
     
     
       13. The image pickup device according to claim 5, wherein the plurality of sensitivity characteristics include two patterns.  
     
     
       14. The image pickup device according to claim 5, wherein the image pickup device changes between a normal mode and a high dynamic range mode.  
     
     
       15. An image pickup method comprising:
 a step for sensing light by a plurality of pixels, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   a step for producing a color and sensitivity mosaic image based on the light sensed by the plurality of pixels;   using the color and sensitivity mosaic image as an input, a step for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics;   using the color and sensitivity mosaic image as an input, a step for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics; and   a step for interpolating the sensitivity compensated pixel information based on the discrimination information.    
     
     
       16. A program stored on a non-transitory computer readable storage medium, which when executed, causes a computer to:
 sense light by a plurality of pixels, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   produce a color and sensitivity mosaic image based on the light sensed by the plurality of pixels;   use the color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics;   use the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics; and   interpolate the sensitivity compensated pixel information based on the discrimination information.    
     
     
       17. An image pickup apparatus comprising:
 a plurality of pixels configured to sense light, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   a photo-electric conversion unit configured to produce a color and sensitivity mosaic image based on the light sensed by the plurality of pixels; and   a restoration unit configured to produce a restoration image based on the color and sensitivity mosaic image by (i) using the color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics, (ii) using the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and (iii) interpolating the sensitivity compensated pixel information based on the discrimination information,   wherein the restoration image has uniform sensitivities of the pixels.    
     
     
       18. The image pickup apparatus according to claim 17, wherein the plurality of color components include three color components.  
     
     
       19. The image pickup apparatus according to claim 17, wherein the plurality of color components include red, green, and blue color components.  
     
     
       20. The image pickup apparatus according to claim 17, wherein the plurality of color components include green and blue color components.  
     
     
       21. The image pickup apparatus according to claim 17, wherein the plurality of color components include red and blue color components.  
     
     
       22. The image pickup apparatus according to claim 17, wherein the plurality of color components include red and green color components.  
     
     
       23. The image pickup apparatus according to claim 17, wherein the plurality of color components include at least a color component besides red, green and blue.  
     
     
       24. The image pickup apparatus according to claim 17, wherein the plurality of color components include red, green, and blue color components and a fourth color component.  
     
     
       25. The image pickup apparatus according to claim 17, wherein the plurality of sensitivity characteristics include two patterns.  
     
     
       26. The image pickup apparatus according to claim 17, wherein the image pickup apparatus changes between a normal mode and a high dynamic range mode.  
     
     
       27. An image pickup method comprising:
 a step for sensing light by a plurality of pixels, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   a step for producing a color and sensitivity mosaic image based on the light sensed by the plurality of pixels; and   producing a restoration image based on the color and sensitivity mosaic image by (i) using the color and sensitivity mosaic image as an input, a step for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics, (ii) using the color and sensitivity mosaic image as an input, a step for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and (iii) a step for interpolating the sensitivity compensated pixel information based on the discrimination information,   wherein the restoration image has uniform sensitivities of the pixels.    
     
     
       28. A program stored on a non-transitory computer readable storage medium, which when executed, causes a computer to:
 sense light by a plurality of pixels, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   produce a color and sensitivity mosaic image based on the light sensed by the plurality of pixels; and   produce a restoration image based on the color and sensitivity mosaic image by (i) using the color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics, (ii) using the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and (iii) interpolating the sensitivity compensated pixel information based on the discrimination information,   wherein the restoration image has uniform sensitivities of the pixels.    
     
     
       29. An image pickup apparatus comprising:
 a plurality of pixels configured to sense light, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   a photo-electric conversion unit configured to produce a color and sensitivity mosaic image based on the light sensed by the plurality of pixels; and   an image processing unit configured to produce a mosaic image based on the color and sensitivity mosaic image by (i) using the color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics, (ii) using the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and (iii) interpolating the sensitivity compensated pixel information based on the discrimination information,   wherein the mosaic image has uniform sensitivities of the pixels and each of the pixels has one of the plurality of color components.    
     
     
       30. The image pickup apparatus according to claim 29, wherein the plurality of color components include three color components.  
     
     
       31. The image pickup apparatus according to claim 29, wherein the plurality of color components include red, green, and blue color components.  
     
     
       32. The image pickup apparatus according to claim 29, wherein the plurality of color components include green and blue color components.  
     
     
       33. The image pickup apparatus according to claim 29, wherein the plurality of color components include red and blue color components.  
     
     
       34. The image pickup apparatus according to claim 29, wherein the plurality of color components include red and green color components.  
     
     
       35. The image pickup apparatus according to claim 29, wherein the plurality of color components include at least a color component besides red, green and blue.  
     
     
       36. The image pickup apparatus according to claim 29, wherein the plurality of color components include red, green, and blue color components and a fourth color component.  
     
     
       37. The image pickup apparatus according to claim 29, wherein the plurality of sensitivity characteristics include two patterns.  
     
     
       38. The image pickup apparatus according to claim 29, wherein the image pickup apparatus changes between a normal mode and a high dynamic range mode.  
     
     
       39. An image pickup method comprising:
 a step for sensing light by a plurality of pixels, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   a step for producing a color and sensitivity mosaic image based on the light sensed by the plurality of pixels; and   producing a mosaic image based on the color and sensitivity mosaic image by (i) using the color and sensitivity mosaic image as an input, a step for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics, (ii) using the color and sensitivity mosaic image as an input, a step for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and (iii) a step for interpolating the sensitivity compensated pixel information based on the discrimination information,   wherein the mosaic image has uniform sensitivities of the pixels and each of the pixels has one of the plurality of color components.    
     
     
       40. A program stored on a non-transitory computer readable storage medium, which when executed, causes a computer to:
 sense light by a plurality of pixels, each of the plurality of pixels having one of a plurality of color components and one of a plurality of sensitivity characteristics to light intensity, wherein pixels having a same color component and a same sensitivity characteristic are arranged in a lattice arrangement, and pixels having the same color component irrespective of the sensitivity characteristic are arranged in a lattice arrangement;   produce a color and sensitivity mosaic image based on the light sensed by the plurality of pixels; and   produce a mosaic image based on the color and sensitivity mosaic image by (i) using the color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics to a same light intensity as the pixels having a same sensitivity characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics, (ii) using the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of a pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and (iii) interpolating the sensitivity compensated pixel information based on the discrimination information,   wherein the mosaic image has uniform sensitivities of the pixels and each of the pixels has one of the plurality of color components.