Image processing apparatus and method for generating a restoration image

Abstract

The present invention relates to anAn image processing apparatus which can restore, from a color and sensitivity mosaic image acquired using a CCD image sensor of the single plate type or the like, a color image signal of a wide dynamic range wherein the sensitivity characteristics of pixels are uniformized and each of the pixels has all of a plurality of color components is provided. A sensitivity uniformization section uniformizes the sensitivities of pixels of a color and sensitivity mosaic image to produce a color mosaic image, and a color interpolation section interpolates color components of the pixels of the color mosaic image M to produce output images R, G and B. The present inventionimage processing apparatus can be applied to a digital camera which converts a picked up optical image into a color image signal of a wide dynamic range.

The invention claimed is as follows: 
     
       1. An image processing apparatus comprising:
 restoration means for generating a restoration image based on a color-and-sensitivity mosaic image wherein (a) each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light, (b) in terms of the sensitivity characteristics of the pixels, the pixels disposed in a same line have a same sensitivity characteristic, and the pixels disposed in different lines, which are adjacent to each other, have different sensitivity characteristics, and (c) in terms of the color components of the pixels, the pixels having the first color component are arranged in a checker pattern irrespective of the pixels' sensitivity characteristics, the pixels having the second color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction, and the pixels having the third color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction, 
 the restoration means including:  
 in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so thatsensitivity compensation means using a color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics are (i) scaled to theto a same light intensity as the pixels having thea same sensitivity characteristicscharacteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics; 
 and (ii) comparedvalidity discrimination means using the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components; and 
 interpolation means for interpolating the sensitivity compensated pixel information based on the discrimination information to produce the restoration image. 
 
     
     
       2. An image processing method comprising:
 a restoration step of generating a restoration image based ona step for providing a color-and-sensitivity mosaic image wherein, (a) each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light, (b) in terms of sensitivity characteristics of the pixels, the pixels disposed in a same line have a same sensitivity characteristic, and the pixels disposed in different lines, which are adjacent to each other, have different sensitivity characteristics, and (c) in terms of color components of the pixels, the pixels having the first color component are arranged in a checker pattern irrespective of the pixels' sensitivity characteristics, the pixels having the second color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction, and the pixels having the third color component are arranged so as to be disposed in adjacent different lines and adjacent in a diagonal direction,;  
 in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so thatusing a color and sensitivity mosaic image as an input, a step for scaling each of the pixels having different sensitivity characteristics are (i) scaled to the a same light intensity as the pixels having the a same sensitivity characteristics characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics;  
 and (ii) comparedusing the color and sensitivity mosaic image as an input, a step for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components; and 
 a step for interpolating the sensitivity compensated pixel information based on the discrimination information to produce a restoration image. 
 
     
     
       3. An image processing apparatus comprising:
 restoration means for generating a restoration image based on a color-and-sensitivity mosaic image wherein (a) each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light, (b) the pixels having the first to third color components are arranged in a Bayer pattern with their respective color components, and (c) the pixels having the first color component have different sensitivity characteristics from each other in different lines, the pixels having the second color component are arranged so as to form a checker pattern with their sensitivity characteristics have different sensitivity characteristics from each other in different lines, and the pixels having the third color component are arranged so as to form a checker pattern with their respective sensitivity characteristics, and 
 the restoration means including:  
 in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so that sensitivity compensation means using a color and sensitivity mosaic image as an input for scaling each of the pixels having different sensitivity characteristics are (i) scaled to the a same light intensity as the pixels having the a same sensitivity characteristics characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics;  
 and (ii) comparedvalidity discrimination means using the color and sensitivity mosaic image as an input for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components; and 
 interpolation means for interpolating the sensitivity compensated pixel information based on the discrimination information to produce the restoration image. 
 
     
     
       4. An image processing method comprising:
 a restoration step of generating a restoration image based ona step for providing a color-and-sensitivity mosaic image wherein (a) each of a plurality of pixels has one of first to third color components and one of a plurality of sensitivity characteristics with respect to intensity of light, (b) the pixels having the first to third color components are arranged in a Bayer pattern with their color components, and (c) the pixels having the first color component have different sensitivity characteristics from each other in different lines, the pixels having the second color component are arranged so as to form a checker pattern with their sensitivity characteristics have different sensitivity characteristics from each other in different lines, and the pixels having, the third color component are arranged so as to form a checker pattern with their respective sensitivity characteristics, and;  
 in the restoration image, for each color component, the sensitivities of the pixels are uniformized, using as inputs (i) color mosaic pattern information, (ii) a color and sensitivity mosaic image, and (iii) sensitivity mosaic pattern information, so thatusing a color and sensitivity mosaic image as an input, a step for scaling each of the pixels having different sensitivity characteristics are (i) scaled to the a same light intensity as the pixels having the a same sensitivity characteristics characteristic to create sensitivity compensated pixel information for each of the pixels having different sensitivity characteristics;  
 and (ii) comparedusing the color and sensitivity mosaic image as an input, a step for comparing each of the pixels having different sensitivity characteristics with a threshold value to discriminate the validity of the pixel value to create discrimination information for each of the pixels having different sensitivity characteristics, and each of the uniformized pixels has all of the plurality of color components; and 
 a step for interpolating the sensitivity compensated pixel information based on the discrimination information to produce a restoration image.