Noise Models for Image Processing

Abstract

A plurality of images of a scene may be obtained. These images may have been captured by an image sensor, and may include a first image and a second image. A particular gain may have been applied to the first image. An effective color temperature and a brightness of a first pixel in the first image may be determined, and a mapping between pixel characteristics and noise deviation of the image sensor may be selected. The pixel characteristics may include pixel brightness. The selected mapping may be used to map at least the brightness of the first pixel to a particular noise deviation. The brightness of the first pixel and the particular noise deviation may be compared to a brightness of a second pixel of the second image. The comparison may be used to determine whether to merge the first pixel and the second pixel.

What is claimed is: 
     
         1 . A method comprising:
 mapping, by a processor of a camera device, a brightness of a first pixel of a first image captured by an image sensor of the camera device to a noise deviation of the image sensor, wherein the noise deviation characterizes error in pixels that have a color temperature of the first pixel, wherein the error is introduced when the image sensor is capturing an image; and   based on the brightness of the first pixel, the noise deviation, and a brightness of a second pixel, determining, by the processor of the camera device, whether to merge the first pixel and the second pixel, wherein the second pixel appears in a second image that was captured by the image sensor.   
     
     
         2 . The method of  claim 1 , wherein a particular gain has been applied to the first image, and wherein the noise deviation characterizes the error in pixels that have the color temperature of the first pixel and have had the particular gain applied. 
     
     
         3 . The method of  claim 2 , wherein the mapping is selected based on the color temperature of the first pixel and the particular gain. 
     
     
         4 . The method of  claim 1 , wherein determining whether to merge the first pixel and the second pixel comprises comparing the brightness of the first pixel and the noise deviation to the brightness of the second pixel. 
     
     
         5 . The method of  claim 4 , wherein comparing the brightness of the first pixel and the noise deviation to the brightness of the second pixel of the second image comprises:
 determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the noise deviation; and   based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.   
     
     
         6 . The method of  claim 5 , wherein merging the first pixel and the second pixel comprises:
 adding the brightness of the first pixel and the brightness of the second pixel to provide a pixel value sum; and   dividing the pixel value sum by two to form a merged pixel.   
     
     
         7 . The method of  claim 5 , wherein the defined range is configured such that the brightness of the second pixel is within the defined range when the first pixel and the second pixel are subject to a sub-pixel alignment error. 
     
     
         8 . The method of  claim 4 , wherein a first tile of pixels contains the first pixel and a second tile of pixels contains the second pixel, wherein comparing the brightness of the first pixel and the noise deviation to the brightness of the second pixel comprises:
 determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the noise deviation;   determining that the root-mean-square difference of pixel brightness values of the first tile and pixel brightness values of the second tile exceeds a root-mean-square difference threshold; and   based on the root-mean-square difference exceeding the root-mean-square difference threshold, merging no pixels from the second tile with the pixels from the first tile.   
     
     
         9 . The method of  claim 4 , wherein comparing the brightness of the first pixel and the noise deviation to the brightness of the second pixel comprises:
 determining that the brightness of the second pixel is outside of a defined range of the brightness of the first pixel, wherein the defined range is based on the noise deviation; and   based on the brightness of the second pixel being outside of the defined range, not merging the first pixel and the second pixel.   
     
     
         10 . The method of  claim 1 , wherein a noise model for the image sensor associates color temperatures to mappings between pixel characteristics and the noise deviation of the image sensor, and wherein the mapping is selected from the mappings of the noise model based on the color temperature. 
     
     
         11 . The method of  claim 10 , wherein the noise model comprises a plurality of discrete points, each point defined by a respective color temperature and associated with a respective mapping, and wherein selecting the mapping comprises:
 determining that the color temperature of the first pixel does not correspond to any of the discrete points;   selecting a discrete point defined by a target color temperature, wherein target color temperature is a closest color temperature to the color temperature of the first pixel; and   selecting the mapping associated with the discrete point.   
     
     
         12 . The method of  claim 1 , wherein mapping the brightness of the first pixel to the noise deviation comprises looking up the brightness of the first pixel in a pre-determined table to derive the noise deviation. 
     
     
         13 . The method of  claim 12 , wherein mapping the brightness of the first pixel to the noise deviation further comprises looking up a color value of the first pixel in the pre-determined table to derive the noise deviation. 
     
     
         14 . The method of  claim 1 , wherein the first image and the second image were captured using different exposure times. 
     
     
         15 . The method of  claim 1 , wherein the mapping comprises a mapping between the pixel brightness and a combination of the noise deviation of the image sensor and noise deviation of other image pipeline components of the camera device. 
     
     
         16 . An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a camera device, cause a processor of the camera device to perform operations comprising:
 mapping a brightness of a first pixel of a first image captured by an image sensor of the camera device to a noise deviation of the image sensor, wherein the noise deviation characterizes error in pixels that have a color temperature of the first pixel, wherein the error is introduced when the image sensor is capturing an image; and   based on the brightness of the first pixel, the noise deviation, and a brightness of a second pixel, determining whether to merge the first pixel and the second pixel, wherein the second pixel appears in a second image that was captured by the image sensor.   
     
     
         17 . The article of manufacture of  claim 16 , wherein determining whether to merge the first pixel and the second pixel comprises comparing the brightness of the first pixel and the noise deviation to the brightness of the second pixel. 
     
     
         18 . The article of manufacture of  claim 17 , wherein comparing the brightness of the first pixel and the noise deviation to the brightness of the second pixel comprises:
 determining that the brightness of the second pixel is within a defined range of the brightness of the first pixel, wherein the defined range is based on the noise deviation; and   based on the brightness of the second pixel being within the defined range, merging the first pixel and the second pixel.   
     
     
         19 . The article of manufacture of  claim 18 , wherein merging the first pixel and the second pixel comprises:
 adding the brightness of the first pixel and the brightness of the second pixel to provide a pixel value sum; and   dividing the pixel value sum by two to form a merged pixel.   
     
     
         20 . A camera device comprising:
 an image sensor;   a processor;   memory; and   program instructions, stored in the memory, that upon execution by the processor cause the camera device to perform operations including:
 mapping a brightness of a first pixel of a first image captured by the image sensor to a noise deviation of the image sensor, wherein the noise deviation characterizes error in pixels that have a color temperature of the first pixel, wherein the error is introduced when the image sensor is capturing an image; and 
 based on the brightness of the first pixel, the noise deviation, and a brightness of a second pixel, determining whether to merge the first pixel and the second pixel, wherein the second pixel appears in a second image that was captured by the image sensor.