Hybrid image sensors with adjustable contrast thresholds

Abstract

Hybrid image sensors with adjustable contrast thresholds (and associated systems, devices, and methods) are disclosed herein. In one embodiment, an imaging system comprises one or more event vision sensor (EVS) pixels, and a plurality of CMOS image sensor (CIS) pixels. Each EVS pixel can be configured to, based on a contrast threshold, capture event data corresponding to contrast information of light incident on the EVS pixel. Each CIS pixel can be configured to capture CIS data corresponding to intensity of light incident on the CIS pixel. The imaging system can further comprise (i) a contrast threshold calibration circuit configured to adjust a value of the contrast threshold over time, and (ii) a deblur circuit configured to generate deblurred image data by deblurring the CIS data captured by the plurality of CIS pixels using at least a portion of the event data captured by the one or more EVS pixels.

What is claimed is: 
     
         1 . An imaging system, comprising:
 an event driven sensing array including one or more event vision sensor (EVS) pixels, wherein each EVS pixel of the one or more EVS pixels is configured to, based at least in part on a contrast threshold, capture event data corresponding to contrast information of light incident on the EVS pixel;   a pixel array including a plurality of CMOS image sensor (CIS) pixels arranged in one or more CIS pixel rows, wherein each CIS pixel of the plurality of CIS pixels is configured to capture CIS data corresponding to intensity of light incident on the CIS pixel;   a contrast threshold calibration circuit configured to adjust a value of the contrast threshold over time; and   a deblur circuit configured to generate deblurred image data by deblurring the CIS data captured by the plurality of CIS pixels using at least a portion of the event data captured by the one or more EVS pixels.   
     
     
         2 . The imaging system of  claim 1 , further comprising:
 a hybrid image sensor including the event driven sensing array, the pixel array, the contrast threshold calibration circuit, and the deblur circuit; and   a physical interface usable to output the deblurred image data from the image sensor.   
     
     
         3 . The imaging system of  claim 1 , further comprising a hybrid image sensor including the event driven sensing array and the pixel array, wherein the contrast threshold calibration circuit is positioned external the hybrid image sensor. 
     
     
         4 . The imaging system of  claim 1 , wherein the contrast threshold calibration circuit is configured to, based at least in part on one or more operating conditions of the imaging system, adjust the value of the contrast threshold using a lookup table or an estimation algorithm. 
     
     
         5 . The imaging system of  claim 4 , wherein the one or more operating conditions include luminance levels of light incident on the imaging system and/or a signal included within the light. 
     
     
         6 . The imaging system of  claim 4 , wherein the one or more operating conditions includes a temperature corresponding to the imaging system. 
     
     
         7 . The imaging system of  claim 1 , wherein the contrast threshold calibration circuit is configured to dynamically adjust the value of the contrast threshold over time. 
     
     
         8 . The imaging system of  claim 1 , wherein the contrast threshold calibration circuit is configured to periodically adjust the value of the contrast threshold at one or more preset times. 
     
     
         9 . The imaging system of  claim 8 , wherein the contrast threshold calibration circuit is configured to adjust the value of the contrast threshold at starts of image frames and/or at starts of exposure periods for at least a subset of the plurality of CIS pixels. 
     
     
         10 . The imaging system of  claim 1 , wherein the contrast threshold is a global contrast threshold used for all EVS pixels of the event driven sensing array. 
     
     
         11 . The imaging system of  claim 1 , wherein the contrast threshold is a local contrast threshold used for less than all EVS pixels of the event driven sensing array. 
     
     
         12 . The imaging system of  claim 1 , wherein the deblur circuit is configured, for each EVS pixel of the one or more EVS pixels, to—
 compute resulting products by, for each event included in the event data captured by the EVS pixel, computing a resulting product of (i) a polarity of the event and (ii) the value of the contrast threshold at a time corresponding to when the event is detected by the EVS pixel, and 
 compute results of an integration by integrating the resulting products over a time period. 
 
     
     
         13 . The imaging system of  claim 12 , wherein the deblur circuit includes an integration buffer configured to store the results of the integration in floating point representation. 
     
     
         14 . The imaging system of  claim 12 , wherein the time period at least partially overlaps with an exposure period for the plurality of CIS pixels, wherein the integration is a first integration, and wherein the deblur circuit is further configured, for each EVS pixel of the one or more EVS pixels, to—
 compute exponentials of the results of the first integration over the time period, and 
 compute results of a second integration by integrating the exponentials over the time period. 
 
     
     
         15 . The imaging system of  claim 14 , wherein the deblur circuit is configured to generate the deblurred image data by deblurring the CIS data using the results of the second integration. 
     
     
         16 . The imaging system of  claim 12 , wherein the time period is an accumulation period that precedes a start of an exposure period for the plurality of CIS pixels, and wherein the deblur circuit is further configured to correct the deblurred image data for rolling shutter distortion using the results of the integration. 
     
     
         17 . The imaging system of  claim 12 , further comprising:
 a hybrid image sensor that includes the event driven sensing array, the pixel array, and the deblur circuit; and   an application processor external to the hybrid image sensor and including an EVS frame buffer,   wherein the time period corresponds to an interpolation frame, and   wherein the deblur circuit is configured to output (a) the results of the integration or (b) exponentials of the results of the integration to the application processor for storage in the EVS frame buffer.   
     
     
         18 . The imaging system of  claim 17 , wherein the application processor further includes a CIS key frame buffer, and wherein the deblur circuit is further configured to output the deblurred image data to the application processor for storage in the CIS key frame buffer. 
     
     
         19 . The imaging system of  claim 18 , wherein the application processor further includes a video frame interpolation circuit configured to generate one or more interpolated frames based at least in part on (i) the deblurred image data and (ii) the results of the integration or the exponentials of the results of the integration. 
     
     
         20 . A method of operating an imaging system, the method comprising:
 adjusting a value of a contrast threshold over time, wherein the contrast threshold is usable by each event vision sensor (EVS) pixel of one or more EVS pixels of the imaging system to capture event data corresponding to contrast information of light incident on the EVS pixel; and   for each EVS pixel of the one or more EVS pixels and over a time period,
 computing resulting products by, for each event included in the event data captured by the EVS pixel, computing a resulting product of (i) a polarity of the event and (ii) the value of the contrast threshold at a time corresponding to when the event is detected by the EVS pixel, and 
 compute results of an integration by integrating the resulting products over the time period. 
   
     
     
         21 . The method of  claim 20 , wherein adjusting the value of the contrast threshold over time includes dynamically adjusting the value of the contrast threshold over time. 
     
     
         22 . The method of  claim 20 , wherein adjusting the value of the contrast threshold over time includes periodically adjusting the value of the contrast threshold over time. 
     
     
         23 . The method of  claim 20 , wherein adjusting the value of the contrast threshold over time includes adjusting the value of the contrast threshold based at least in part on light levels of light incident on the imaging system and/or signals included within the light. 
     
     
         24 . The method of  claim 20 , wherein adjusting the value of the contrast threshold over time includes adjusting the value of the contrast threshold based at least in part on a temperature of the imaging system. 
     
     
         25 . The method of  claim 20 , further comprising, for each EVS pixel of the one or more EVS pixels and over the time period, storing a floating point representation of the results of the integration in a buffer. 
     
     
         26 . The method of  claim 20 , wherein:
 the integration is a first integration;   the time period at least partially overlaps with an exposure period for CMOS image sensor (CIS) pixels corresponding to the one or more EVS pixels; and   the method further comprises—
 for each EVS pixel of the one or more EVS pixels and over the time period—
 computing exponentials of the result of the first integration; and 
 computing results of a second integration by integrating the exponentials over the time period, and 
 
 deblurring CIS data captured by the CIS pixels during the exposure period using the results of the second integration. 
   
     
     
         27 . The method of  claim 20 , wherein:
 the time period corresponds to an accumulation period for the one or more EVS pixels, wherein the accumulation period precedes an exposure period for CMOS image sensor (CIS) pixels that correspond to the one or more EVS pixels; and   the method further comprises correcting CIS data for rolling shutter distortion using the results of the integration, wherein the CIS data is captured by the CIS pixels during the exposure period.   
     
     
         28 . The method of  claim 20 , wherein:
 the time period corresponds to an interpolation frame;   the method further comprises, for each EVS pixel of the one or more EVS pixels, outputting (a) the results of the integration or (b) an exponential of the results of the integration, to an EVS frame buffer coupled to a video frame interpolation circuit.   
     
     
         29 . The method of  claim 28 , further comprising:
 capturing, over an exposure period, CMOS image sensor (CIS) data using a plurality of CIS pixels corresponding to the one or more EVS pixels;   generating deblurred and/or rolling-shutter-distortion-corrected CIS data based at least in part on the CIS data and the results of the integration; and   outputting the deblurred and/or rolling-shutter-distortion-corrected CIS data to a CIS frame buffer coupled to the video frame interpolation circuit.   
     
     
         30 . The method of  claim 29 , further comprising interpolating at least one frame of image data based at least in part on (a) the results of the integration or the exponential of the results of the integration in the EVS frame buffer, and (b) the deblurred and/or rolling-shutter-distortion-corrected CIS data in the CIS frame buffer.