Image distortion correction of a camera with a rolling shutter

Abstract

Correcting image distortion during camera motion using a system including a processor and a camera having a rolling shutter. Multiple image frames are captured by the camera equipped with the rolling shutter. The captured image frames include a base image frame and a previous image frame. Multiple time stamps are recorded respectively for multiple corresponding image points in the previous and base image frames. For the corresponding image points, multiple ego-motions are computed responsive to the time stamps of the corresponding image points of the base image frame and the previous image frame to correct the image distortion caused by the rolling shutter.

What is claimed is: 
     
         1 . A system for processing images captured by a moving rolling shutter camera, the system comprising:
 an interface configured to receive, from the rolling shutter camera, a plurality of images of an environment of the camera, captured while the camera is moving;   a processor configured to:
 correct for a global ego-motion of the camera between a first image from the plurality of images and a second image from the plurality of images; 
 correct the first image for a rolling shutter distortion, to give rise to a corrected first image; and 
 match pixels' locations in the corrected first image with respective pixels' locations in at least the second image. 
   
     
     
         2 . The system according to  claim 1 , wherein the global ego-motion of the camera is represented by data assuming a first capture time simultaneously for all pixels of of the first image and a second capture time simultaneously for all pixels of the second image. 
     
     
         3 . The system according to  claim 2 , wherein the processor is configured to perform the correction for the global ego-motion of the camera by taking into account the data representing the egomotion of the camera between the assumed first capture time and the assumed second capture time. 
     
     
         4 . The system according to  claim 2 , wherein the global ego-motion is at least one of: a translation and a rotation of the camera between the first capture time of the first image and the second capture time of the second image. 
     
     
         5 . The system according to  claim 1 , wherein the correction of the first image for the rolling shutter distortion uses a plurality of different time values, for respective rows or for respective pixels in the first image. 
     
     
         6 . The system according to  claim 5 , wherein the time values correspond to capture times of the respective rows or pixels during operation of the rolling shutter camera. 
     
     
         7 . The system according to  claim 5 , wherein the processor is configured to perform the rolling shutter distortion correction by taking into account data representing a motion of the camera between exposure times of different rows of the rolling shutter camera. 
     
     
         8 . The system according to  claim 1 , wherein the processor is further configured to:
 utilize an ego-motion matrix to process the first image,   apply a global ego-motion correction to the ego-motion matrix, wherein the global ego-motion correction does not take into account different exposure times of different rows or different pixels of the rolling shutter camera with respect to the first image; and   apply a polynomial expression representing ego-motion variation between different rows or different pixels due to the rolling shutter camera.   
     
     
         9 . The system according to  claim 1 , wherein the processor is further configured to adjust a time stamp associated with a picture element in the first image according to a predefined distortion per picture element model. 
     
     
         10 . The system according to  claim 1 , wherein the processor is configured to:
 identify a plurality of pairs of image points, wherein a first image point of a given pair of image points is an image point in the corrected first image, and a second image point of the image point pair is a point in the second image which corresponds to the first image point;   associate each of the first and the second image points in each pair of image points with respective first and second epipolar lines; and   determine a distance between the first and second epipolar lines of each pair of image points.   
     
     
         11 . The system according to  claim 1 , wherein the processor is further configured to compute depth information based on locations of pixels in the corrected first image and based on locations of corresponding pixels in at least the second image. 
     
     
         12 . The system according to  claim 1 , wherein the processor is further configured to determine a distance to at least a portion of an object imaged in the first image and in the second image based on a relation between corresponding pixel locations in the first and second images. 
     
     
         13 . A method for processing images captured by a rolling shutter camera, the method comprising:
 configuring an interface to receive, from the rolling shutter camera, a plurality of images, of an environment of the camera, captured while the camera is moving;   configuring a processor to:
 correct for a global ego-motion between a first image from the plurality of images and a second image from the plurality of images; 
 correct the first image for the rolling shutter distortion, giving rise to a corrected first image; and 
 match pixels' locations in the corrected first image with respective pixels' locations in at least the second image. 
   
     
     
         14 . The method according to  claim 13 , further comprising:
 representing the global ego-motion by data assuming a first capture time simultaneously for all pixels of of the first image and a second capture time simultaneously for all pixels of the second image.   
     
     
         15 . The method according to  claim 14 , further comprising:
 configuring the processor to perform the global ego-motion correction taking into account the data representing the global egomotion of the camera between the assumed first capture time and the assumed second capture time.   
     
     
         16 . The method according to  claim 14 , wherein the global ego-motion is at least one of: a translation and a rotation of the camera between a capture time of the first image and a capture time of the second image. 
     
     
         17 . The method according to  claim 13 , further comprising:
 configuring the processor to use a plurality of different time values for rolling-shutter distortion correction for respective rows or for respective pixels in the first image.   
     
     
         18 . The method according to  claim 17 , wherein the time values correspond to capture times of the respective rows or pixels during operation of the rolling shutter camera. 
     
     
         19 . The method according to  claim 17 , further comprising:
 configuring the processor to perform the rolling shutter distortion correction taking into account data representing a motion of the camera between exposure times of different rows of the rolling shutter camera.   
     
     
         20 . The method according to  claim 13 , further comprising:
 configuring the processor to:
 utilize an ego-motion matrix to process the first image, 
 apply a global ego-motion correction to the ego-motion matrix, wherein the global ego-motion correction does not take into account different exposure times of different rows or different pixels of the rolling shutter camera with respect to the first image, and 
 apply a polynomial expression representing ego-motion variation between different rows or different pixels due to the rolling shutter camera. 
   
     
     
         21 . The method according to  claim 13 , further comprising:
 configuring the processor to adjust a time stamp associated with a pixel in the first image according to a predefined distortion per picture element model.   
     
     
         22 . The method according to  claim 13 , further comprising:
 configuring the processor to:   identify a plurality of pairs of image points, wherein a first image point of a given pair of image points is an image point in the corrected first image, and a second image point of the image point pair is a point in the second image which corresponds to the first image point;   associate each of the first and the second image points in each pair of image points with respective first and second epipolar lines; and   determine a distance between the first and second epipolar lines of each pair of image points.   
     
     
         23 . The method according to  claim 13 , further comprising:
 configuring the processor to compute depth information based on locations of pixels in the corrected first image and based on locations of corresponding pixels in at least the second image.   
     
     
         24 . The method according to  claim 13 , further comprising:
 configuring the processor to determine a distance to at least a portion of an object imaged in the first image and in the second image based on a relation between corresponding pixel locations in the first and second images.   
     
     
         25 . A method for processing images captured by a rolling shutter camera, the method comprising:
 receiving, from the rolling shutter camera, a plurality of images, of an environment of the camera, captured while the camera is moving;   correcting for a global ego-motion between a first image from the plurality of images and a second image from the plurality of images;   correcting the first image for the rolling shutter distortion, giving rise to a corrected first image; and   matching pixels' locations in the corrected first image with respective pixels' locations in at least the second image.