Direct vision sensor for 3D computer vision, digital imaging, and digital video

Abstract

A method and apparatus for directly sensing both the focused image and the three-dimensional shape of a scene are disclosed. This invention is based on a novel mathematical transform named Rao Transform (RT) and its inverse (IRT). RT and IRT are used for accurately modeling the forward and reverse image formation process in a camera as a linear shift-variant integral operation. Multiple images recorded by a camera with different camera parameter settings are processed to obtain 3D scene information. This 3D scene information is used in computer vision applications and as input to a virtual digital camera which computes a digital still image. This same 3D information for a time-varying scene can be used by a virtual video camera to compute and produce digital video data.

1. A method of determining a focused image f(x,y) of a scene with respect to a camera, said scene having a 3D shape z(x,y) which is characterized by a shape parameter vector z, and said camera characterized by a set of camera parameters e p , said method comprising:
 (a) recording a set of images g p (x,y) of said object with different camera parameter settings e p , for p=1,2,3, . . . ,P; 
 (b) obtaining a set of normalized images by preprocessing said set of images g p (x,y) to reduce noise by filtering and normalizing them with respect to average brightness and magnification so that said set of normalized images are related to said focused image f(x,y) by a linear integral of the product of said focused image with a non-localized shift-variant point spread function h′; 
 (c) computing a set of derivatives g p   (m,n) (x,y) of said set of normalized images to obtain a set of image derivatives where m and n are the integer order of derivatives with respect to x and y respectively, for m,n=0,1,2,3, . . . ; 
 (d) determining the localized shift-variant point spread function (SV-PSF) h from said non-localized PSF h′ using the novel Rao Localization Transform expressed in the form
     h ( x,y,α,β;e   p   ,z )= h′ ( x+α,y+β,x,y;e   p   ,z ) 
 
 
     where (x,y) specifies the position where image brightness is measured and (α,β) specifies the position of surrounding image points which are contributing to light measured at (x,y);
 (e) deriving a set of expressions for the various moments with respect to α and β of the derivatives with respect to x and y of the localized SV-PSF h i,j   (m,n)  in terms of the camera parameters e p  and shape parameter vector z; 
 (f) deriving a system of P algebraic equations that express said focused image f(x,y) as a weighted sum of the derivatives of said set of normalized images g p   (m,n) (x,y) in terms of the Inverse Rao Transform (IRT) coefficients that are functions of h i,j   (m,n) ; 
 (g) solving said system of P algebraic equations to obtain said focused image f(x,y) at each point (x,y), and providing it as digital image output. 
 
   
   
     2. The method of  claim 1  wherein said non-localized shift-variant point spread function h′ is a Gaussian function. 
   
   
     3. The method of  claim 1  wherein said non-localized shift-variant point spread function h′ is a Cylindrical or pill-box function. 
   
   
     4. The method of  claim 1  wherein said non-localized shift-variant point spread function h′ is determined by physical or wave optics model of image formation. 
   
   
     5. The method of  claim 1 , wherein said focused image f(x,y) and 3D shape z(x,y) along with desired camera parameters are provided as input to a virtual camera simulation software and an output image f′(x,y) that would be recorded by a real camera corresponding to said virtual camera is computed and provided as output. 
   
   
     6. The method of  claim 1  wherein said output image f(x,y) is computed at regular time intervals for a time-varying scene and the resulting computed output image sequence is synthesized into video data and provided as video output. 
   
   
     7. The method of  claim 6  wherein said non-localized shift-variant point spread function h′ is a Gaussian function. 
   
   
     8. The method of  claim 6  wherein said non-localized shift-variant point spread function h′ is a Cylindrical or pill-box function. 
   
   
     9. The method of  claim 6  wherein said non-localized shift-variant point spread function h′ is determined by physical or wave optics model of image formation. 
   
   
     10. A method of determining a 3D shape z(x,y) of a scene with respect to a camera, said 3D shape z(x,y) characterized by a shape parameter vector z, said scene having a focused image f(x,y), and said camera characterized by a set of camera parameters e p , said method comprising:
 (a) recording a set of images g p (x,y) of said object with different camera parameter settings e p , for p=1,2,3, . . . ,P; 
 (b) obtaining a set of normalized images by preprocessing said set of images g p (x,y)to reduce noise by filtering and normalizing them with respect to average brightness and magnification so that said set of normalized images are related to said focused image f(x,y) by a linear integral of the product of said focused image with a non-localized shift-variant point spread function h′; 
 (c) computing a set of derivatives g p   (m,n) (x,y) of said set of normalized images to obtain a set of image derivatives where m and n are the integer order of derivatives with respect to x and y respectively, for m,n=0,1,2,3, . . . ; 
 (d) determining the localized shift-variant point spread function (SV-PSF) h from said non-localized PSF h′ using the novel Rao Localization Transform expressed in the form
     h ( x,y,α,β;e   p   ,z )= h ′( x+α,y+β,x,y;e   p   ,z ) 
 
 
     where (x,y) specifies the position where image brightness is measured and (α,β) specifies the position of surrounding image points which are contributing to light measured at (x,y);
 (e) deriving a set of expressions for the various moments with respect to α and β of the derivatives with respect to x and y of the localized SV-PSF h i,j   (m,n)  in terms of the camera parameters e p  and shape parameter vector z; 
 (f) deriving a system of P algebraic equations that express said focused image f(x,y) as a weighted sum of the derivatives of said set of normalized images g p   (m,n) (x,y) in terms of the Inverse Rao Transform (IRT) coefficients that are functions of h i,j   (m,n) ; 
 (g) solving said system of P algebraic equations to obtain said shape parameter vector z and said 3D shape z(x,y) at each point (x,y), and providing it as digital shape output. 
 
   
   
     11. The method of  claim 10  wherein said non-localized shift-variant point spread function h′ is a Gaussian function. 
   
   
     12. The method of  claim 10  wherein said non-localized shift-variant point spread function h′ is a Cylindrical or pill-box function. 
   
   
     13. The method of  claim 10  wherein said non-localized shift-variant point spread function h′ is determined by physical or wave optics model of image formation. 
   
   
     14. The method of  claim 10  wherein said output shape z(x,y) is computed at regular time intervals for a time-varying scene and the resulting computed output shape sequence is synthesized into time-varying shape data and provided output. 
   
   
     15. The method of  claim 14  wherein said non-localized shift-variant point spread function h′ is a Gaussian function. 
   
   
     16. The method of  claim 14  wherein said non-localized shift-variant point spread function h′ is a Cylindrical or pill-box function. 
   
   
     17. The method of  claim 14  wherein said non-localized shift-variant point spread function h′ is determined by physical or wave optics model of image formation. 
   
   
     18. Apparatus for determining the focused image and 3D shape of an object with respect to a camera, said apparatus comprising a camera characterized by a set of camera parameters, a camera controller for setting said set of camera parameters to desired values, an image detector for capturing images, an image processing computer with software components operatively connected for (i) image normalization for standardizing images with respect to average brightness and magnification, (ii) noise reduction filter, (iii) image derivative computation module, (iv) a unique and essential software component for computing Inverse Rao Transform coefficients, and (v) software for solving algebraic equations to obtain focused image and 3D shape. 
   
   
     19. The apparatus of  claim 18  with an additional software component for virtual image camera simulation. 
   
   
     20. The apparatus of  claim 18  with and additional software component for virtual video camera simulation.