Method and multi-camera portable device for producing stereo images

Abstract

A method and a multi-camera portable device for producing stereo images. The device has at least two image sensors, at least one of which is arranged on a first face of the portable device and a second one of which is arranged on a second face of the portable device opposite to the first face thereof. The method involves substantially simultaneously recording a first image containing a picture of an object and a second image containing a picture of a mirrored view of the object, obtaining the direction and the length of a mirroring vector from the distance ratio of a first distance in reality and a corresponding second distance in the second image, obtaining a capturing focal length of the second image sensor, and transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair.

I claim: 
     
       1. A method for producing a stereo image of an object with using a multi-camera portable device having at least two image sensors, at least a first one of said image sensors being arranged on a first face of the portable device and a second one of said image sensors being arranged on a second face of the portable device opposite to the first face thereof, the method comprising the steps of:
 arranging the object in front of the first image sensor so that the object falls within the field of view of the first image sensor, 
 arranging an external mirror in front of the second image sensor so that a mirrored view of the object and a mirrored view of the portable device fall within the field of view of the second image sensor, 
 substantially simultaneously recording, by the first image sensor, a first initial image containing a picture of the object, and by the second image sensor, a second initial image containing (i) a picture of the mirrored view of the object appearing in the mirror and (ii) a picture of at least a portion of the mirrored view of the portable device appearing in the mirror, thereby producing an initial pair of images, 
 finding the center of the picture of the mirrored view of the second image sensor within the second initial image, 
 determining the distance ratio of a first distance between two points of the portable device in reality and a second distance of the respective points in the second initial image for obtaining the direction and the length of a mirroring vector, which points from the center of the second image sensor to the center of the virtual second image sensor, 
 obtaining a capturing focal length of the second image sensor, and 
 by using said mirroring vector and the capturing focal length of the second image sensor, transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair from the first initial image and the second initial image for the object. 
 
     
     
       2. The method of  claim 1 , wherein the multi-camera portable device is any one of a mobile phone, a smart phone, a phablet, a tablet computer, a notebook. 
     
     
       3. The method of  claim 1 , wherein finding the center of the picture of the mirrored view of the second image sensor in the second initial image comprises the steps of:
 a. finding a sub-image within the second initial image, wherein said sub-image best matches to the first initial image, 
 b. determining a plurality of candidate positions and sizes of the picture of the mirrored view of the second image sensor in the second initial image by using various selected values of the relative focal length of the second image sensor and various selected distance values between the second image sensor and the object to find the center point of the picture of the mirrored view of the second image sensor in said plurality of candidate positions in the second initial image, 
 c. selecting a best estimate for the position and the size of the picture of the mirrored view of the second image sensor in the second initial image by using a standard generalized Hough transform. 
 
     
     
       4. The method of  claim 1 , wherein the capturing focal length of the second image sensor is obtained from a controller of the second image sensor. 
     
     
       5. The method of  claim 1 , wherein the capturing focal length of the second image sensor is obtained by performing the steps of:
 a. creating several coordinate transformation matrices with different values of a relative focal length of the second image sensor for mapping the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor, said relative focal length being defined as the ratio of the focal length and the pixel size of the second image sensor, 
 b. generating a series of candidate stereo image pairs, wherein in each of the candidate stereo image pairs, the first initial image is transformed into the coordinate system of the virtual second image sensor with a different coordinate transformation matrix, and 
 c. selecting the best matching stereo image pair, 
 d. determining the capturing focal length of the second image sensor from the relative focal length belonging to the coordinate transformation matrix of the best matching stereo image pair. 
 
     
     
       6. The method of  claim 1 , wherein the method further comprises the step of producing a depth map of the object using the stereo image pair containing the object. 
     
     
       7. A non-transitory data storage medium comprising a set of processor-readable instructions, said instructions being adapted, when executed by a processor of a multi-camera portable device, to carry out the steps of:
 instructing a first image sensor of the device to record a first initial image containing a picture of an object arranged within the field of view of the first image sensor, and a second image sensor of the device to substantially simultaneously record a second initial image containing (i) a picture of a mirrored view of the object appearing in a mirror and (ii) a picture of at least a portion of a mirrored view of the multi-camera portable device itself appearing in the mirror, thereby to produce an initial pair of images, wherein said mirrored view of the second image sensor and said portion of the mirrored view of the device are arranged within the field of view of the second image sensor, 
 finding the center of the picture of the mirrored view of the second image sensor within the second initial image, 
 determining a distance ratio of a first distance between two points of the device in reality and a second distance of the respective points in the second initial image for obtaining a direction and a length of a mirroring vector, which points from the center of the second image sensor to the center of the virtual second image sensor, obtaining a capturing focal length of the second image sensor, and 
 by using said mirroring vector and the capturing focal length of the second image sensor, transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair from the first initial image and the second initial image for the object. 
 
     
     
       8. A method of producing a stereo image of an object with using a multi-camera device having at least two image sensors, the method comprising the steps of:
 arranging a first image sensor on one side of a portable device and another image sensor on the opposite side of the portable device and an external mirror in front of the first image sensor in which the first image sensor captures a picture of an object and substantially simultaneously second image sensor captures picture of the mirrored image of the object and mirrored image of at least a portion of the portable device; 
 determining a distance ratio of a first distance between two points of the portable device in reality and a second distance of the respective points in the second initial image for obtaining the direction and the length of a mirroring vector, which points from the center of the second image sensor to the center of the virtual second image sensor; obtaining a capturing focal length of the second image sensor, and by using said mirroring vector and the capturing focal length of the second image sensor, transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair from the first initial image and the second initial image for the object. 
 
     
     
       9. A method comprising:
 obtaining a first image, the first image comprising a first picture of an object;   obtaining a second image comprising a second picture and a third picture, the second picture comprising a mirror view of the object, and the third picture comprising a mirror view of a camera;   identifying a center of the third picture comprising the mirror view of the camera;   determining a relative focal length defined for the camera;   determining a distance ratio based at least on the relative focal length for the camera and the center of the third picture comprising the mirror view of the camera;   determining a mirroring vector based on the distance ratio and the center of the third picture comprising the mirror view of the camera;   determining a coordinate transformation from a first coordinate system of the first image and a second coordinate system of the second image based at least on the distance ratio, the relative focal length of the camera, and the mirroring vector; and   generating a stereo image pair of the object based at least on the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image.   
     
     
       10. The method of claim 9, further comprising measuring a distance associated with the object based on the stereo image pair of the object. 
     
     
       11. The method of claim 9, further comprising generating a depth image of the object based on the stereo image pair of the object. 
     
     
       12. The method of claim 9, further comprising generating a three dimensional (3D) rendering of the object based on the stereo image pair of the object. 
     
     
       13. The method of claim 9, wherein determining the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image is based on a coordinate transformation associated with a coordinate system from a perspective of a camera taking the first image and a coordinate system from a perspective of a camera taking the second image. 
     
     
       14. The method of claim 9, wherein the second coordinate system of the second image comprises a coordinate system corresponding to the second picture comprising a mirror view of the object in the second image. 
     
     
       15. A non-transitory data storage medium comprising a set of processor-readable instructions that when executed by a processor cause the processor to execute a method comprising:
 obtaining a first image, the first image comprising a first picture of an object;   obtaining a second image comprising a second picture and a third picture, the second picture comprising a mirror view of the object, and the third picture comprising a mirror view of a camera;   identifying a center of the third picture comprising the mirror view of the camera;   determining a relative focal length defined for the camera;   determining a distance ratio based at least on the relative focal length for the camera and the center of the third picture comprising the mirror view of the camera;   determining a mirroring vector based on the distance ratio and the center of the third picture comprising the mirror view of the camera;   determining a coordinate transformation from a first coordinate system of the first image and a second coordinate system of the second image based at least on the distance ratio, the relative focal length of the camera, and the mirroring vector; and   generating a stereo image pair of the object based at least on the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image.   
     
     
       16. The non-transitory data storage medium of claim 15, wherein the method executed by the processor further comprises measuring a distance associated with the object based on the stereo image pair of the object. 
     
     
       17. The non-transitory data storage medium of claim 15, wherein the method executed by the processor further comprises generating a depth image of the object based on the stereo image pair of the object. 
     
     
       18. The non-transitory data storage medium of claim 15, wherein the method executed by the processor further comprises generating a three dimensional (3D) rendering of the object based on the stereo image pair of the object. 
     
     
       19. The non-transitory data storage medium of claim 15, wherein determining the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image is based on a coordinate transformation associated with a coordinate system from a perspective of a camera taking the first image and a coordinate system from a perspective of a camera taking the second image. 
     
     
       20. The non-transitory data storage medium of claim 15, wherein the second coordinate system of the second image comprises a coordinate system corresponding to the second picture comprising a mirror view of the object in the second image.