Methods for creating an image for a three-dimensional display, for calculating depth information, and for image processing using the depth information

Abstract

A method is proposed for automatically obtaining depth information from a 2-D motion image, so as to create an image for a 3-D display. Further, methods are proposed for selecting appropriate frames for the calculation of the depth information, or discontinuing the calculation, and for conducting image processing using the depth information. Examples of various types of image processing can be listed as including the creation of a viewfinder image seen from a different point, natural scaling operations to an image area, and separation of a desired image area. First, a motion information of an object on a screen is extracted by block matching or the like. Second, the actual movement of the object in the 3-D world is calculated. Since the viewfinder image is a projection of a 3-D space, it is possible to obtain the original 3-D movement of the object, based on the movements of a plurality of representative points through an inverse transformation, the representative points being provided in the viewfinder image. Resultantly, 3-D coordinates of the object are identified, so that depth information of the object is obtained. Afterwards, a parallax is calculated based on the depth information, so as to create right and left eye images from the input viewfinder image. Alternatively, image processing, such as separation of an object having a depth within a predetermined range, is carried out based on the depth information.

What is claimed is:  
     
         1 . A method for creating an image for a 3-D display, comprising: 
 a step of extracting depth information from a 2-D motion image; and    a step of creating an image for a 3-D display according to the depth information.    
     
     
         2 . A method for creating an image for a 3-D display according to    claim 1   , wherein the step of extracting depth information comprises: 
 a step of detecting a movement of the 2-D motion image;    a step of calculating a relative 3-D movement between a scene and a shooting viewpoint of the 2-D motion image; and    a step of calculating relative distances from the shooting viewpoint to respective parts in 3-D space, which are projected in 2-D motion image, based on the relative 3-D movement and the 2-D movements of the respective image parts.    
     
     
         3 . A method for calculating depth information, based on frames included in a 2-D motion image, comprising: 
 a step of selecting two frames with an appropriately large movement between them from the 2-D motion image; and    a step of calculating the depth information, based on the two frames.    
     
     
         4 . A method for calculating depth information according to    claim 3   , further comprising: 
 a step of providing a plurality of representative points in a reference frame;    a step of determining a plurality of corresponding points in another frame, so as to correspond to each of the representative points; and    a step of obtaining a positional relationship between the representative and corresponding points, wherein    a position of a corresponding point in another frame is predicted according to the positional relationship between the representative and corresponding points, so as to limit an area for searching the corresponding point in another frame.    
     
     
         5 . A method for calculating depth information according to    claim 3   , further comprising: 
 a step of providing a plurality of representative points in a reference frame;    a step of determining a plurality of corresponding points in another frame, so as to correspond to each of the representative points; and    a step of obtaining a positional relationship between the representative and corresponding points;    each of the representative points being classified either as a characteristic or a non-characteristic point; wherein    when more than a predetermined number of characteristic points move between the reference and other frames, such that a total of their movements exceeds a predetermined value, it is judged that a movement between the reference and the other frame is appropriately large, and the reference and the other frames are therefore selected.    
     
     
         6 . A method for calculating depth information according to    claim 3   , further comprising: 
 a step of providing a plurality of representative points in a reference frame;    a step of determining a plurality of corresponding points in another frame, so as to correspond to each of the representative points; and    a step of obtaining a positional relationship between the representative and corresponding points;    the representative points being classified either as a characteristic or a non-characteristic point, wherein    when more than a predetermined number of the characteristic points move between the reference and other frames such that a variance of their movements exceeds a predetermined value, it is judged that a movement between the reference and other frames is appropriately large, and the reference and other frames are therefore selected.    
     
     
         7 . A method for calculating depth information according to any one of    claim 3   , wherein 
 the calculation of the depth information is discontinued when the two frames with an appropriately large movement between them cannot be selected from the 2-D motion image.    
     
     
         8 . A method for calculating depth information according to any one of    claim 5   , wherein 
 a corresponding point of a representative point relating to an image area having a geometric characteristic, is adjusted to be positioned such that an image area related thereto retain the geometric characteristic.    
     
     
         9 . A method for calculating depth information according to    claim 8   , wherein 
 the image area having a geometric characteristic is an area including a straight line.    
     
     
         10 . A method for calculating depth information, comprising: 
 a step of providing a plurality of representative points in a reference frame;    a step of conducting evaluation of similarity of images between an image area including specific points which are arbitrarily set in another frame, and a nearby image area including the representative points in the reference frame;    a step of evaluating relative positional acceptability among the specific points;    a step of determining the specific points as corresponding points of the representative points when both evaluations provide favorable results;    a step of conducting a search for a best point where both evaluations provide a most favorable results, while moving one of the corresponding points with all the other corresponding points fixed at their current positions;    a step of conducting a positional change of the one of the corresponding points to the best point, which was found during the search;    a step of sequentially conducting the search and positional change with respect to all of the corresponding points; and    a step of calculating the depth information according to a positional relationship between the representative points and the corresponding points, the corresponding points having been determined through a series of the above steps.    
     
     
         11 . A method for calculating depth information according to    claim 10   , wherein 
 after the search and positional change is conducted for all of the corresponding points, positional accuracy thereof is improved by solving an Euler-Lagrange differential equation indicates a condition where a combined value of both of the evaluations is an extremum.    
     
     
         12 . A method for clculating depth information according to any one of    claim 10   , wherein 
 the evaluation on similarity of images is conducted by biased block matching where the similarity is correctly evaluated to be highest when the blocks including the identical object are tested, regardless of shooting conditions, and    relative positional acceptability of images is evaluated using the function of distance between neighboring image parts, and,    the results of the above evaluations are treated in terms of distance in the color space and the pixel space respectively so that they can be combined together and can be used for the evaluation in determining corresponding points.    
     
     
         13 . A method for calculating depth information according to    claim 12   , wherein 
 the evaluation of similarity of images is conducted by biased block matching within the limited correction range, which has been pre-determined.    
     
     
         14 . A method for calculating depth information, comprising: 
 a step of providing a plurality of representative points in a reference frame;    a step of determining a plurality of corresponding points in another frame, so as to correspond to each of the representative points; and    a step of obtaining a positional relationship between at least a characteristic point among the representative points and its corresponding point, wherein    a point whose position moves steadily among a plurality of frames shot at different times is selected as the characteristic point.    
     
     
         15 . A method for calculating depth information, comprising: 
 a step of providing a plurality of representative points in a reference frame;    a step of determining a plurality of corresponding points in another frame, so as each to correspond to each of the representative points; and    a step of obtaining a positional relationship between at least a characteristic point among the representative points and its corresponding point, wherein    a point is selected as the characteristic point, whose displacement is substantially consistent between frames simultaneously shot, and further substantially consistent or changes substantially consistently in between other frames simultaneously shot at a close but different time.    
     
     
         16 . A method for calculating depth information, comprising: 
 a step of providing a plurality of representative points in a reference image;    a step of determining a plurality of corresponding points in another image, so as to correspond to each of the representative points, respectively;    a step of obtaining a positional relationship between the representative and corresponding points; and    a step of calculating depth information according to the positional relationship, wherein    the calculation of the depth information is discontinued when less than a predetermined number of characteristic points are selected from the representative points.    
     
     
         17 . A method for calculating depth information according to    claim 16   , wherein 
 the representative and corresponding points are respectively provided in two frames which are included in a 2-D motion image.    
     
     
         18 . A method for calculating depth information, based on two frames included in a 2-D motion image, wherein 
 the calculation of the depth information is discontinued when movement is small between the two frames.    
     
     
         19 . A method for calculating depth information of a 2-D image, wherein 
 when a depth of any point in a certain image is calculated as negative, the depth is interpolated by using depth information of points close-by having positive depth values.    
     
     
         20 . A method for image processing using depth information, comprising, 
 a step of creating a stereo image by giving a parallax to a 2-D image according to its depth information, wherein    the parallax is transformed so as to fall within a predetermined range, so that the stereo image is created according to the transformed parallax.    
     
     
         21 . A method for image processing according to    claim 20   , wherein 
 the parallax is linearly compressed so as to fall within the predetermined range having a point arbitrarily determined as a middle value of the predetermined range.    
     
     
         22 . A method for image processing according to    claim 20   , wherein 
 the parallax off the predetermined range is uniformly transformed to be a closer value of either an upper or a lower limitation value of the predetermined range.    
     
     
         23 . A method for image processing according to    claim 20   , wherein 
 the parallax is non-linearly transformed such that transformed values smoothly converge to upper and lower limitation values of the predetermined range, to thereby fall within the predetermined range.    
     
     
         24 . A method for image processing using depth information, comprising, 
 a step of creating a stereo image by giving a parallax to a 2-D image according to its depth information, wherein    the parallax is variable, which is originally determined according to the depth information.    
     
     
         25 . A method for image processing using depth information, comprising: 
 a step of creating a stereo image by giving a parallax to a 2-D image according to its depth information; and    a step of displaying the stereo image on a stereo image display apparatus, wherein    a process to be conducted on the 2-D image so as to cause the parallax is determined according to a display condition unique to the stereo image display apparatus.    
     
     
         26 . A method for image processing according to    claim 25   , wherein 
 the display condition is determined according to a size of a display screen of the stereo image display apparatus and an assumed distance from the display screen to a viewer, and    the process to be conducted on the 2-D image so as to cause a desired parallax is determined individually based on the display condition thus determined.    
     
     
         27 . A method for image processing using depth information, comprising, 
 a step of creating a stereo image by giving a parallax for every image part of a 2-D image according to its depth information, wherein    an uneven image frame outline caused by the parallax which has been given is corrected.    
     
     
         28 . A method for image processing using depth information, comprising, 
 a step of creating a stereo image by giving a parallax for every image part of a 2-D image according to its depth information, wherein    a desired shape of an image frame is achieved by cutting off a peripheral part of the image.    
     
     
         29 . A method for conducting an image processing to a 2-D image according to its depth information, wherein 
 an image area subject to the image processing is determined, based on the depth information.    
     
     
         30 . A method for image processing according to    claim 29   , wherein 
 the image processing is a process for changing a size of the image area.    
     
     
         31 . A method for image processing according to    claim 30   , wherein 
 the process for changing a size of the image area is a process for expanding a size of an image area with a smaller depth so as to be relatively larger than a size of an image area with a larger depth.    
     
     
         32 . A method for image processing according to    claim 29   , wherein 
 the image processing is a process for conducting separation of a desired image area.    
     
     
         33 . A method for image processing according to    claim 32   , wherein 
 the separation is conducted with respect to an image area having a depth within a predetermined range.    
     
     
         34 . A method for image processing according to    claim 32   , wherein 
 an image area which has been separated is combined with another image.    
     
     
         35 . A method for conducting the image processing with respect to a 2-D image according to its depth information, wherein 
 images with viewpoints at a plurality of points on a hypothetical moving path, where a shooting point of the 2-D image is hypothetically moved, are created for use as a slow motion image, based on the depth information.