Image transformation and synthesis methods

Abstract

A system for generating images of a scene as the scene would be observed from an arbitrary location. A plurality of discrete images, typically video images, taken at different viewpoints, as, for example, by a plurality of cameras pointing outwardly on a curving locus are converted to an offset epipolar image. The offset epipolar image includes a plurality of linesets, each such lineset incorporating one scanning line from each of the discrete video images. Each line in the virtual image is reconstructed from a lineset of the epipolar image. The reconstruction may include interpolation between pixel data representing lines from adjacent discrete images and mapping of pixels from one or more lines representing one or more adjacent discrete images onto the pixel line of the virtual image. The nature of the mapping depends upon the viewpoint selected for the virtual image. The system can provide real time stereoscopic telepresence, i.e., a virtual viewpoint images for each eye as the observer moves his or her head.

What is claimed is:  
     
         1 . A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint comprising the steps of: 
 (a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewpoints, each said discrete image including an array of pixel data in first and second image dimensions;    (b) constructing a first epipolar image for said first dimension from said discrete images, said first epipolar image including a plurality of linesets, each said lineset including one line of pixel data in said first dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second dimension, said lines of pixel data within each lineset being ordered in an order corresponding to the order of said discrete viewpoints;    (c) providing pixel data for said synthetic image as a plurality of virtual viewpoint pixel lines extending in said first image dimension and offset from one another in said second image dimension by (i) associating each virtual viewpoint line with a line set in said first epipolar image corresponding to the location of such line in said second dimension and (ii) for each pixel within each virtual viewpoint line, deriving synthetic pixel data from pixel data in the associated line set.    
     
     
         2 . A method as claimed in    claim 1    wherein said step of deriving synthetic pixel data includes the steps of selecting a plurality of lines within the associated line set corresponding to discrete viewpoints in the vicinity of said virtual viewpoint and deriving the synthetic pixel data from the pixel data in the so-selected lines.  
     
     
         3 . A method as claimed in    claim 2    wherein said step of deriving synthetic pixel data for each synthetic pixel includes the step of choosing a plurality of pixels in the selected lines adjacent to the position of the synthetic pixel in said first dimension and deriving the synthetic pixel data from the pixel data in said chosen pixels of said selected lines.  
     
     
         4 . A method as claimed in    claim 3    wherein said selected lines in the line set include lines corresponding to discrete viewpoints bracketing the virtual viewpoint.  
     
     
         5 . A method as claimed in    claim 4    wherein said step of deriving pixel data for each synthetic pixel includes the step of interpolating the pixel data of the chosen pixels in said selected lines.  
     
     
         6 . A method as claimed in    claim 3    wherein for each synthetic pixel in the virtual viewpoint image line, said step of selecting said pixels in said selected lines includes the steps of, (i) setting an offset distance, (ii) pixels in the bracketing lines offset from one another in said first direction by said offset distance and bracketing the first direction location of the synthetic image pixel, and (iii) testing said chosen pixels of said bracketing lines to determine if the pixel data in said chosen pixels of said bracketing lines match one another within a preselected limit and, if not, repeating steps (i) through (iii) with a different offset distance on each repetition until such pixel data matches in step (iii).  
     
     
         7 . A method as claimed in    claim 6    further comprising the step of providing initial offset information for said offset epipolar image specifying an initial offset in said first dimension between each pair of adjacent lines in each line set, said step of setting said offset distance including the step of initially setting said offset distance to equal said initial offset.  
     
     
         8 . A method as claimed in    claim 7    wherein said initial offset corresponds to the theoretical offset between pixels in adjacent lines of the line set representing a feature in said scene positioned at infinite distance from said viewpoints.  
     
     
         9 . A method as claimed in    claim 8    wherein said step of adjusting said offset distance is performed so as to increase said offset, distance on each repetition.  
     
     
         10 . A method as claimed in    claim 1    wherein said step of deriving pixel data from the pixel data in other pixels of the associated line set includes the step of processing the pixel data in such line set to derive boundaries between regions of the line set having different pixel data corresponding to different objects, each such boundary defining a curve in an epipolar plane having a first epipolar coordinate corresponding to pixel location in said first dimension and a second epipolar coordinate corresponding to viewpoint location, assigning each synthetic pixel to an object based upon the location of such pixel relative to said boundaries within said epipolar plane, and deriving the synthetic pixel data for each synthetic pixel from discrete image pixel data in said line set representing the same object.  
     
     
         11 . A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location and viewing direction and having a predetermined field of view, the method comprising the steps of: 
 (a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete view directions from a plurality of discrete viewpoints on a predetermined viewpoint locus, said viewing directions being disposed at different angles relative to a reference line in said first dimension, each said discrete image including an array of pixel data in said first image dimension and in a second image dimension orthogonal thereto, whereby position of each pixel in said first image dimension within each image will represent the angle between the viewing direction of the image and a ray direction from said pixel to a point in the scene;    (b) constructing a first epipolar image from said discrete images, said first epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to the order of said viewing directions relative to said first dimension, whereby each said line set defines an epipolar plane having a first epipolar coordinate corresponding to viewing direction and having a second epipolar coordinate corresponding to ray azimuth relative to an index line;    (c) selecting a base viewpoint on said viewpoint locus; and    (d) forming a line of the virtual viewpoint image from each said line set by (i) providing a base line within the line set corresponding to the base viewpoint; (ii) mapping pixel data of the base line into the virtual viewpoint image line; (iii) selecting supplementary pixels from one or more additional lines of the line set, said supplementary pixels being adjacent to one or both ends of the base line in said epipolar coordinates and (iv) incorporating data from said supplementary pixels at one or both ends of the virtual viewpoint image line so that the virtual viewpoint image line includes pixel data for said predetermined field of view.    
     
     
         12 . A method as claimed in    claim 11    wherein said viewpoint locus is curvilinear, said discrete viewing directions pointing across said locus from a camera side towards an object side thereof.  
     
     
         13 . A method as claimed in    claim 12    wherein the view direction for each said discrete viewpoint is substantially orthogonal with respect to said locus.  
     
     
         14 . A method as claimed in    claim 12    wherein said camera side is the interior of the locus, adjacent the center of curvature thereof, and said viewing directions for said discrete viewpoints point outwardly, away from the center of curvature.  
     
     
         15 . A method as claimed in    claim 14    wherein said virtual viewpoint is recessed inwardly toward the center of curvature of the locus, said step of selecting supplementary pixels being performed so that at least some of the supplementary pixels correspond to ray directions substantially parallel to the ray azimuth of the end pixel of the base line at one or both ends thereof.  
     
     
         16 . A method as claimed in    claim 15    wherein said step of mapping said pixels from said base line to said virtual viewpoint line includes the step of compressing the pixel data in said base line into a smaller number of pixels and mapping said smaller number of pixels onto only a portion of said virtual viewpoint line.  
     
     
         17 . A method as claimed in    claim 14    wherein said virtual viewpoint has a view direction skewed from the base view direction, said step of selecting supplementary pixels so that the ray azimuths of the supplementary pixels added at an end of the base line vary progressively in the direction of skew.  
     
     
         18 . A method as claimed in    claim 14    wherein said step of selecting supplementary pixels is performed by selecting the pixel having each ray azimuth from the additional line which has the view direction closest to the base view direction of all lines incorporating pixels at such ray direction.  
     
     
         19 . A method as claimed in    claim 11    further comprising the step of providing a lookup table setting forth a plurality of viewpoints and view directions and pixel location data specifying supplementary pixels for each such viewpoint and view direction, said step of selecting said supplementary pixels including the steps of retrieving the pixel virtual viewpoint and virtual view direction from said lookup table and selecting the supplementary pixels in accordance with said location data.  
     
     
         20 . A method as claimed in    claim 19    wherein said step of retrieving location data includes the step of interpolating between location data for adjacent viewpoints and view directions.  
     
     
         21 . A method as claimed in    claim 11    wherein said base viewpoint is located between two of said discrete viewpoints and wherein said step of providing said base line in each said line set includes the step of deriving pixel data for pixels constituting said base line from pixel data in lines of said line set corresponding to discrete viewpoints in the vicinity of said base viewpoint.  
     
     
         22 . A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location and viewing direction, the method comprising the steps of: 
 (a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewing directions from a plurality of discrete viewpoint locations on a predetermined viewpoint locus, said viewing directions being disposed at different angles relative to a reference line in a first dimension, each said discrete image including an array of pixel data in a first image dimension and in a second image dimension orthogonal thereto, whereby position of each pixel in said first image dimension within each image will represent the angle between the viewing direction of the image and a ray direction from said pixel to a point in the scene;    (b) constructing a first epipolar image from said discrete images, said first epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to the order of said viewing directions relative to said first dimension, whereby each said line set defines an epipolar plane having a first epipolar coordinate corresponding to viewing direction and having a second epipolar coordinate corresponding to ray azimuth in said first dimension relative to said reference line;    (c) forming a line of the virtual viewpoint image from each said line set, each such line including a plurality of pixels each having a view azimuth in said first dimension, by selecting a plurality of pixel sets in the virtual viewpoint line, each such pixel set including one pixel or a plurality of mutually adjacent pixels, each such pixel set defining a principal ray line extending from the virtual viewpoint location at a principal view azimuth close to the view azimuths of the pixels in such pixel set and, for each said pixel set: (i) providing a viewpoint on said locus as a base viewpoint such that an intercept of the principal ray line of the pixel set on the viewpoint locus is adjacent the base viewpoint; (ii) selecting a base line within the line set corresponding to the base viewpoint; and (iii) for each pixel in the set, selecting one or more pixels of the base line having ray azimuths close to the view azimuth of the pixel in the set and mapping pixel data of the selected pixels of the base line into such pixel of the set.    
     
     
         23 . A method as claimed in    claim 22    wherein each said pixel set includes only one pixel of the virtual viewpoint image line, and wherein the principal view azimuth of each set is the view azimuth of the pixel constituting such set.  
     
     
         24 . A method as claimed in    claim 23    further comprising the step of providing a lookup table setting forth a plurality of virtual viewpoint location and view azimuths, and location data specifying one or more pixels within a base line for each such virtual viewpoint location and view azimuth, said step of selecting said base line and said pixels in said base line for each pixel set including the steps of selecting the pixels in accordance with said location data for the virtual viewpoint and view azimuth of each pixel.  
     
     
         25 . A method as claimed in    claim 22    wherein said viewpoint locus is curvilinear, said discrete viewing directions pointing across said locus from a camera side towards an object side thereof.  
     
     
         26 . A method as claimed in    claim 25    wherein the view direction for each said discrete viewpoint is substantially orthogonal to said locus.  
     
     
         27 . A method as claimed in    claim 25    wherein said camera side is the interior of the locus, adjacent the center of curvature thereof, and said viewing directions for said discrete viewpoints point outwardly, away from the center of the curvature.  
     
     
         28 . A method as claimed in    claim 22    further comprising the step of providing a lookup table setting forth a plurality of viewpoint location and principal view directions and base view location data specifying base views for each such virtual viewpoint location and principal view direction, said step of selecting said base view for each pixel set including the step of selecting the base view from said lookup table in accordance with said base view location data.  
     
     
         29 . A method as claimed in    claim 22    wherein said base viewpoint for at least some pixel sets is located between two of said discrete viewpoints and wherein, for each said base viewpoint, said step of providing said base line in each said line set includes the step of deriving pixel data for pixels constituting such base line from pixel data in lines of said line set corresponding to discrete viewpoints in the vicinity of said base viewpoint.  
     
     
         30 . A method as claimed in    claim 22    wherein said step of providing said base line for each pixel set includes selecting the line in the associated line set corresponding to the discrete viewpoint closest to the intercept of the principal ray line on the viewpoint locus and providing the so-selected line as pixel data of the base line.  
     
     
         31 . A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location, the method comprising the steps of: 
 (a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete view directions from a plurality of discrete viewpoint locations on a two-dimensional viewpoint locus, each said discrete view direction defining a view direction vector, each said discrete image including pixel data for pixels in an array extending in a first image dimension and in a second image dimension orthogonal thereto, whereby position of each pixel within each image will represent a ray offset vector between the view direction vector of the discrete image and a ray direction vector from said pixel through the discrete viewpoint to a point in the scene;    (b) assigning virtual image pixel data for each pixel in a two-dimensional array of virtual pixels by selecting a plurality of pixel sets in the virtual array, each such pixel set including one pixel or a plurality of mutually adjacent pixels, each said pixel set defining a principal ray vector extending from the virtual viewpoint location in a principal view direction close to the ray vector directions of the pixels in such pixel set and, for each said pixel set: (i) determining an intercept of the principal ray vector on the viewpoint locus; (ii) providing an image on said locus close to said intercept as a base image for the pixel set; and (iii) for each virtual pixel in the set, selecting one or more pixels of the base image having ray directions close to the view direction of the virtual pixel and mapping pixel data of the selected pixels of the base image into such pixel of the set    
     
     
         32 . A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location, the method comprising the steps of: 
 (a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete view directions from a plurality of discrete viewpoints distributed in two orthogonal directions on a viewpoint locus;    (b) transforming said plurality of discrete images into a plurality of three-dimensional transform images each including some pixel data from each of the discrete images;    (c) selecting information from a plurality of said three-dimensional transform images according to a selection scheme based upon said selected view location and combining the so-selected information; and    (d) mapping the selected information from the transform images into the virtual viewpoint image.    
     
     
         33 . A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location, the method comprising the steps of: 
 (a) providing a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete view directions from a plurality of discrete viewpoints distributed in two orthogonal dimensions of said scene on a viewpoint locus, each said discrete view defining a view location vector having a first component in an azimuth dimension and a second component in an elevation dimension orthogonal to the azimuth direction, each said discrete image including pixel data for pixels in an array extending in first and second image dimensions orthogonal to one another and to the view location vector, whereby the position of each pixel in the first and second image directions correspond to the difference in azimuth and elevation, respectively, between the view location vector of the discrete image and a ray direction vector from said pixel through the discrete viewpoint to a point in the scene depicted by said pixel;    (b) constructing a first epipolar image from said discrete images, said first epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered so that each said line set defines an epipolar space having a first epipolar coordinate corresponding to the azimuth component of the view location vector, having a second epipolar coordinate corresponding to the elevation component of the view location vector and also corresponding to the elevations of the ray direction vectors of the pixels, and having a third epipolar coordinate corresponding to the azimuth of the ray direction vectors of the pixels and;    (c) assigning virtual image pixel data for each pixel in a two-dimensional array of virtual pixels by selecting a plurality of pixel sets in the virtual array, each such pixel set including one pixel or a plurality of mutually adjacent pixels, each said pixel in the set defining a ray direction vector from the pixel through the virtual viewpoint location and having azimuth and elevation, each said pixel set defining a principal ray vector extending through the virtual viewpoint and having azimuth and elevation close to the azimuths and elevations of the pixels in such pixel set and, for each said pixel set: (i) determining an intercept of the principal ray vector on the viewpoint locus; (ii) providing a line of one said lineset having first and second epipolar coordinates corresponding to a location close to said intercept as a base line for the pixel set; (iii) for each virtual pixel in the set, selecting one or more pixels of the base line having a third epipolar coordinate close to the ray direction azimuth of the virtual pixel; and (iv) mapping pixel data of the selected pixels of the base image into such pixel of the pixel set.    
     
     
         34 . A method as claimed in    claim 33    wherein each said pixel set includes only one pixel of the virtual viewpoint image, and wherein the principal ray vector of each pixel set is the ray direction vector of the pixel constituting such set.  
     
     
         35 . A method as claimed in    claim 34    further comprising the step of providing a lookup table setting forth a plurality of virtual viewpoint locations pixel locations in said first and second image dimensions, and location data specifying one or more pixels within a base line for each such virtual viewpoint location and pixel location, said step of selecting said base line and said pixels in said base line for each pixel set including the steps of and selecting the pixels in accordance with said location data for the virtual viewpoint and pixel location of each pixel.  
     
     
         36 . A method as claimed in    claim 33    wherein said viewpoint locus is a sphere or a portion of a sphere, said discrete viewpoint directions being substantially radial with respect to said sphere or portion of a sphere.  
     
     
         37 . A method as claimed in    claim 33    further comprising the step of providing a lookup table setting forth a plurality of viewpoint location and principal view directions and base view location data specifying base views for each such virtual viewpoint location and principal view direction, said step of selecting said base view for each pixel set including the step of selecting the base view from said lookup table in accordance with said base view location data.  
     
     
         38 . A method as claimed in    claim 33    wherein said base viewpoint for at least some pixel sets is located between a plurality of said discrete viewpoints and wherein, for each such base viewpoint, said step of providing said base line in each said line set includes the step of deriving pixel data for pixels constituting such base line from pixel data in lines of said line set corresponding to discrete viewpoints in the vicinity of said base viewpoint.  
     
     
         39 . A method as claimed in    claim 33    wherein said step of providing said base line for each pixel set includes selecting the line in the associated line set corresponding to the discrete viewpoint closest to the intercept of the principal ray line on the viewpoint locus and providing the so-selected line as pixel data of the base line.  
     
     
         40 . A method as claimed in any one of claims  1 ,  11 ,  22 ,  31 ,  32  or  33  wherein said step of providing said discrete images includes the steps of capturing images of a real scene by means of one or more cameras and correcting each said captured image for distortion introduced by the camera.  
     
     
         41 . A method as claimed in    claim 40    wherein said step of capturing images by means of one or more cameras includes the steps of moving said one or more cameras with respect to the scene and capturing different discrete images at different positions of said one or more cameras.  
     
     
         42 . A method of providing telepresence comprising the step of detecting the disposition of a real observer as the observer moves, selecting at least one virtual viewpoint location and direction corresponding to a viewpoint location and view direction of the real observer, synthesizing a virtual viewpoint image by a method as claimed in any one of claims  1 ,  11 ,  22 ,  31 ,  32  and  33  for each selected virtual viewpoint location and direction and displaying the virtual viewpoint image to the observer substantially in real time, so that the observer sees the correct virtual viewpoint image for a new disposition substantially immediately as he moves to the new disposition.  
     
     
         43 . A method as claimed in    claim 42    wherein said step of selecting at least one virtual viewpoint location and direction includes the step of selecting a pair of virtual viewpoint locations offset from one another by an interpupillary distance, said steps of synthesizing and displaying being conducted so as to display a binocular pair of images, one to each eye of the observer.  
     
     
         44 . A method as claimed in    claim 42    wherein said step of selecting at least one virtual viewpoint image includes the step of selecting a plurality of virtual viewpoints simultaneously corresponding to the locations of a plurality of viewers, said synthesizing step including the step of synthesizing a virtual viewpoint image for each said virtual viewpoint and said displaying step including the step of displaying each said virtual viewpoint image to the associated observer so that each observer sees one or more virtual viewpoint images associated with his position substantially in real time as he moves.  
     
     
         45 . A method of compressing a set of images of a scene including a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewpoints, each said discrete image including an array of pixel data arranged in a first image dimension corresponding to position of depicted objects in a first dimension in real space and in a second image dimension orthogonal thereto, the method including the steps of: 
 (a) constructing one or more epipolar images from said discrete images, each said epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to an order of said viewpoints in said first real dimension; and    (b) compressing the pixel data in said line sets to form one or more compressed epipolar images.    
     
     
         46 . A method as claimed in    claim 45    further comprising the step of transmitting or storing said compressed epipolar images.  
     
     
         47 . A method as claimed in    claim 46    further comprising the step of decompressing said compressed epipolar images.  
     
     
         48 . A method as claimed in    claim 45    wherein said step of compressing the pixel data in said line sets includes is performed by compressing the data in each said line set independently of the data in the other said line sets.  
     
     
         49 . A method as claimed in    claim 45    wherein said set of discrete images of said scene includes a plurality of subsets of discrete images representing the scene at different times, all of the images in each said subset representing the scene at the same time, said step of forming one or more epipolar images includes the step of constructing an epipolar images from the discrete images in each said subset, whereby each said epipolar image corresponds on one said time, said compressing step including the step of comparing data in a first said epipolar image with data in at least one other said epipolar image from a different time to determine the differences therebetween.  
     
     
         50 . A method as claimed in    claim 49    wherein said comparing step includes the step of comparing each said line set in said first epipolar image with a corresponding line set in one or more other epipolar images.  
     
     
         51 . A method of combining a first set of images of a first scene and a second set of images of a second scene, each said set of images including a plurality of discrete images corresponding to the images of the scene observed from a plurality of discrete viewpoints, each said discrete image including an array of pixel data arranged in a first image dimension corresponding to position of depicted objects in a first dimension in real space and in a second image dimension orthogonal thereto, the method including the steps of: 
 (a) constructing a first epipolar image from said first set of discrete images and a second epipolar image from said second set of discrete images, each said epipolar image including a plurality of line sets, each said line set including one line of pixel data in said first image dimension from each said discrete image, all of the lines in each said line set corresponding to the same location in said second image dimension, said lines of pixel data within each line set being ordered in an order corresponding to an order of said viewpoints in said first real dimension; and    (b) combining the pixel data in said line sets of said first and second epipolar images to form combined line sets constituting a combined epipolar image.    
     
     
         52 . A method as claimed in    claim 51    wherein said step of combining said pixel data includes the step of combining pixel data of each line set in said first epipolar image with one line set in said second epipolar image.  
     
     
         53 . A method as claimed in    claim 52    wherein said step of combining said pixel data in said line sets includes the step of deriving new pixel data for each pixel in each combined line set by combining pixel data for the corresponding pixel in the line set of the first epipolar image with pixel data for the corresponding pixel in the line set of the second epipolar image according to a combining formula which varies from pixel to pixel.  
     
     
         54 . A method as claimed in    claim 53    wherein said combining formula is a function of the pixel data in said second line set.  
     
     
         55 . A method of providing telepresence comprising the steps of: 
 (a) providing a plurality of discrete two-dimensional images corresponding to the image of the scene observed from a plurality of discrete viewpoints on a predetermined viewpoint locus;    (b) transforming said plurality of discrete images into two-dimensional transform images each including some information from a plurality of said discrete images;    (c) displaying the virtual viewpoint image to the observer substantially in real time, so that the observer sees the correct virtual viewpoint image for a new disposition substantially immediately as he or she moves to the new disposition.    
     
     
         56 . A method as claimed in    claim 55    wherein each said discrete image includes pixel data for a plurality of pixels in a first ordered array, and each said transform image pixel data for a plurality of pixels in a second ordered array said step of transforming said discrete images including the step of incorporating pixel data for a set of pixels from each said discrete image into each said transform image.  
     
     
         57 . A method as claimed in    claim 56    wherein said step of synthesizing said virtual viewpoint image includes the step of selecting pixel data for a set of pixels from each said transform image and incorporating the so-selected pixel data into an ordered array to form said virtual viewpoint image.  
     
     
         58 . A method as claimed in    claim 55    wherein said step of selecting at least one virtual view location and direction includes the step of selecting a pair of virtual view locations offset from one another by an interpupillary distance, said steps of synthesizing and displaying being conducted so as to display a binocular pair of images, one to each eye of the observer.  
     
     
         59 . A method as claimed in    claim 55    wherein said step of detecting the disposition of an observer includes the step of detecting the dispositions of a plurality of observers simultaneously, said synthesizing and displaying steps including the steps of synthesizing and displaying a plurality of virtual viewpoint images simultaneously so that a virtual viewpoint image corresponding to the disposition of each observer is displayed to that observer.