System and process for viewing and navigating through an interactive video tour

Abstract

A system and process for providing an interactive video tour of a tour site to a user is presented. In general, the system and process provides an image-based rendering system that enables users to explore remote real world locations, such as a house or a garden. The present approach is based directly on filming an environment, and then using image-based rendering techniques to replay the tour in an interactive manner. As such, the resulting experience is referred to as Interactive Video Tours. The experience is interactive in that the user can move freely along a path, choose between different directions of motion at branch points in the path, and look around in any direction. The user experience is additionally enhanced with multimedia elements such as overview maps, video textures, and sound.

1 - 46 . (canceled)  
   
   
       47 . In a computer system having a user interface comprising a display and user interface selection device, a process comprising: 
 an inputting step for inputting data associated with the video tour, said inputting step comprising, 
 a video data inputting step for inputting video data comprising a sequence of video frames each of which represents a panoramic view of the tour site from a particular location along said path, and  
 a scene data inputting step for inputting scene data, which given a viewing direction and optionally a user-specified direction of motion along the tour path, is used to determine which video frame and what part of that video frame is needed to render and display the next image of the video tour;  
   a displaying step for displaying a viewing and navigation window on the display, wherein the window comprises a plurality of sectors for viewing and controlling the video tour; and    an implementing step for implementing commands entered by a user directly via said selection device and via the user selecting displayed items within the viewing and navigation window sectors using said selection device, so as to give the user continuous control over both the viewing position along a path through the tour site and viewing direction at points on the path.    
   
   
       48 . The process of  claim 47 , wherein the path through the tour site has at least one branch point from each of which the path either turns an abrupt corner or diverges in more than one direction thereby allowing the user to follow any part of the path extending away from the branch, and wherein the scene data inputting step comprises a branch point inputting step for inputting a branch points table which for each video frame associated with a branch point identifies the next video frame that is to be played in view of the current user-specified viewing direction.  
   
   
       49 . The process of  claim 48 , wherein the branch point inputting step comprises a defining step for defining a range of viewing directions for each part of the path diverging from each branch point such that when a user specifies a viewing direction within the defined range for a part of the path extending away from a viewing location that is a branch point, the frame associated with the first specifiable location along that part of the path from the branch point is identified as the next video frame that is to be played.  
   
   
       50 . The process of  claim 47 , wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a portion of the frame associated with a current viewing position which corresponds to a prescribed field of view around a current viewing direction, and wherein the scene data inputting step comprises a frame rotation inputting step for inputting a frame rotation table which for each video frame identifies a global orientation direction corresponding to a direction designated when the frame was captured as a zero degree line from which the user-specified viewing direction is measured thereby facilitating finding the portion of the frame to be displayed.  
   
   
       51 . The process of  claim 50 , wherein the plurality of viewing and navigation window sectors comprises a sector for displaying an icon representing an object of interest residing in the tour site which when selected by the user causes additional information about the object to be displayed, and wherein the scene data inputting step further comprises an object information inputting step for inputting an object information table that for a prescribed set of frame ranges and viewing directions associated with the frames of those frame ranges in which the object of interest appears, identifies the particular icon that is to be displayed and link details needed to access said information about the object.  
   
   
       52 . The process of  claim 50 , wherein displaying a portion of the frame associated with the current viewing position which corresponds to a prescribed field of view around the current viewing direction comprises displaying a replaceable object within the displayed frame portion, and wherein the scene data inputting step comprises an object replacement information inputting step for inputting an object replacement information table that for a prescribed set of frame ranges and viewing directions associated with the frames of those frame ranges in which the replaceable object appears, identifies a menu that is to be displayed in the viewing and navigation window comprising a list of replacement objects and link details for each replacement object listed that are needed to access said replacement objects which are to be displayed in place of the replaceable object upon selection of that replacement object from the menu by the user.  
   
   
       53 . The process of  claim 50 , wherein displaying a portion of the frame associated with the current viewing position which corresponds to a prescribed field of view around the current viewing direction plurality comprises displaying a replaceable object within the displayed frame portion, and wherein the scene data inputting step comprises an object replacement information inputting step for inputting an object replacement information table that for a prescribed set of frame ranges and viewing directions associated with the frames of those frame ranges in which the replaceable object appears, identifies link details for a group of replacement objects that are needed to access each of said objects, which are displayed one at a time in place of the replaceable object in a prescribed cyclical order upon entry of a change command by the user.  
   
   
       54 . The process of  claim 47 , wherein the method employed to capture the video frames introduced jitter, and wherein the scene data inputting step comprises a frame rotation inputting step fo inputting a frame rotation table which for each video frame identifies a jitter correction factor thereby allowing the jitter to be removed from each frame prior to a portion thereof being rendered and displayed.  
   
   
       55 . The process of  claim 47 , wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a map graphic of the tour site, and wherein the scene data inputting step comprises a map region inputting step for inputting a map region table which for each of a prescribed set of frame ranges identifies the particular map graphic that is to be displayed whenever a portion of a video frame within that frame range is rendered and displayed.  
   
   
       56 . The process of  claim 47 , wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a name associated with a region of the tour site that the user is currently viewing, and wherein the scene data inputting step comprises a map region inputting step fo inputting a map region table that for each of a prescribed set of frame ranges, each of which is associated with a particular region of the tour site, identifies the name associated with that region.  
   
   
       57 . The process of  claim 47 , wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a map graphic of the tour site that comprises a line representing the tour path and an indicator which identifies the point on the tour path from which the user is currently viewing the tour site, and wherein the scene data inputting comprises a path inputting step for inputting a path table which for each video frame identifies a location on the map graphic representing the point on the tour path that the frame was captured thereby facilitating the placement of said indicator on the map graphic.  
   
   
       58 . The process of  claim 47 , wherein the plurality of viewing and navigation window sectors comprises a sector for displaying a map graphic of the tour site that comprises a line representing the tour path, and wherein the scene data inputting step comprises a path inputting step for inputting a path table which for each video frame identifies a location on the map graphic representing the point on the tour path that the frame was captured, and wherein whenever the user selects a point on the tour path line of the map graphic, the next frame from which a portion is rendered and displayed is the frame identified in the path table as corresponding to the user-selected path point, thereby allowing the user to view another part of the tour site without having to follow the tour path to that part.  
   
   
       59 . A computer-implemented process for providing an interactive video tour of a tour site to a user, comprising: 
 (a) a video data inputting step for inputting video data comprising a sequence of video frames each of which represents a panoramic view of the tour site from a particular location along a path through the tour site;    (b) a scene data inputting step for inputting scene data which comprises information to identify the frame associated with each specifiable location along the path, and that given a viewing direction and a user-specified direction of motion along the tour path, is used to determine which video frame and what part of that video frame is needed to render and display the next image of the video tour;    (c) for a given viewing position within the site along the path, an identifying step for identifying the frame associated with that position using the scene data and designating it as the current frame;    (d) a direction inputting step for inputting a current user-specified viewing direction, and a current user-specified direction of motion along the tour path, if one;    (e) an obtaining step for obtaining from the video data that portion of the current frame which corresponds to a prescribed field of view around the current viewing direction;    (f) a rendering step for rendering the obtained portion of the current frame and displaying it to the user, while simultaneously identifying the frame needed to display the next image of the video tour in view of the user-specified direction of motion along the tour path using the scene data;    (g) a designating step for designating the newly identified frame as the current frame in lieu of the previously identified frame; and    (h) a repeating step for repeating steps (d) through (g) for as long as the user wants to view the video tour.    
   
   
       60 . The process of  claim 59 , wherein the video data is compressed using a compression method which allows the decompression of any frame, not just the next sequential frame, and which allows any portion of a frame to be decompressed without having to decompress the rest of the frame, and wherein the obtaining step comprises a decompressing step for exclusively decompressing just said portion of the current frame.  
   
   
       61 . The process of  claim 59 , wherein the scene data comprises a global orientation direction for each video frame which corresponds to a direction designated when the frame was captured as a zero degree line from which the viewing direction is measured, and wherein the obtaining step comprises a frame portion identifying step for identifying said portion of the current frame which corresponds to the prescribed field of view around the current viewing direction by measuring the viewing direction from the global orientation direction.  
   
   
       62 . The process of  claim 61 , wherein the method employed to capture the video frames introduced jitter, and wherein the scene data comprises a jitter correction factor for each video frame, and wherein the obtaining step further comprises: 
 a jitter correction factor obtaining step for obtaining the jitter correction factor associated with the current frame from the scene data; and    an employing step for employing said jitter correction factor to remove the jitter from said portion of the current frame.    
   
   
       63 . The process of  claim 59 , wherein the identifying step comprises a previously designated frame designating step for designating the previously designated frame as the newly designated frame whenever no direction of motion is specified.  
   
   
       64 . The process of  claim 59 , wherein the path through the tour site has at least one branch point from each of which the path either turns an abrupt corner or diverges in more than one direction such that the user can follow any part of the path extending away from the branch, and wherein the scene data comprises a branch points table which for each video frame associated with a branch point identifies the next video frame that is to be played in view of the current viewing direction, and wherein identifying step comprises: 
 a determining step for determining from the scene data whether the current frame is associated with a branch point; and    whenever the current frame is not associated with a branch point, 
 a next sequential frame identifying step for identifying the next sequential frame of the video data as the newly identified frame if the user has specified a forward motion direction as related to what was the forward direction when the video frames were captured, and  
 a previous sequential frame identifying step for identifying the previous sequential frame of the video data as the newly identified frame if the user has specified a backwards motion direction as related to what was the forward direction when the video frames were captured.  
   
   
   
       65 . The process of  claim 61 , wherein the path through the tour site has at least one branch point from each of which the path either turn an abrupt corner or diverges in more than one direction such that the user can follow any part of the path extending away from the branch, and wherein the scene data comprises a branch points table which for each video frame associated with a branch point identifies the next video frame that is to be played in view of the current viewing direction, and wherein identifying step comprises: 
 a determining step for determining from the scene data whether the current frame is associated with a branch point; and    whenever the current frame is associated with a branch point, a second frame identifying step for identifying the frame of the video data specified in the scene data as the next frame for the current viewing direction as the newly identified frame.    
   
   
       66 . The process of  claim 65 , wherein a substantial divergence in the global orientation direction may exist between a frame associated with a branch point and the frame designated as the newly identified frame, and wherein the obtaining step comprises: 
 an estimating step for estimating the relative rotation between the frames using the global orientation information for those frames from the scene data; and    a correcting step for correcting for any mismatch in order to minimize a visual jump that the user can see when the tour image associated with the frame portion is displayed.