Method and Apparatus for Providing Hybrid Reality Environment

Abstract

Methods for generating a hybrid reality environment of real and virtual objects are provided. One method includes receiving, by a server, data of a first user located at a first physical location and data of a second user located at a second physical location. The second physical location is different from the first physical location. The method includes generating, by the server, a scene of an augmented reality (AR) based environment for the first user. The AR based environment includes images of the first physical location and one or more virtual objects related to the second user that are generated based on the data received from the second user. The method generates, by the server, a scene of an augmented virtuality (AV) based environment for the second user. The AV based environment includes one or more virtual objects related to the second user and one or more virtual objects related to the first user that are generated based on the data received from the first user. The method includes sending data for displaying the scene of the AR based environment to a first device of the first user. The first user can interact with the one or more virtual objects related to the second user in real time within the AR based environment via the first device. The method further sends data for displaying the scene of the AV based environment to a second device of the second user. The second user can control a virtual object related to the second user to interact with the one or more virtual objects related to the first user in real time within the AV based environment via the second device.

What is claimed is: 
     
         1 . A method of generating a hybrid reality environment of real and virtual objects, comprising:
 at a computer system having one or more processors and memory for storing a program to be executed by the one or more processors:   receiving data of a first user located at a first physical location;   receiving data of a second user located at a second physical location, the second physical location being different from the first physical location;   generating, for the first user, a scene of an augmented reality (AR) based environment, the AR based environment including images of the first physical location and one or more virtual objects related to the second user that are generated based on the data received from the second user;   generating, for the second user, a scene of an augmented virtuality (AV) based environment, the AV based environment including one or more virtual objects related to the second user and one or more virtual objects related to the first user that are generated based on the data received from the first user;   sending signals for displaying the scene of the AR based environment to the first user, such that the first user can interact with the one or more virtual objects related to the second user in real time within the AR based environment; and   sending signals for displaying the scene of the AV based environment to the second user, such that the second user can control a virtual object related to the second user to interact with the one or more virtual objects related to the first user in real time within the AV based environment.   
     
     
         2 . The method of  claim 1 , wherein the data of the first user includes data associated with a location, or an orientation, or a posture, or a movement, or a combination of two or more thereof, of the first user. 
     
     
         3 . The method of  claim 1 , wherein the data of the first user includes three-dimensional (3D) data associated with the first user. 
     
     
         4 . The method of  claim 3 , wherein the 3D data comprises data associated with a computer generated 3D geographic map enhanced by texture mapping using earth satellite image data. 
     
     
         5 . The method of  claim 1 , wherein the data of the second user includes face image data of the second user. 
     
     
         6 . The method of  claim 5 , wherein the face image data of the second user includes real-time video and audio data packets extracted from image data captured by a video camera directed toward the second user at the second physical location. 
     
     
         7 . The method of  claim 1 , wherein generating the scene of the AR based environment includes scanning a background environment surrounding the first user using a three-dimensional (3D) depth sensor. 
     
     
         8 . The method of  claim 1 , wherein generating the scene of the AR based environment or the scene of the AV based environment includes generating a virtual object related to the second user by mapping a picture of the second user onto a surface of an AR object associated with the second user. 
     
     
         9 . The method of  claim 1 , wherein generating the scene of the AR based environment includes generating the scene for displaying a movement of a virtual object related to the second user, the movement being controlled by inputs received from the second user in substantial real time. 
     
     
         10 . The method of  claim 1 , wherein generating the scene of the AV based environment includes generating the scene for displaying a movement of a virtual object related to the first user that corresponds to a movement of the first user in substantial real time. 
     
     
         11 . The method of  claim 1 , wherein the scene of the AR based environment displayed to the first user does not include a virtual object related to the first user. 
     
     
         12 . The method of  claim 1 , wherein the scene of the AV based environment displayed to the second user does not include a virtual object related to the second user. 
     
     
         13 . The method of  claim 1 , wherein the scene of the AR based environment displayed to the first user forms at least a part of a first set of scenes and the scene of the AV based environment displayed to the second user forms at least a part of a second set of scenes; and
 sending signals for displaying, to a third user at a third physical location a mixed reality view that includes a combination of at least part of the first set of scenes and at least part of the second set of scenes.   
     
     
         14 . The method of  claim 1 , wherein sending signals for displaying the scene of the AR based environment to the first user includes sending signals to a computer device associated with the first user such that the scene of the AR based environment is displayed on the computer device of the first user. 
     
     
         15 . The method of  claim 1 , wherein sending signals for displaying the scene of the AV based environment to the second user includes sending signals to a computer device associated with the second user such that the scene of the AV based environment is displayed on the computer device of the second user. 
     
     
         16 . The method of  claim 1 , wherein the AV based environment is a virtualized realization of the AR based environment. 
     
     
         17 . A method of generating a hybrid reality environment of real and virtual objects, comprising:
 receiving, by a server, data of a first user located at a first physical location and data of a second user located at a second physical location, the second physical location being different from the first physical location;   generating, by the server, a scene of an augmented reality (AR) based environment for the first user, the AR based environment including images of the first physical location and one or more virtual objects related to the second user that are generated based on the data received from the second user;   generating, by the server, a scene of an augmented virtuality (AV) based environment for the second user, the AV based environment including one or more virtual objects related to the second user and one or more virtual objects related to the first user that are generated based on the data received from the first user;   sending, by the server, data for displaying the scene of the AR based environment to a first device of the first user, such that the first user can interact with the one or more virtual objects related to the second user in real time within the AR based environment via the first device; and   sending, by the server, data for displaying the scene of the AV based environment to a second device of the second user, such that the second user can control a virtual object related to the second user to interact with the one or more virtual objects related to the first user in real time within the AV based environment via the second device.   
     
     
         18 . The method of  claim 17 , further comprising,
 receiving, by the server, data captured by a third device of the first physical location, the first physical location including AR markers;   sending, by the server, data to the third device to augment and display images of the first physical location to include one or more virtual objects related to the first user and one or more virtual objects related to the second user.   
     
     
         19 . The method of  claim 17 , wherein the scene of the AR based environment displayed to the first user forms at least a part of a first set of scenes and the scene of the AV based environment displayed to the second user forms at least a part of a second set of scenes; and
 sending, by the server, data for displaying to a third user at a third physical location a mixed reality view that includes a combination of at least part of the first set of scenes and at least part of the second set of scenes.   
     
     
         20 . The method of  claim 17 , wherein the one or more virtual objects related to the second user include relative orientation and/or position data related to a view into the scene of the AV based environment, and the one or more virtual objects related to the first user include relative orientation and/or position data related to the view into the scene of the AR based environment.