Interactivity With A Mixed Reality

Abstract

Methods of interacting with a mixed reality are presented. A mobile device captures an image of a real-world object where the image has content information that can be used to control a mixed reality object through an offered command set. The mixed reality object can be real, virtual, or a mixture of both real and virtual.

What is claimed is: 
     
         1 . A computer-implemented method for generating and synchronizing interactive Augmented Reality (AR) displays, comprising:
 capturing live video of a real-world, physical environment and displaying the live video on a display of a device;   combining an information layer and the live video, the information layer related to one or more objects in the live video;   modeling computer-generated imagery based on the live video;   displaying the computer-generated imagery representing one or more objects in the live video on the display;   overlaying the information layer on the computer-generated imagery;   receiving sensor data from one or more onboard motion sensors indicating that the device is in motion; and   synchronizing the display of the live video, the computer-generated imagery and the information layer on the display using the sensor data.   
     
     
         2 . The method of  claim 1 , further comprising: displaying a navigation control for navigating the computer-generated imagery on the display. 
     
     
         3 . The method of  claim 1 , further comprising:
 performing object recognition on the live video;   obtaining object information for one or more recognized objects; and   generating the information layer for the one or more recognized objects.   
     
     
         4 . The method of  claim 3 , where generating the information layer, further comprises:
 including in the information layer one or more links to one or more network resources providing information related to the one or more recognized objects.   
     
     
         5 . The method of  claim 3 , further comprising: labeling the recognized one or more objects in the live video display. 
     
     
         6 . The method of  claim 5 , where at least one label includes content related to the recognized object. 
     
     
         7 . The method of  claim 1 , further comprising:
 receiving input through the stylus, the input identifying an object in the live video;   performing object recognition on the live video to identify the identified object;   obtaining object information for the recognized object; and   generating an information layer based on the object information.   
     
     
         8 . The method of  claim 1 , where the device is a mobile device. 
     
     
         9 . The method of  claim 1 , where an object in the real-world, physical environment captured by the live video is a document, the method further comprising:
 generating annotations related to the document; and   overlaying the annotations proximate the document on the live video.   
     
     
         10 . The method of  claim 1 , where an object in the real-world, physical environment captured by the live video is a machine, or part thereof, the method further comprising:
 generating annotations related to the object; and   overlaying the annotations on the live video or computer-generated imagery.   
     
     
         11 . The method of  claim 1 , where the one or more onboard motion sensors include an accelerometer. 
     
     
         12 . The method of  claim 1 , where modeling the computer-generated imagery based on the live video includes generating a three-dimensional model of the one or more objects in the live video. 
     
     
         13 . A computing device, comprising:
 a stylus configured for receiving touch input;   a display;   one or more onboard motion sensors configured for sensing motion of the apparatus;   a video camera for capturing live video of a real-world, physical environment for display on the display; and   a processor communicatively coupled to the stylus, the motion sensor and the video camera, the processor programmed to:
 combine an information layer and live video in the display of the device; 
 model computer-generated imagery based on the live video; 
 display computer-generated imagery representing one or more objects in the live video on a display; 
 overlay the information layer on the computer-generated imagery; 
 receive sensor data from the one or more onboard motion sensors indicating that the apparatus is in motion; and 
 synchronize the live video, computer-generated imagery and the information layer using the sensor data. 
   
     
     
         14 . The computing device of  claim 13 , where the processor is further configured for:
 displaying a navigation control for navigating the computer-generated imagery.   
     
     
         15 . The computing device of  claim 13 , where modeling the computer-generated imagery based on the live video includes generating a three-dimensional model of the one or more objects in the live video. 
     
     
         16 . The computing device of  claim 13 , where the one or more onboard motion sensors include an accelerometer.