Team augmented reality system

Abstract

A system for combining live action and virtual images in real time into a final composite image as viewed by a user through a head mounted display, and which uses a self-contained tracking sensor to enable large groups of users to use the system simultaneously and in complex walled environments, and a color keying based algorithm to determine display of real or virtual imagery to the user.

1 . A system comprising:
 a helmet mounted display (HMD) for a user;   a front-facing camera or cameras; and   a low latency keying module configured to mix virtual and live action environments and objects in an augmented reality game or simulation.   
     
     
         2 . The system of  claim 1  wherein the keying module is configured to composite the live action image from the front facing camera with a rendered virtual image from the point of view of the HMD, and send the composited image to the HMD so the user can see the combined image. 
     
     
         3 . The system of  claim 1  wherein the keying module is configured to take in a live action image from the camera, and perform a color difference and despill operation on the image to determine how to mix it with an image of a virtual environment. 
     
     
         4 . The system of  claim 1  wherein the camera is mounted to the front of the HMD and facing forward, to provide a view of the real environment in the direction that the user is looking. 
     
     
         5 . The system of  claim 1  further comprising an upward-facing tracking sensor configured to be carried by the user of the HMD and to detect overhead tracking markers. 
     
     
         6 . The system of  claim 5  wherein the sensor is configured to determine a position of the user in a physical space, and the keying module is configured to determine which areas of the physical space will be visually replaced by virtual elements. 
     
     
         7 . The system of  claim 1  wherein the virtual elements will be visually replaced when areas of the live action environment are painted a solid blue or green color. 
     
     
         8 . The system of  claim 1  wherein the sensor is configured to calculate the position of the HMD in a physical environment, and that information is used to render a virtual image from the correct point of view that is mixed with the live action view and displayed in the HMD. 
     
     
         9 . The system of  claim 1  wherein each user of the HMD has a separate tracking sensor and rendering computer, whose function is independent of the sensors and rendering computers of the other users. 
     
     
         10 . The system of  claim 9  wherein a tracking system of the sensor is not dependent on the other users because it can calculate the complete position and orientation of the HMD based upon the view of the overhead markers without communicating with any external sensors. 
     
     
         11 . The system of  claim 1  wherein the front facing camera or cameras are configured to provide a real time view of the environment that the user is facing. 
     
     
         12 . The system of  claim 1  wherein the low latency is on the order of 25 milliseconds. 
     
     
         13 . The system of  claim 1  wherein the sensor is a self-contained 6DOF tracking sensor. 
     
     
         14 . The system of  claim 1  wherein the keying module is configured to allow an environment designer to determine which components of an environment of the user are to be optically passed through and which are to be replaced by virtual elements. 
     
     
         15 . The system of  claim 1  wherein the keying module is configured to handle transitions between virtual and real worlds in a game or simulation by reading the image from the front facing camera, performing a color difference key process on the image to remove the solid blue or green elements from the image, and then combining this image with a virtual rendered image. 
     
     
         16 . The system of  claim 1  wherein the keying module is embodied in low latency programmable hardware. 
     
     
         17 . The system of  claim 1  wherein the keying module is configured to calculate the color difference between the red, green and blue elements of a region of a live action image, to use that difference to determine the portions of the live action image to remove, and use a despill calculation to limit the amount of blue or green in the image and remove colored fringes from the image. 
     
     
         18 . The system of  claim 1  wherein the number of users of the self-contained tracking system can be greater than five because the tracking system can calculate its position based on a view of overhead markers without needing to communicate with an external tracking computer. 
     
     
         19 . The system of  claim 1  wherein the users of the self-contained tracking system can be located very close to each other without experiencing tracking problems, as the tracking system can calculate its position even when occluded to either side. 
     
     
         20 . The system of  claim 1  wherein the users of the self-contained tracking system can walk very close to head height walls without experiencing tracking problems, as the tracking system can calculate its position even when occluded to either side. 
     
     
         21 - 52 . (canceled)