System and method for augmented reality maintenance

Abstract

A method enables a user to use an augmented reality system and a mobile device to display augmented reality by overlaying a virtual 3D object over a physical 3D object. In one implementation, the method includes: enabling a user to capture a live camera image of a physical 3D object via camera on the mobile device; determining if there are image tags on the physical 3D object in the live camera image; using the one or more image tags to access information used by the augmented reality system; and overlaying the virtual 3D object onto a live camera image of the physical 3D object on a display of the mobile device to produce augmented reality effects, wherein the virtual 3D object delivers content a user that assists in performing maintenance on the physical 3D object.

What is claimed is: 
     
         1 . A method of enabling an augmented reality interaction system for a mobile device user to overlay a virtual 3D component over a physical 3D component with which the virtual 3D component has an association, the method comprising:
 enabling a user to capture a live camera image of a physical 3D object via a camera on the mobile device;   determining if there are image tags on the physical 3D object in the live camera image;   in response to identifying images tags on the physical 3D object, producing a virtual rendering of the physical 3D object;   determining the virtual 3D component to be displayed over an image of a physical 3D object, wherein the virtual 3D component includes a plurality of elements each having a depth value;   comparing a virtual depth rendering of the virtual 3D component to a virtual depth rendering of the virtual 3D object;   in response to the comparison, subtracting the elements of the virtual 3D component that have a lower depth value than elements of the virtual 3D object at corresponding spatial positions; and   overlaying the virtual 3D component without the subtracted elements onto a live camera image of the physical 3D object on a display of the mobile device to produce augmented reality effects, wherein the virtual 3D component delivers content to a user that assists in performing maintenance on the physical 3D object.   
     
     
         2 . The method of  claim 1 , wherein maintenance assistance provided includes displaying text, displaying a pre-recorded video, displaying an augmented virtual device overlaid over the display of the physical 3D object or combinations thereof. 
     
     
         3 . The method of  claim 1 , wherein the physical 3D object has an inside and an outside and includes image tags outside of the physical 3D object that identify the physical 3D object to the augmented reality maintenance system, and wherein the physical 3D object also includes image tags inside of the physical 3D object that assist with displaying augmented reality effects within the physical 3D object. 
     
     
         4 . The method of  claim 1 , wherein the physical 3D object has an inside and an outside and includes image tags inside of the physical 3D object that assist with displaying augmented reality effects within the physical 3D object. 
     
     
         5 . The method of  claim 1 , wherein augmented reality content is overlaid over a captured camera image if an image tag identified in the captured image of the physical 3D object. 
     
     
         6 . The method of  claim 1 , further comprising: using a depth buffer to determine which elements of a 3D rendering are visible to the camera. 
     
     
         7 . The method of  claim 6 , further comprising: determining the z-value of each element is compared against an existing depth buffer z-value at the same screen co-ordinate, wherein if the existing z-value is greater, the pixel is drawn, and wherein if the existing z-value is less than an existing depth buffer z-value at the same screen co-ordinate, then the pixel is discarded. 
     
     
         8 . The method of  claim 6 , further comprising: rendering the virtual 3D object by mapping the image tag co-ordinates to the physical 3D object co-ordinates using vector subtraction. 
     
     
         9 . The method of  claim 8 , further comprising: combining the camera display image with the rendering of an additional virtual 3D component minus the virtual 3D object. 
     
     
         10 . The method of  claim 1 , wherein a CAD model having spatial parameters is associated with an image tag. 
     
     
         11 . The method of  claim 1 , wherein virtual depth rendering of the physical 3D object extracts 2D information from the display of the physical 3D object to produce a 3D image of the physical 3D object. 
     
     
         12 . The method of  claim 1 , further comprising: overlaying virtual graphics in a position within 3D space on the display of the mobile device that augment the display of the physical 3D object with 3D to create complex augmented reality. 
     
     
         13 . The method of  claim 1 , wherein an object recognition system of the augmented reality interaction system establishes a communication channel associated with the augmented reality component and the physical 3D object so that additional elements of content are augmented on the mobile device display, and elements of the content are influenced by the augmented state. 
     
     
         14 . The method of  claim 1 , wherein the augmented reality interaction system includes a translation system for extracting a first language from the display of the physical 3D object and translates the first language into a second language that is visible to the user in place of the first language. 
     
     
         15 . A method of enabling a user to use an augmented reality system and a mobile device to display augmented reality by overlaying a virtual 3D object over a physical 3D object, the method comprising:
 enabling a user to capture a live camera image of a physical 3D object via a camera on the mobile device;   determining if there are identifiers on the physical 3D object in the live camera image;   using the one or more identifiers to access information used by the augmented reality system; and   overlaying virtual 3D objects onto a live camera image of the physical 3D object on a display of the mobile device to produce augmented reality effects, wherein the virtual 3D objects deliver targeted content a user that assist in performing maintenance on the physical 3D object.   
     
     
         16 . The method of  claim 15 , wherein the physical 3D object includes image tags that assist with displaying augmented reality effects within the physical 3D object. 
     
     
         17 . The method of  claim 16 , wherein augmented reality content is overlaid over a captured camera image if an image tag is identified in the captured image within the physical 3D object. 
     
     
         18 . An augmented reality system to provide information for a physical 3D object for a user of a mobile device including a camera and a video display, the mobile device in communication with the system through a communication network, the system comprising:
 a host server:   the host server configured to receive over the network a live camera image of the physical 3D object captured by the camera on the mobile device and to compare the image of the physical 3D object to a database of identifiers to identify the physical 3D object and access associated augmented reality information;
 wherein if the physical 3D object is identified, the server accesses and transmits the associated augmented reality information to the mobile device to display the associated augmented reality information onto a live camera image of the physical 3D object to produce augmented reality effects that assist in performing maintenance on the physical 3D object. 
   
     
     
         19 . The system of  claim 18 , wherein the physical 3D object includes an image tag identifier, the host server configured to receive over the network a live camera image of the physical 3D object and image tag captured by the camera on the mobile device and to compare the image tag to a database of image tag identifiers to identify the physical 3D object and access associated augmented reality information.