Real-time shared augmented reality experience

Abstract

A method and system are provided for enabling a shared augmented reality experience. The system may include one or more onsite devices for generating AR representations of a real-world location, and one or more offsite devices for generating virtual AR representations of the real-world location. The AR representations may include data and/or content incorporated into live views of a real-world location. The virtual AR representations of the AR scene may incorporate images and data from a real-world location and include additional AR content. The onsite devices may synchronize content used to create the AR experience with the offsite devices in real time, such that the onsite AR representations and the offsite virtual AR representations are consistent with each other.

1 . A computer-implemented method for providing a shared augmented reality (AR) experience, the method comprising:
 communicating, from a server to an onsite device in proximity to a real-world location, a data bundle comprising a first data set consisting of an available AR content item, a second data set consisting of initial trackable data, and a third data set consisting of initial position and geometry data relating to one or more physical features of the real-world location, based on a location coordinate of the onsite device, wherein the initial trackable data comprises information relating to one or more trackable features that are disposed at the real-world location and findable by the onsite device;   capturing imaging information of the real-world location using an imaging sensor of the onsite device;   visualizing, using the onsite device, an onsite AR representation of the real-world location by presenting the available AR content item incorporated into a live view of the real-world location;   updating at least one data set of the data bundle to generate an updated data bundle, using the onsite device;   communicating the updated data bundle from the onsite device to a remote offsite device via the server; and   visualizing, at the offsite device and using the updated data bundle, a virtual reality (VR) representation of the real-world location by creating virtual copies of the one or more physical features of the real-world location and incorporating the AR content item into the VR representation to generate an offsite virtual AR representation.   
     
     
         2 . The method of  claim 1 , wherein updating the at least one data set comprises updating the trackable data using the captured imaging information from the onsite device. 
     
     
         3 . The method of  claim 1 , wherein updating the at least one data set comprises updating the position and geometry data using the captured imaging information from the onsite device. 
     
     
         4 . The method of  claim 1 , wherein updating the at least one data set comprises changing at least one aspect of the AR content item using the onsite device. 
     
     
         5 . The method of  claim 4 , wherein changing at least one aspect of the AR content item comprises:
 receiving, from an input component of the onsite device, an onsite user instruction to edit the AR content item in the onsite AR representation; and   updating, at the onsite device, the AR representation based on the onsite user instruction.   
     
     
         6 . The method of  claim 5 , wherein the onsite user instruction to edit the AR content item comprises an instruction to create, alter, move, or remove the AR content item. 
     
     
         7 . The method of  claim 1 , wherein the trackable data comprises a point cloud having associated feature descriptors. 
     
     
         8 . The method of  claim 7 , further comprising: determining the initial trackable data using third-party sources. 
     
     
         9 . The method of  claim 8 , wherein determining the initial trackable data using third-party sources comprises cross-referencing two or more data sources relating to the real-world location. 
     
     
         10 . The method of  claim 8 , wherein the third-party sources include additional position and geometry data created from images of the real-world location using photogrammetry image processing technology. 
     
     
         11 . The method of  claim 1 , wherein the one or more trackable features comprise a fiducial marker. 
     
     
         12 . The method of  claim 1 , wherein the server comprises a server cluster. 
     
     
         13 . The method of  claim 1 , wherein the imaging sensor comprises a camera. 
     
     
         14 . The method of  claim 1 , wherein the imaging sensor comprises a structured light sensor. 
     
     
         15 . The method of  claim 1 , further comprising:
 editing, using the offsite device, at least one aspect of the AR content item to create an edited AR content item;   communicating data comprising the edited AR content item from the offsite device to the onsite device via the server; and   updating the onsite AR representation at the onsite device using the data comprising the edited AR content item.   
     
     
         16 . The method of  claim 1 , wherein updates to the data bundle are communicated between the onsite device and the offsite device in real-time. 
     
     
         17 . The method of  claim 1 , wherein updates to the data bundle are communicated between the onsite device and the offsite device asynchronously. 
     
     
         18 . The method of  claim 1 , further comprising:
 receiving, at the onsite device from the offsite device, an offsite user instruction to create a second AR content item; and   updating, at the onsite device, the onsite AR representation based on the offsite user instruction, such that the second AR content item is presented at a synchronized position and having a synchronized state in both the onsite AR representation and the offsite virtual AR representation.   
     
     
         19 . The method of  claim 1 , further comprising synchronizing a change to the onsite AR representation of the onsite device with a plurality of offsite virtual AR representations on respective offsite devices and with a plurality of other onsite AR representations on respective other onsite devices. 
     
     
         20 . The method of  claim 1 , wherein the AR content item comprises a 3D object. 
     
     
         21 . The method of  claim 1 , wherein the AR content item contains a hierarchy of AR objects. 
     
     
         22 . The method of  claim 1 , further comprising establishing, by the onsite device, an event in which one or more other devices are given permission to change the AR content item. 
     
     
         23 . The method of  claim 22 , wherein the one or more other devices comprise at least one other onsite device. 
     
     
         24 . The method of  claim 22 , wherein establishing the event includes sending one or more invitations to the other devices. 
     
     
         25 . The method of  claim 1 , wherein the offsite device maintains and follows a point of view (POV) of the onsite device when visualizing the offsite virtual AR representation. 
     
     
         26 . The method of  claim 1 , wherein the offsite device utilizes an offsite POV that is independent of the POV of the onsite device. 
     
     
         27 . The method of  claim 26 , wherein the POV of the offsite virtual AR representation comprises a first-person view from a virtual avatar of a user of the offsite device in the offsite virtual AR representation, wherein the virtual avatar is incorporated into the onsite AR representation. 
     
     
         28 . The method of  claim 26 , wherein the POV of the offsite virtual AR representation comprises a third-person view of a virtual avatar of a user of the offsite device in the offsite virtual AR representation, wherein the virtual avatar is incorporated into the onsite AR representation. 
     
     
         29 . The method of  claim 1 , wherein communication between the onsite device and the offsite device is conducted over a network. 
     
     
         30 . The method of  claim 29 , wherein the network comprises a server. 
     
     
         31 . The method of  claim 29 , wherein the network is ad hoc. 
     
     
         32 . The method of  claim 1 , wherein the onsite device is a first onsite device of a plurality of onsite devices, the method further comprising:
 communicating, from the first onsite device to a second one of the plurality of onsite devices, the updated data bundle;   visualizing, using the second onsite device, the AR content item in the real-world location from a point of view proximate the first onsite device to create a second onsite AR representation; and   synchronizing changes to the AR content item between the first onsite device and the second onsite device.   
     
     
         33 . The method of  claim 1 , further comprising storing information corresponding to the updated data bundle on a storage device, such that changes persist between sessions. 
     
     
         34 . The method of  claim 33 , wherein the stored information persists for a selected amount of time before being erased from the storage device. 
     
     
         35 . A computer-implemented method for providing a shared augmented reality (AR) experience, the method comprising:
 receiving at an onsite sensing device in proximity to a real-world location, from a server, initial trackable data relating to a trackable present at the real-world location, and initial position and geometry data relating to one or more physical features of the real-world location, based on a location coordinate of the onsite sensing device;   capturing on-location data of the real-world location using a sensor of the onsite sensing device;   generating updated trackable data and updated position and geometry data using the on-location data captured by the onsite sensing device;   sending, from the onsite sensing device to the server, the updated trackable data and the updated position and geometry data;   sending, from the server to an offsite computing device remote to the real-world location, the updated trackable data and the updated position and geometry data, based on a request for information relating to the real-world location;   visualizing, at the offsite computing device remote to the real-world location, a virtual reality (VR) representation of the real-world location, including creating virtual copies of the one or more physical features of the real-world location based on the updated position and geometry data;   creating AR content at the offsite computing device and incorporating the AR content into the VR representation to produce an offsite virtual AR representation;   calculating a spatial relationship including a relative position, an orientation, and a scale of the AR content relative to the updated trackable data;   communicating the AR content created at the offsite computing device and the calculated spatial relationship from the offsite computing device to an onsite AR device; and   visualizing, at the onsite AR device, an onsite AR representation of the real-world location, including presenting the AR content overlaid on a live view of the real-world location, wherein the AR content is positioned relative to the updated trackable data using the calculated spatial relationship.   
     
     
         36 . The method of  claim 35 , wherein the onsite sensing device and the onsite AR device comprise a same single device. 
     
     
         37 . The method of  claim 35 , wherein the onsite sensing device and the onsite AR device are present at the real-world location simultaneously. 
     
     
         38 . The method of  claim 35 , further comprising synchronizing a status of the AR content between the onsite AR representation and the offsite virtual AR representation. 
     
     
         39 . The method of  claim 38 , wherein the offsite computing device is a first offsite computing device of a plurality of offsite computing devices, the method further comprising:
 synchronizing a change in the AR content between the first offsite computing device and a second one of the plurality of offsite computing devices.   
     
     
         40 . The method of  claim 38 , wherein the onsite AR device is a first onsite AR device of a plurality of onsite AR devices disposed at the real-world location, the method further comprising:
 communicating the AR content and the calculated spatial relationship from the offsite computing device to a second onsite AR device of the plurality of onsite AR devices; and   updating a second onsite AR representation of the second onsite AR device to include the AR content.   
     
     
         41 . The method of  claim 40 , wherein the AR content persists in the second onsite AR representation when the first onsite AR device ceases communication. 
     
     
         42 . The method of  claim 40 , wherein the first and second onsite AR devices are disposed in the real-world location at different, mutually exclusive times. 
     
     
         43 . The method of  claim 35  further comprising:
 determining the initial tracking data by cross-referencing street view image data and depth information from one or more third-party sources to generate a point cloud with associated feature descriptors; and 
 generating the VR representation of the one or more physical features of the real-world location by overlaying the street view image data onto geometry generated from the depth information. 
 
     
     
         44 . A system for providing a shared augmented reality (AR) experience, the system comprising:
 one or more onsite computing devices each having an imaging sensor and configured to obtain environmental data and to generate an AR representation of a real-world location by presenting AR content incorporated into a respective live view of the real-world location;   one or more offsite computing devices remote to the real-world location, each of the one or more offsite devices configured to generate a respective virtual AR representation of the real-world location by presenting the AR content incorporated into a respective virtual reality (VR) view of the real-world location; and   one or more processors in communication with stored instructions configured to synchronize each of the one or more onsite computing devices with each of the one or more offsite computing devices, such that the AR representations and the virtual AR representations are substantially consistent with each other, the stored instructions being executable by the one or more processors to:
 obtain current environmental data comprising trackable data, position data, and geometry data, by capturing imagery of the real-world location using an imaging sensor of at least one of the onsite computing devices, 
 produce changed AR content in response to user input, 
 make the changed AR content and the current environmental data available to the one or more onsite computing devices and to the one or more offsite computing devices, and 
 use the one or more offsite computing devices to produce the virtual AR representation of the real-world location by presenting the changed AR content incorporated into the respective VR view of the real-world location, characteristics of the VR view being based on the current environmental data. 
   
     
     
         45 . A computer-implemented method for providing a shared augmented reality (AR) experience, the method comprising:
 receiving, at an onsite device in proximity to a real-world location, a location coordinate of the onsite device;   sending, from the onsite device to a server, a request for AR content and first information based on the location coordinate of the onsite device, the first information including data defining a spatial relationship between the AR content and a trackable in the real-world location, and initial position and geometry data of one or more physical features in the real-world location;   receiving the first information at the onsite device;   capturing imaging data of the real-world location using an imaging sensor of the onsite device;   visualizing, at the onsite device, an AR representation of the real-world location by presenting the AR content incorporated into a live view of the real-world location using the first information;   updating the first information, using the captured imaging data, to create second information including an updated spatial relationship between the AR content and the trackable, and updated position and geometry data of the one or more physical features in the real-world location;   sending the AR content and the second information to an offsite device remote to the real-world location;   visualizing, using the offsite device, a virtual reality (VR) representation of the real-world location by creating virtual copies of the physical features of the real-world location using the second information, wherein the offsite device incorporates the AR content into the VR representation, creating a virtual AR representation of the real-world location; and   synchronizing a change made to the AR representation by the onsite device with the virtual AR representation of the offsite device.