Augmented reality environment based manipulation of multi-layered multi-view interactive digital media representations

Abstract

Various embodiments of the present disclosure relate generally to systems and methods for generating multi-view interactive digital media representations in a virtual reality environment. According to particular embodiments, a plurality of images is fused into a first content model and a first context model, both of which include multi-view interactive digital media representations of objects. Next, a virtual reality environment is generated using the first content model and the first context model. The virtual reality environment includes a first layer and a second layer. The user can navigate through and within the virtual reality environment to switch between multiple viewpoints of the content model via corresponding physical movements. The first layer includes the first content model and the second layer includes a second content model and wherein selection of the first layer provides access to the second layer with the second content model.

What is claimed is: 
     
         1 . A method for generating a multi-view interactive digital media representation in an augmented reality environment comprising:
 obtaining a first plurality of images of a first object and a second plurality of images of a second object, the first plurality of images and the second plurality of images captured from a plurality of different perspectives around the first object and the second object respectively, wherein the first plurality of images include first images that overlap and the second plurality of images include second images that overlap;   fusing the first plurality of images into a first multi-view interactive digital media representation (MVIDMR) of the first object by removing first background information from the first plurality of images and connecting the first plurality of images together into a first three-dimensional spatial graph, wherein the first MVIDMR is generated directly from the first plurality of images without using any 3D polygon model;   fusing the second plurality of images into a second (MVIDMR) of the second object by removing second background information from the second plurality of images and connecting the second plurality of images together into a second three-dimensional spatial graph, wherein the second MVIDMR is generated directly from the second plurality of images without using any 3D polygon model;   obtaining a real-time dynamic real-world image data to provide an augmented reality environment for the first MVIDMR and the second MVIDMR, wherein the first MVIDMR and the second MVIDMR are configured such that a user can manipulate the first MVIDMR and the second MVIDMR to view them from a plurality of different perspectives, wherein a user perspective changes as the user moves through the augmented reality environment;   identifying a first spatial location for a first tag on the first MVIDMR; and   associating the first tag with the first location, wherein the first tag is automatically propagated into a plurality of different perspective views of the first MVIDMR at the first spatial location.   
     
     
         2 . The method of  claim 1 , wherein manipulating the first MVIDMR comprises rotating the first MVIDMR. 
     
     
         3 . The method of  claim 1 , wherein manipulating the first MVIDMR comprises lifting the first MVIDMR. 
     
     
         4 . The method of  claim 1 , wherein the first plurality of images is obtained from a plurality of users. 
     
     
         5 . The method of  claim 1 , wherein the first plurality of images is obtained from a plurality of cameras. 
     
     
         6 . The method of  claim 1 , wherein the first MVIDMR in the augmented reality environment is enhanced using automatic frame selection to smooth transitions between frames. 
     
     
         7 . The method of  claim 6 , wherein the first MVIDMR in the augmented reality environment is enhanced using view interpolation. 
     
     
         8 . The method of  claim 1 , wherein the first plurality of images includes images with different temporal information. 
     
     
         9 . The method of  claim 1 , wherein the MVIDMR includes a locally convex surround view of the object. 
     
     
         10 . The method of  claim 1 , wherein the augmented reality environment is configured such that the user can appear to be closer to the first MVIDMR than the second MVIDMR and then subsequently closer to the second MVIDMR than the first MVIDMR. 
     
     
         11 . A system for generating a multi-view interactive digital media representation in an augmented reality environment comprising:
 an input interface configured to obtain a first plurality of images of a first object and a second plurality of images of a second object, the first plurality of images and the second plurality of images captured from a plurality of different perspectives around the first object and the second object respectively, wherein the first plurality of images include first images that overlap and the second plurality of images include second images that overlap;   a processor configured to fuse the first plurality of images into a first multi-view interactive digital media representation (MVIDMR) of the first object by removing first background information from the first plurality of images and connecting the first plurality of images together into a first three-dimensional spatial graph, wherein the first MVIDMR is generated directly from the first plurality of images without using any 3D polygon model, wherein the processor is further configured to fuse the second plurality of images into a second (MVIDMR) of the second object by removing second background information from the second plurality of images and connecting the second plurality of images together into a second three-dimensional spatial graph, wherein the second MVIDMR is generated directly from the second plurality of images without using any 3D polygon model;   an image sensor configured to obtain a real-time dynamic real-world image data to provide an augmented reality environment for the first MVIDMR and the second MVIDMR, wherein the first MVIDMR and the second MVIDMR are configured such that a user can manipulate the first MVIDMR and the second MVIDMR to view them from a plurality of different perspectives, wherein a user perspective changes as the user moves through the augmented reality environment;   wherein a first spatial location for a first tag on the first MVIDMR is identified and the first tag is associated with the first location, wherein the first tag is automatically propagated into a plurality of different perspective views of the first MVIDMR at the first spatial location.   
     
     
         12 . The system of  claim 11 , wherein manipulating the first MVIDMR comprises rotating the first MVIDMR. 
     
     
         13 . The system of  claim 11 , wherein manipulating the first MVIDMR comprises lifting the first MVIDMR. 
     
     
         14 . The system of  claim 11 , wherein the first plurality of images is obtained from a plurality of users. 
     
     
         15 . The system of  claim 11 , wherein the first plurality of images is obtained from a plurality of cameras. 
     
     
         16 . The system of  claim 11 , wherein the first MVIDMR in the augmented reality environment is enhanced using automatic frame selection to smooth transitions between frames. 
     
     
         17 . The system of  claim 16 , wherein the first MVIDMR in the augmented reality environment is enhanced using view interpolation. 
     
     
         18 . The system of  claim 11 , wherein the first plurality of images includes images with different temporal information. 
     
     
         19 . The system of  claim 11 , wherein the MVIDMR includes a locally convex surround view of the object. 
     
     
         20 . A non-transitory computer readable medium comprising computer code for generating a multi-view interactive digital media representation in an augmented reality environment comprising, the non-transitory computer readable medium comprising:
 computer code for obtaining a first plurality of images of a first object and a second plurality of images of a second object, the first plurality of images and the second plurality of images captured from a plurality of different perspectives around the first object and the second object respectively, wherein the first plurality of images include first images that overlap and the second plurality of images include second images that overlap;   computer code for fusing the first plurality of images into a first multi-view interactive digital media representation (MVIDMR) of the first object by removing first background information from the first plurality of images and connecting the first plurality of images together into a first three-dimensional spatial graph, wherein the first MVIDMR is generated directly from the first plurality of images without using any 3D polygon model;   computer code for fusing the second plurality of images into a second (MVIDMR) of the second object by removing second background information from the second plurality of images and connecting the second plurality of images together into a second three-dimensional spatial graph, wherein the second MVIDMR is generated directly from the second plurality of images without using any 3D polygon model;   computer code for obtaining a real-time dynamic real-world image data to provide an augmented reality environment for the first MVIDMR and the second MVIDMR, wherein the first MVIDMR and the second MVIDMR are configured such that a user can manipulate the first MVIDMR and the second MVIDMR to view them from a plurality of different perspectives, wherein a user perspective changes as the user moves through the augmented reality environment;   computer code for identifying a first spatial location for a first tag on the first MVIDMR; and   computer code for associating the first tag with the first location, wherein the first tag is automatically propagated into a plurality of different perspective views of the first MVIDMR at the first spatial location.