3D Interactions with Web Content

Abstract

Embodiments of a 3D web interaction system are disclosed that allow a user to select a content item from a browser, displayed in an artificial reality environment, and present a corresponding version of the content item in the artificial reality environment. The 3D web interaction system can create the version of the selected content item in different ways depending on whether the selected content item is associated with 3D content and, if so, the type of the associated 3D content. For example, the 3D web interaction system can create and present different versions of the selected content item depending on whether the selected content item is(a) not associated with 3D content, (b) associated with “environment content,” or (c) associated with one or more 3D models.

1 - 20 . (canceled) 
     
     
         21 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform steps comprising:
 receiving, from a process initiated by the loading or viewing of a website in a virtual browser displayed in an artificial reality (XR) environment, an identification of one or more content items included in a current website for the browser;   resolving at least one content item, of the one or more content items, as being associated with a first 3D content item or as being a 2D content item capable of being converted into a second 3D content item;   performing a content item prefetch, from a remote source, by: 
 prefetching the first 3D content item from the remote source; or 
 prefetching the 2D content item from the remote source and converting 
 the 2D content item into the second 3D content item; locally storing the first 3D content or the second 3D content item as a 3D content 
 
 item for display outside the virtual browser; identifying a user’s selection corresponding to the at least one content item and, 
 in response, retrieving the 3D content item from the local storage; and providing the retrieved 3D content item outside the virtual browser as a virtual 
 object that the user can manipulate. 
   
     
     
         22 . The computer-readable storage medium of  claim 21 , wherein the process, initiated by the loading or viewing of the website and that provides the identification of the one or more content items included in the current website, is initiated by a plug-in installed for the virtual browser or by a script loaded into the virtual browser. 
     
     
         23 . The computer-readable storage medium of  claim 21 , wherein the receiving is performed using inter-process communication (IPC) messages. 
     
     
         24 . The computer-readable storage medium of  claim 21 , wherein the resolving, the at least one content item as being associated with the first 3D content item or as being the 2D content item capable of being converted into the second 3D content item, is based on based on specialized tags, in the website, for the at least one content item. 
     
     
         25 . The computer-readable storage medium of  claim 21 , wherein the resolving, the at least one content item as being associated with the first 3D content item or as being the 2D content item capable of being converted into the second 3D content item, is based on content post-fix identifiers. 
     
     
         26 . The computer-readable storage medium of  claim 21 , wherein the resolving, the at least one content item as being associated with the first 3D content item or as being the 2D content item capable of being converted into the second 3D content item, includes the resolving the at least one content item as being associated with the first 3D content item. 
     
     
         27 . The computer-readable storage medium of  claim 26 , wherein the resolving, the at least one content item as being associated with the first 3D content item, is based on identifiers imbedded in the at least one content item content item. 
     
     
         28 . The computer-readable storage medium of  claim 21 , wherein the resolving, the at least one content item as being associated with the first 3D content item or as being the 2D content item capable of being converted into the second 3D content item, includes the resolving the at least one content item as being the 2D content item capable of being converted into the second 3D content item. 
     
     
         29 . The computer-readable storage medium of  claim 28 , wherein the 2D content item is an image, and wherein the converting the 2D content item into the second 3D content item includes providing the image to a machine learning model trained to produce a panoramic or 3D image, which is the second 3D content item, based on a 2D image. 
     
     
         30 . The computer-readable storage medium of  claim 21 ,
 wherein the retrieved 3D content is a partial view into an 3D environment;   wherein the user manipulations of the partial view cause the partial view, into the 3D environment, to exceed a threshold, causing the 3D environment to be displayed; and   wherein the causing the 3D environment to be displayed replaces a current artificial reality environment with the 3D environment.   
     
     
         31 . The computer-readable storage medium of  claim 21 , wherein the user selection is recognized by determining the user made a motion that A) selects the at least one content item in the virtual browser and B) pulls away from the virtual browser. 
     
     
         32 . The computer-readable storage medium of  claim 21 , wherein, following identifying the user selection, the at least one content item is not displayed in the browser; and following a determination that the user released the 3D content, the at least one content item reappears in the browser. 
     
     
         33 . The computer-readable storage medium of  claim 21 , wherein the at least one content item is displayed in the virtual browser with an indication that the at least one content item is associated with the 3D content. 
     
     
         34 . A method comprising:
 receiving, from a process initiated by the loading or viewing of a website in a virtual browser displayed in an artificial reality (XR) environment, an identification of one or more content items included in a current website for the browser;   resolving at least one content item, of the one or more content items, as being associated with a first 3D content item or as being a 2D content item capable of being converted into a second 3D content item;   performing a content item prefetch, from a remote source, by:
 prefetching the first 3D content item from the remote source; or 
 prefetching the 2D content item from the remote source and converting 
 the 2D content item into the second 3D content item; locally storing the first 3D content or the second 3D content item as a 3D content 
 
 item for display outside the virtual browser; identifying a user’s selection corresponding to the at least one content item and, 
 in response, retrieving the 3D content item from the local storage; and providing the retrieved 3D content item outside the virtual browser as a virtual 
 object that the user can manipulate. 
   
     
     
         35 . The method of  claim 34 , wherein the resolving, the at least one content item as being associated with the first 3D content item or as being the 2D content item capable of being converted into the second 3D content item, is based on based on specialized tags, in the website, for the at least one content item. 
     
     
         36 . The method of  claim 34 , wherein the resolving, the at least one content item as being associated with the first 3D content item or as being the 2D content item capable of being converted into the second 3D content item, includes the resolving the at least one content item as being associated with the first 3D content item. 
     
     
         37 . The method of  claim 34 ,
 wherein the resolving, the at least one content item as being associated with the first 3D content item or as being the 2D content item capable of being converted into the second 3D content item, includes the resolving the at least one content item as being the 2D content item capable of being converted into the second 3D content item;   wherein the 2D content item is an image; and   wherein the converting the 2D content item into the second 3D content item includes providing the image to a machine learning model trained to produce a panoramic or 3D image, which is the second 3D content item, based on a 2D image.   
     
     
         38 . A computing system comprising:
 one or more processors; and   one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform steps comprising: 
 receiving, from a process initiated by the loading or viewing of a website in a virtual browser displayed in an artificial reality (XR) environment, an identification of one or more content items included in a current website for the browser; 
 resolving at least one content item, of the one or more content items, as being associated with a first 3D content item or as being a 2D content item capable of being converted into a second 3D content item; 
 performing a content item prefetch, from a remote source, by:
 prefetching the first 3D content item from the remote source; or 
 prefetching the 2D content item from the remote source and converting the 2D content item into the second 3D content item; 
 
 locally storing the first 3D content or the second 3D content item as a 3D content item for display outside the virtual browser; 
 identifying a user’s selection corresponding to the at least one content item and, in response, retrieving the 3D content item from the local storage; and 
 providing the retrieved 3D content item outside the virtual browser as a virtual object that the user can manipulate. 
   
     
     
         39 . The computing system of  claim 38 , 
 wherein the retrieved 3D content is a partial view into an 3D environment;   wherein the user manipulations of the partial view cause the partial view, into the 3D environment, to exceed a threshold, causing the 3D environment to be displayed; and   wherein the causing the 3D environment to be displayed replaces a current artificial reality environment with the 3D environment.   
     
     
         40 . The computing system of  claim 38 , wherein the at least one content item is displayed in the virtual browser with an indication that the at least one content item is associated with the 3D content.