Systems and Methods of Interacting with a Virtual Grid in a Three-dimensional (3D) Sensory Space

Abstract

The technology disclosed relates to selecting a virtual item from a virtual grid in a three-dimensional (3D) sensory space. It also relates to navigating a virtual modality displaying a plurality of virtual items arranged in a grid by and automatically selecting a virtual item in a virtual grid at a terminal end of a control gesture of a control object responsive to a terminal gesture that transitions the control object from one physical arrangement to another. In one implementation, the control object is a hand. In some implementations, physical arrangements of the control object include at least a flat hand with thumb parallel to fingers, closed, half-open, pinched, curled, fisted, mime gun, okay sign, thumbs-up, ILY sign, one-finger point, two-finger point, thumb point, or pinkie point.

What is claimed is: 
     
         1 . A method including:
 creating a real-time digital representation of a real-world physical environment in which a user of a head mounted device is positioned, the environment further comprising a 3D sensory space with a defined volume of interest;   generating, for continuous display by the head mounted device, a live video stream including the real-time digital representation of the real-world physical environment;   providing the user of the head mounted device with the generated live video stream;   identifying, by the head mounted device, a virtual interactive item, from a library of virtual interactive items, that corresponds to and provides information about a real-world marker identified from the live video stream;   generating, for display, (i) 3D virtual imagery including virtual imagery corresponding to the identified virtual interactive item, and (ii) virtual imagery corresponding to a plurality of virtual items responsive to gestures in the 3D sensory space, wherein the generated 3D virtual imagery is superimposed, as a free-floating virtual modality in the real-world physical environment, allowing the user to simultaneously view both the 3D virtual imagery and the real-world physical environment;   providing the user of the head mounted device with the generated 3D virtual imagery;   detecting, using images provided in the generated live video stream, a gesture in the defined volume of interest of the 3D sensory space by the user of the head mounted device;   interpreting the detected gesture as selecting one virtual item from the library of virtual interactive items; and   executing an action associated with the selected virtual item.   
     
     
         2 . The method of  claim 1 , further including
 superimposing, for display by the head mounted device, the identified virtual interactive item onto the corresponding identified real-world marker.   
     
     
         3 . The method of  claim 1 , wherein
 the head mounted device includes one or more projectors that project imagery into the real-world physical environment, and wherein the method further includes:   projecting, by the one or more projectors of the head mounted device, the identified virtual interactive item onto the identified real-world marker in the real-world physical environment.   
     
     
         4 . The method of  claim 1 , wherein:
 the identifying of the virtual interactive item identifies two or more virtual interactive items, from the library of virtual interactive items, that correspond to and provide information about the identified real-world marker; and   the identified two or more virtual interactive items are included in the 3D virtual imagery generated for display.   
     
     
         5 . The method of  claim 1 , wherein
 the detected gesture is a scooping gesture in which a representation of a hand of a user appears to start from a position behind a virtual item and then proceed in a motion that appears to scoop up the virtual item from behind.   
     
     
         6 . The method of  claim 1 , wherein
 the identified virtual interactive item is superimposed in place of the identified real-world marker, so as to replace the identified real-world marker from view of the user.   
     
     
         7 . The method of  claim 1  wherein
 the real-world marker comprises at least one of a two- or three-dimensional barcode. 
 
     
     
         8 . The method of  claim 1  wherein
 the real-world marker comprises an image of a real-world item. 
 
     
     
         9 . A non-transitory computer-readable recording medium having computer instructions recorded thereon, the computer instructions, when executed by one or more processors, cause the one or more processors to perform the method of  claim 1 . 
     
     
         10 . A system comprising a memory storing computer instructions and one or more processors, the computer instructions, when executed by the one or more processors, cause the one or more processors to perform the method of  claim 1 .