Grasping virtual objects in augmented reality

Abstract

An augmented reality system which enables grasping of virtual objects is described such as to stack virtual cubes or to manipulate virtual objects in other ways. In various embodiments a user's hand or another real object is tracked in an augmented reality environment. In examples, the shape of the tracked real object is approximated using at least two different types of particles and the virtual objects are updated according to simulated forces exerted between the augmented reality environment and at least some of the particles. In various embodiments 3D positions of a first one of the types of particles, kinematic particles, are updated according to the tracked real object; and passive particles move with linked kinematic particles without penetrating virtual objects. In some examples a real-time optic flow process is used to track motion of the real object.

1 - 20 . (canceled) 
     
     
         21 . A computing device arranged to calculate a virtual object in a virtual reality environment, the computing device comprising:
 an input arranged to receive a plurality of images of at least part of a real object in an interaction volume associated with the virtual reality environment;   a 3D tracker arranged to:
 track 3D motion and/or deformation of a surface of the real object in the interaction volume using the received plurality of images; and 
 approximate the surface of the real object by positioning, within the contour of the real object, a plurality of non-intersecting kinematic particles; and 
   a physics simulator calculating position of the virtual object in the virtual reality environment according to simulated forces between the virtual object and the real object, the simulated forces being determined at least using the tracked 3D motion and/or deformation and the plurality of kinematic particles.   
     
     
         22 . A device as claimed in  claim 21 , wherein the physics simulator calculates the position of the virtual object in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment. 
     
     
         23 . A device as claimed in  claim 21 , wherein the physics simulator determines the simulated forces using passive particles that do not enter the virtual object, the passive particles being linked to kinematic particles which approximate the surface of at least part of the real object. 
     
     
         24 . A device as claimed in  claim 21 , wherein the 3D tracker is arranged to delete at least one of the plurality of kinematic particles. 
     
     
         25 . A device as claimed in  claim 21 , wherein the plurality of images comprise depth maps and RGB images. 
     
     
         26 . A device as claimed in  claim 21 , wherein the 3D tracker:
 calculates a 2D optical flow field from intensity images; and   calculates a 3D optical flow field from the 2D optical flow field by looking up depth values of elements of the 2D optical flow field in depth maps associated with the intensity images.   
     
     
         27 . A device as claimed in  21 , wherein the 3D tracker comprises a graphics processing unit for calculating an optical flow field of the surface of the real object. 
     
     
         28 . A device as claimed in  claim 21 , wherein the 3D tracker comprises a graphics processing unit arranged to calculate a surface of the real object. 
     
     
         29 . A device as claimed in  claim 21 , wherein the 3D tracker comprises a graphics processing unit for calculating a 3D shape of the real object. 
     
     
         30 . A device as claimed in  claim 21 , wherein the 3D tracker is at least partially implemented using hardware logic selected from any one or more of the following: a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, and a graphics processing unit. 
     
     
         31 . A computer-implemented method of calculating a virtual object in a virtual reality environment, the method comprising:
 receiving, at a processor, a plurality of images of at least part of a real object in an interaction volume associated with the virtual reality environment;   tracking 3D motion and/or deformation of a surface of the real object in the interaction volume using the received plurality of images;   approximating the surface of the real object by positioning, within the contour of the real object, a plurality of non-intersecting kinematic particles; and   calculating a position of the virtual object in the virtual reality environment according to simulated forces between the virtual object and the real object, the simulated forces being determined at least using the tracked 3D motion and/or deformation and the plurality of kinematic particles.   
     
     
         32 . A method as claimed in  claim 31 , further comprising calculating the position of the virtual object in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment. 
     
     
         33 . A method as claimed in  claim 31 , further comprising calculating the simulated forces using passive particles that do not enter the virtual object, the passive particles being linked to kinematic particles which approximate the surface of at least part of the real object. 
     
     
         34 . A method as claimed in  claim 31 , further comprising using the tracked 3D motion and/or deformation to delete at least one of the plurality of kinematic particles. 
     
     
         35 . A method as claimed in  claim 31 , further comprising:
 calculating a 2D optical flow field from intensity images; and   calculating a 3D optical flow field from the 2D optical flow field by looking up depth values of elements of the 2D optical flow field in depth maps associated with the intensity images.   
     
     
         36 . One or more computer-readable storage media comprising computer-executable instructions that, when executed by one or more processors, performed operations comprising:
 receiving, at a processor, a plurality of images of at least part of a real object in an interaction volume associated with the virtual reality environment;   tracking 3D motion and/or deformation of a surface of the real object in the interaction volume using the received plurality of images;   approximating the surface of the real object by positioning, within the contour of the real object, a plurality of non-intersecting kinematic particles; and   calculating a position of the virtual object in the virtual reality environment according to simulated forces between the virtual object and the real object, the simulated forces being determined at least using the tracked 3D motion and/or deformation and the plurality of kinematic particles.   
     
     
         37 . The one or more computer-readable storage media of  claim 36 , wherein the computer executable instructions further cause the one or more processors to perform an operation comprising calculating the position of the virtual object in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment. 
     
     
         38 . The one or more computer-readable storage media of  claim 36 , wherein the computer executable instructions further cause the one or more processors to perform an operation comprising displaying the virtual object in the virtual reality environment according to the calculated position of the virtual object. 
     
     
         39 . The one or more computer-readable storage media of  claim 36 , wherein the computer executable instructions further cause the one or more processors to perform an operation comprising displaying the virtual object and the virtual reality environment in a robust, stable manner when the at least part of the real object penetrates the virtual object in the virtual reality environment. 
     
     
         40 . The one or more computer-readable storage media of  claim 36 , wherein the real object is a hand of a user.