In-line mediation for manipulating three-dimensional content on a display device

Abstract

A user holds the mobile device upright or sits in front of a nomadic or stationary device, views the monitor from a suitable distance, and physically reaches behind the device with her hand to manipulate a 3D object displayed on the monitor. The device functions as a 3D in-line mediator that provides visual coherency to the user when she reaches behind the device to use hand gestures and movements to manipulate a perceived object behind the device and sees that the 3D object on the display is being manipulated. The perceived object that the user manipulates behind the device with bare hands corresponds to the 3D object displayed on the device. The visual coherency arises from the alignment of the user's head or eyes, the device, and the 3D object. The user's hand may be represented as an image of the actual hand or as a virtualized representation of the hand, such as part of an avatar.

1 . A method of detecting manipulation of a digital 3D object displayed on a device having a front side with a display and a back side, the method comprising:
 detecting a hand within a specific area of the back side of the device, the back side having an sensor;   displaying the hand on the display;   tracking movement of the hand within the specific area of the back side, wherein said movement is caused by a user intending to manipulate the displayed 3D object;   detecting a collision between the displayed hand and the displayed 3D object; and   modifying an image of the 3D object displayed on the device,   wherein the device is a 3D in-line mediator between the user and the 3D object.   
     
     
         2 . A method as recited in  claim 1  further comprising:
 detecting a hand gesture within the specific area.   
     
     
         3 . A method as recited in  claim 1  wherein modifying an image of the 3D object further comprises:
 deforming the image of the 3D object.   
     
     
         4 . A method as recited in  claim 1  wherein modifying an image of the 3D object further comprises:
 moving the image of the 3D object.   
     
     
         5 . A method as recited in  claim 1  further comprising:
 displaying the modified image on the device.   
     
     
         6 . A method as recited in  claim 1  wherein the user reaches behind the device to manipulate a perceived object corresponding to the 3D object, such that the hand is within the specific area of the back side of the device. 
     
     
         7 . A method as recited in  claim 6  further comprising:
 providing the user with visual coherency when the user reaches behind the device.   
     
     
         8 . A method as recited in  claim 1  wherein tracking movement of the hand further comprises:
 processing depth data of said hand movement.   
     
     
         9 . A method as recited in  claim 1  further comprising:
 executing tracking software.   
     
     
         10 . A method as recited in  claim 1  wherein the sensor is a tracking component that faces outward from the back side of the device and wherein the sensor is a camera. 
     
     
         11 . A method as recited in  claim 1  wherein displaying the hand further comprising:
 displaying a composited image of the hand on the display.   
     
     
         12 . A method as recited in  claim 1  wherein displaying the hand further comprising:
 displaying a virtualized image of the hand on the display.   
     
     
         13 . A method as recited in  claim 1  further comprising:
 providing haptic feedback to the hand when a collision is detected between the displayed hand and the 3D object.   
     
     
         14 . A method as recited in  claim 1  wherein there is no contact between the hand and the back side of the device or with the display. 
     
     
         15 . A device having a display, the device comprising:
 a processor;   a memory storing digital 3D content data;   a tracking sensor component for tracking movement of an object in proximity of the device, wherein the tracking sensor component is on a back side of the device facing away from a user;   a hand tracking module for processing movement data related to a user hand; and   a hand-3D object collision module for detecting a collision between the user hand and a 3D object.   
     
     
         16 . A device as recited in  claim 15  further comprising:
 a face tracking sensor component for tracking face movement in proximity of a front side of a device; and   a face tracking module for processing face movement data related to user face movement in front of the device.   
     
     
         17 . A device as recited in  claim 15  further comprising:
 a hand gesture detection module for detecting user hand gestures made within range of the tracking sensor component.   
     
     
         18 . A device as recited in  claim 15  further comprising:
 a tactile feedback controller for providing tactile feedback to the user hand.   
     
     
         19 . A device as recited in  claim 15  wherein a tracking sensor component is a camera-based component. 
     
     
         20 . A device as recited in  claim 15  wherein a tracking sensor component is one of an image differentiator, infrared detector, optic flow component, and spectral processor. 
     
     
         21 . A device as recited in  claim 15  wherein the tracking sensor component tracks movements of the hand when the user moves the hand behind the device with the range of the tracking sensor component. 
     
     
         22 . A device as recited in  claim 15  wherein the device is one of a mobile device, a nomadic device, and a stationary device. 
     
     
         23 . A device as recited in  claim 15  further comprising a network interface for connecting to a network to receive digital 3D content data. 
     
     
         24 . An apparatus for manipulating digital 3D content, the apparatus having a front side with a display and a back side, the apparatus comprising:
 means for detecting a hand within a specific area of the back side of the apparatus, the back side having an sensor;   means for displaying the hand on the apparatus;   means for tracking movement of the hand within the specific area of the back side, wherein said movement is caused by a user intending to manipulate a displayed 3D object;   means for detecting a collision between the displayed hand and the displayed 3D object; and   means for modifying an image of the 3D object displayed on the apparatus,   wherein the apparatus is a 3D in-line mediator between the user and the 3D object.   
     
     
         25 . An apparatus as recited in  claim 24  further comprising:
 means for detecting a hand gesture within the specific area.   
     
     
         26 . An apparatus as recited in  claim 24  wherein means for modifying an image of the 3D object further comprises:
 means for moving the image of the 3D object.   
     
     
         27 . An apparatus as recited in  claim 24  further comprising:
 means for displaying the modified image on the apparatus.   
     
     
         28 . An apparatus as recited in  claim 24  wherein the user reaches behind the apparatus to manipulate a perceived object corresponding to the 3D object, such that the hand is within the specific area of the back side of the apparatus. 
     
     
         29 . An apparatus as recited in  claim 24  wherein means for tracking movement of the hand further comprises:
 means for processing depth data of said hand movement.   
     
     
         30 . An apparatus as recited in  claim 24  wherein the sensor is a tracking component that faces outward from the back side of the apparatus and wherein the sensor is a camera. 
     
     
         31 . An apparatus as recited in  claim 24  wherein means for displaying the hand further comprises:
 means for displaying a composited image of the hand on the apparatus.   
     
     
         32 . An apparatus as recited in  claim 24  further comprising:
 means for providing haptic feedback to the hand when a collision is detected between the displayed hand and the 3D object.