Method and apparatus for ego-centric 3d human computer interface

Abstract

In the method, a processor generates a three dimensional interface with at least one virtual object, defines a stimulus of the interface, and defines a response to the stimulus. The stimulus is an approach to the virtual object with a finger or other end-effector to within a threshold of the virtual object. When the stimulus is sensed, the response is executed. Stimuli may include touch, click, double click, peg, scale, and swipe gestures. The apparatus includes a processor that generates a three dimensional interface with at least one virtual object, and defines a stimulus for the virtual object and a response to the stimulus. A display outputs the interface and object. A camera or other sensor detects the stimulus, e.g. a gesture with a finger or other end-effector, whereupon the processor executes the response. The apparatus may be part of a head mounted display.

We claim: 
     
         1 . A method, comprising:
 in a processor, generating a three dimensional interface;   generating at least one virtual object in said three dimensional interface;   defining a stimulus of said three dimensional interface, the said stimulus including approaching said at least one virtual object with a end-effector to within a threshold;   defining a response to said stimulus;   sensing said stimulus; and   executing said response.   
     
     
         2 . The method of  claim 1 , wherein:
 said response includes said processor registering a touch input.   
     
     
         3 . The method of  claim 1 , wherein:
 said threshold includes a distance between said end-effector and said virtual object.   
     
     
         4 . The method of  claim 1 , wherein:
 said threshold includes contact between said end-effector and said virtual object.   
     
     
         5 . The method of  claim 1  further comprising:
 defining an interaction zone associated with said virtual object, said threshold including contact with said interaction zone. 
 
     
     
         6 . The method of  claim 1 , wherein:
 said end-effector includes a stylus.   
     
     
         7 . The method of  claim 1 , wherein:
 said end-effector includes a hand.   
     
     
         8 . The method of  claim 1 , wherein:
 said end-effector includes a finger.   
     
     
         9 . The method of  claim 1 , wherein:
 said at least one virtual object defines said response therefor.   
     
     
         10 . The method of  claim 1 , wherein:
 said stimulus further includes withdrawing said end-effector from said at least one virtual object beyond said threshold within a first time interval.   
     
     
         11 . The method of  claim 10 , wherein:
 said response includes said processor registering a single-click input.   
     
     
         12 . The method of  claim 10 , wherein said stimulus further comprises:
 repeating said approach to said at least one virtual object with said end-effector to within said threshold within a second time interval; and   repeating said withdrawal from said end-effector from said at least one virtual object beyond said threshold within said first time interval.   
     
     
         13 . The method of  claim 12 , wherein:
 said response includes said processor registering a double-click input.   
     
     
         14 . The method of  claim 1 , wherein said stimulus further comprises:
 maintaining said end-effector within said threshold of said virtual object for at least a third time interval.   
     
     
         15 . The method of  claim 14 , wherein:
 said response includes said processor registering a peg input.   
     
     
         16 . The method of  claim 14 , wherein:
 said response includes engaging a location of said at least one virtual object with said end-effector.   
     
     
         17 . The method of  claim 14 , comprising:
 if said end-effector is withdrawn from said at least one virtual object beyond said threshold, terminating said response.   
     
     
         18 . The method of  claim 14 , wherein said stimulus includes:
 approaching said at least one virtual object with first and second end-effectors to within said threshold, said first and second end-effectors being separated by a range; and   maintaining said first and second end-effectors within said threshold of said virtual object and maintaining said first and second end-effectors separated by said range for at least said third time interval.   
     
     
         19 . The method of  claim 18 , wherein:
 said response includes said processor registering a peg input.   
     
     
         20 . The method of  claim 18 , wherein:
 said first and second end-effectors include first and second fingers on a hand.   
     
     
         21 . The method of  claim 18 , wherein said stimulus includes:
 said first and second end-effectors approaching said at least one virtual object to within said threshold within a peg time interval.   
     
     
         22 . The method of  claim 18 , wherein said stimulus includes:
 said first and second end-effectors approaching said at least one virtual object to within said threshold substantially simultaneously.   
     
     
         23 . The method of  claim 18  further comprising:
 terminating said response if a separation of said first and second end-effectors is outside said range. 
 
     
     
         24 . The method of  claim 14 , wherein said stimulus further includes:
 approaching said at least one virtual object with a third end-effector to within said threshold,   maintaining said third end-effector within said threshold of said at least one virtual object for at least a third time interval, and   altering a distance between said first and third end-effectors.   
     
     
         25 . The method of  claim 24 , wherein:
 said response includes said processor registering a scale input.   
     
     
         26 . The method of  claim 25 , wherein:
 said response includes a change in a magnitude of at least one dimension of said at least one virtual object.   
     
     
         27 . The method of  claim 14 , wherein said stimulus further includes:
 approaching said at least one virtual object with third and fourth end-effectors to within said threshold, said third and fourth end-effectors being separated by said range,   defining a first composite point from said first and second end-effectors,   defining a second composite point from said third and fourth end-effectors, and   altering a distance between said first and second composite points.   
     
     
         28 . The method of  claim 27 , wherein:
 said response includes said processor registering a scale input.   
     
     
         29 . The method of  claim 1 , wherein said stimulus further includes:
 translating said end-effector, while maintaining said end-effector within said threshold.   
     
     
         30 . The method of  claim 29 , wherein:
 said response includes said processor registering a swipe input.   
     
     
         31 . The method of  claim 29 , wherein:
 said stimulus includes at least one of translating said end-effector up, down, left, right, in, out, clockwise, and counterclockwise, and   said response includes a corresponding one of a group of said processor registering an up-swipe, a down-swipe, a left-swipe, a right-swipe, an in-swipe, an out-swipe, a clockwise-swipe, and a counterclockwise-swipe.   
     
     
         32 . An apparatus, comprising:
 a processor, said processor adapted to generate a three dimensional interface and at least one virtual object therein, said processor adapted to define a stimulus of said three dimensional interface, said stimulus including approaching said at least one virtual object with an end-effector to within a threshold, said processor adapted to define a response to said stimulus, and execute said response;   at least one display, said at least one display adapted to output said three dimensional interface; and   at least one sensor, said sensor adapted to sense said stimulus.   
     
     
         33 . The apparatus of  claim 32 , wherein:
 said response includes said processor registering a touch input.   
     
     
         34 . The apparatus of  claim 32 , wherein:
 said stimulus further includes withdrawing said end-effector from said object beyond said threshold within a first time interval.   
     
     
         35 . The apparatus of  claim 34 , wherein:
 said response includes said processor registering a single-click input.   
     
     
         36 . The apparatus of  claim 32 , wherein:
 said stimulus further includes approaching said at least one virtual object with said end-effector to within said threshold within a second time interval; and   withdrawing said end-effector from said object beyond said threshold within said first time interval.   
     
     
         37 . The apparatus of  claim 36 , wherein:
 said response includes said processor registering a double-click input.   
     
     
         38 . The apparatus of  claim 32 , wherein:
 said stimulus further includes maintaining said end-effector within said threshold of said virtual object for at least a third time interval.   
     
     
         39 . The apparatus of  claim 38 , wherein:
 said response includes said processor registering a peg input.   
     
     
         40 . The apparatus of  claim 38 , wherein:
 said response includes engaging a location of said virtual object with said end-effector.   
     
     
         41 . The apparatus of  claim 32 , wherein:
 said stimulus further includes approaching said virtual object with first and second end-effectors to within said threshold, said first and second end-effectors being separated by a range.   
     
     
         42 . The apparatus of  claim 41 , wherein:
 said response includes said processor registering a peg input.   
     
     
         43 . The apparatus of  claim 32 , wherein said stimulus further includes:
 approaching said at least one virtual object with a third end-effector to within said threshold,   maintaining said third end-effector within said threshold of said at least one virtual object for at least a third time interval, and   altering a distance between said first and third end-effectors.   
     
     
         44 . The apparatus of  claim 43 , wherein:
 said response includes said processor registering a scale input.   
     
     
         45 . The apparatus of  claim 44 , wherein:
 said response includes a change in a magnitude of at least one dimension of said at least one virtual object.   
     
     
         46 . The apparatus of  claim 32 , wherein:
 said stimulus further includes approaching said virtual object with third and fourth end-effectors to within said threshold, said third and fourth end-effectors being separated by said range;   defining a first composite point from said first and second end-effectors;   defining a second composite point from said third and fourth end-effectors; and   altering a distance between said first and second composite points.   
     
     
         47 . The apparatus of  claim 46 , wherein:
 said response includes said processor registering a scale input.   
     
     
         48 . The apparatus of  claim 44 , wherein:
 said stimulus further includes translating said end-effector, while maintaining said end-effector within said threshold.   
     
     
         49 . The apparatus of  claim 48 , wherein:
 said response includes said processor registering a swipe input.   
     
     
         50 . The apparatus of  claim 44 , wherein:
 said display includes a stereo display.   
     
     
         51 . The apparatus of  claim 44 , wherein:
 said end-effector includes a stylus.   
     
     
         52 . The apparatus of  claim 44 , wherein:
 said processor, said display, and said sensor are disposed on a wearable device.   
     
     
         53 . The apparatus of  claim 44 , wherein:
 said processor, said display, and said sensor are disposed on a head mounted display.   
     
     
         54 . A head mounted display, comprising:
 a body adapted to be worn on a head of a user;   a processor engaged with said body, said processor being adapted to generate a stereo three dimensional interface and at least one virtual object therein, to define a stimulus of said at least one virtual object in said stereo three dimensional interface, to define a response to said stimulus, and to execute said response, wherein said stimulus comprises approaching said at least one virtual object with an end-effector to within a threshold;   first and second displays in communication with said processor, said first and second displays being adapted to output said stereo three dimensional interface and said at least one virtual object, said first and second displays being engaged with said body such that said first and second displays are disposed proximate and substantially aligned with eyes of said user when said user wears said body; and   first and second image sensors in communication with said processor, said first and second image sensors being adapted to generate stereo three dimensional images, said first and second image sensors being engaged with said body such that a field of view of said first and second image sensors includes a region in front of said user when said body is worn by said user, said sensors being adapted to detect said stimulus.   
     
     
         55 . An apparatus, comprising:
 means for generating a three dimensional interface;   means for generating at least one virtual object in said three dimensional interface;   means for defining a stimulus of said three dimensional interface, wherein said stimulus includes approaching said at least one virtual object with an end-effector to within a threshold;   means for defining a response to said stimulus;   means for sensing said stimulus; and   means for executing said response.