Surface ui for gesture-based interaction

Abstract

Disclosed is a unique system and method that facilitates gesture-based interaction with a user interface. The system involves an object sensing configured to include a sensing plane vertically or horizontally located between at least two imaging components on one side and a user on the other. The imaging components can acquire input images taken of a view of and through the sensing plane. The images can include objects which are on the sensing plane and/or in the background scene as well as the user as he interacts with the sensing plane. By processing the input images, one output image can be returned which shows the user objects that are in contact with the plane. Thus, objects located at a particular depth can be readily determined. Any other objects located beyond can be “removed” and not seen in the output image.

1 . A method of determining a command, comprising:
 capturing an image of an object with a camera;   determining a gesture based at least partly upon the image;   detecting an audio input; and   determining, at one or more processors, the command based at least partly upon the gesture or the audio input.   
   
   
       2 . The method of  claim 1 , further comprising:
 determining a depth of the object; and   determining the command based at least partly upon the depth of the object.   
   
   
       3 . The method of  claim 2 , wherein determining the depth of the object includes capturing a second image of the object with a second camera. 
   
   
       4 . The method of  claim 1 , wherein the camera is a video camera. 
   
   
       5 . The method of  claim 1 , wherein the camera detects infrared light. 
   
   
       6 . The method of  claim 1 , wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image. 
   
   
       7 . A computer-readable medium having instruction that cause a processor to execute steps, the steps comprising:
 capturing an image of an object with a camera;   determining a gesture based at least partly upon the image;   detecting an audio input; and   determining, at one or more processors, a command based at least partly upon the gesture or the audio input.   
   
   
       8 . The computer-readable medium of  claim 7 , the steps further comprising:
 determining a depth of the object; and   determining the command based at least partly upon the depth of the object.   
   
   
       9 . The computer-readable medium of  claim 8 , wherein determining the depth of the object includes capturing a second image of the object with a second camera. 
   
   
       10 . The computer-readable medium of  claim 7 , wherein the camera is a video camera. 
   
   
       11 . The computer-readable medium of  claim 7 , wherein the camera detects infrared light. 
   
   
       12 . The computer-readable medium of  claim 7 , wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image. 
   
   
       13 . A command determining system, comprising:
 a camera configured to capture an image of an object;   a first determiner configured to determine a gesture based at least partly upon the image;   an audio detection unit configured to detect an audio input; and   a second determiner configured to determine the command based at least partly upon the gesture or the audio input.   
   
   
       14 . The command determining system of  claim 13 , further comprising:
 a third determiner configured to determine a depth of the object, wherein the second determiner is further configured to determine the command based at least partly upon the depth of the object.   
   
   
       15 . The command determining system of  claim 14 , wherein determining the depth of the object includes capturing a second image of the object with a second camera. 
   
   
       16 . The command determining system of  claim 13 , wherein the camera is a video camera. 
   
   
       17 . The command determining system of  claim 13 , wherein the camera detects infrared light. 
   
   
       18 . The command determining system of  claim 13 , wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.