Architecture for controlling a computer using hand gestures

Abstract

Architecture for implementing a perceptual user interface. The architecture comprises alternative modalities for controlling computer application programs and manipulating on-screen objects through hand gestures or a combination of hand gestures and verbal commands. The perceptual user interface system includes a tracking component that detects object characteristics of at least one of a plurality of objects within a scene, and tracks the respective object. Detection of object characteristics is based at least in part upon image comparison of a plurality of images relative to a course mapping of the images. A seeding component iteratively seeds the tracking component with object hypotheses based upon the presence of the object characteristics and the image comparison. A filtering component selectively removes the tracked object from the object hypotheses and/or at least one object hypothesis from the set of object hypotheses based upon predetermined removal criteria.

1 . A method of determining a command, comprising:
 capturing an image of an object with a camera;   determining a gesture based at least partly upon the image;   detecting an audio input; and   determining, at one or more processors, the command based at least partly upon the gesture and the audio input.   
   
   
       2 . The method of  claim 1 , further comprising:
 determining a depth of the object; and   determining the command based at least partly upon the depth of the object.   
   
   
       3 . The method of  claim 2 , wherein determining the depth of the object includes capturing a second image of the object with a second camera. 
   
   
       4 . The method of  claim 1 , wherein the camera is a video camera. 
   
   
       5 . The method of  claim 1 , wherein the camera detects visible light. 
   
   
       6 . The method of  claim 1 , wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image. 
   
   
       7 . A computer-readable medium having instruction that cause a processor to execute steps, the steps comprising:
 capturing an image of an object with a camera;   determining a gesture based at least partly upon the image;   detecting an audio input; and   determining, at one or more processors, a command based at least partly upon the gesture and the audio input.   
   
   
       8 . The computer-readable medium of  claim 7 , the steps further comprising:
 determining a depth of the object; and   determining the command based at least partly upon the depth of the object.   
   
   
       9 . The computer-readable medium of  claim 8 , wherein determining the depth of the object includes capturing a second image of the object with a second camera. 
   
   
       10 . The computer-readable medium of  claim 7 , wherein the camera is a video camera. 
   
   
       11 . The computer-readable medium of  claim 7 , wherein the camera detects visible light. 
   
   
       12 . The computer-readable medium of  claim 7 , wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image. 
   
   
       13 . A command determining system, comprising:
 a camera configured to capture an image of an object;   a first determiner configured to determine a gesture based at least partly upon the image;   an audio detection unit configured to detect an audio input; and   a second determiner configured to determine the command based at least partly upon the gesture and the audio input.   
   
   
       14 . The command determining system of  claim 13 , further comprising:
 a third determiner configured to determine a depth of the object, wherein the second determiner is further configured to determine the command based at least partly upon the depth of the object.   
   
   
       15 . The command determining system of  claim 14 , wherein determining the depth of the object includes capturing a second image of the object with a second camera. 
   
   
       16 . The command determining system of  claim 13 , wherein the camera is a video camera. 
   
   
       17 . The command determining system of  claim 13 , wherein the camera detects visible light. 
   
   
       18 . The command determining system of  claim 13 , wherein determining the gesture includes capturing a second image of the object with the camera and comparing the image with the second image.