Enhanced 3d interfacing for remote devices

Abstract

Operating a computerized system includes presenting user interface elements on a display screen. A first gesture made in a three-dimensional space by a part of a body of a user is detected. In response to the first gesture, an area of the display screen selected by the user is identified, and a magnification level of one or more of the user elements appearing in the selected area on the display screen is increased. After increasing the magnification level, a second gesture made by the part of the body of the user is detected so as to select one of the user interface elements that appear in the selected area.

1 . A method for operating a computerized system, comprising the steps of:
 presenting user interface elements on a display screen of the computerized system;   detecting a first gesture made in a three-dimensional space by a distal portion of an upper extremity of a user while a segment of the distal portion thereof rests on a surface;   responsively to the first gesture, identifying an area of the display screen selected by the user and displaying a corresponding user interface element; and   after displaying the corresponding user interface element, detecting a second gesture made by the distal portion while the segment continues to rest on the surface so as to select one of the user interface elements that appears in the selected area.   
     
     
         2 . The method according to  claim 1 , further comprising mapping an operation to the corresponding user interface element, wherein the second gesture causes the operation to be performed. 
     
     
         3 . The method according to  claim 1 , further comprising:
 mapping a three-dimensional location of the distal portion of the upper extremity to two parameters of a two-dimensional parametric surface, which is a section of a sphere that corresponds to possible locations of the distal portion of the upper extremity while the segment of the distal portion rests on a surface; and   mapping the two parameters to corresponding parameters in a planar two-dimensional coordinate system of the display screen where the user interface elements are located.   
     
     
         4 . The method according to  claim 3 , wherein the first gesture describes a first arc-like movement forming at least a portion of a horizontal circular arc mapped to the section of the sphere as a pointing command and wherein the second gesture comprises a second arc-like movement mapped to the section of the sphere as a selection command. 
     
     
         5 . The method according to  claim 4 , wherein the pointing command comprises pointing to a letter on the display screen and wherein the selection command comprises inputting the letter to the system. 
     
     
         6 . The method according to  claim 5 , wherein a threshold is defined, and comprising displaying a subset of letters when a magnitude of the arc-like movement is less than the threshold and inputting the letter when the magnitude of the arc-like movement is greater than the threshold. 
     
     
         7 . The method according to  claim 6 , wherein the subset of letters is shifted on the display screen using a language model to determine a probability of a preferred letter, and wherein inputting the letter is performed in a continuous motion. 
     
     
         8 . The method according to  claim 1 , wherein the segment of the distal portion comprises an elbow. 
     
     
         9 . The method according to  claim 1 , wherein the segment of the distal portion comprises a wrist. 
     
     
         10 . The method according to  claim 1 , wherein the segment of the distal portion comprises a forearm. 
     
     
         11 . A method for operating a computerized system, comprising the steps of:
 presenting user interface elements on a display screen of the computerized system;   detecting a first gesture made in a three-dimensional space by a part of a body of a user;   responsively to the first gesture, identifying an area of the display screen selected by the user and increasing a magnification level of one or more of the user interface elements appearing in the selected area on the display screen; and   after increasing the magnification level, detecting a second gesture made by the part of the body of the user so as to select one of the user interface elements that appears in the selected area.   
     
     
         12 . The method according to  claim 11 , and comprising detecting a third gesture made by the part of the body, and decreasing the magnification level responsively to the third gesture. 
     
     
         13 . The method according to  claim 12 , wherein the first gesture and the third gesture comprise circular motions of a hand of the user in opposite, respective directions. 
     
     
         14 . The method according to  claim 11 , wherein detecting the second gesture comprises actuating a shortcut on the display screen in response to the second gesture. 
     
     
         15 . The method according to  claim 11 , wherein detecting the second gesture comprises selecting a magnified alphanumeric symbol indicated by the second gesture on the display screen. 
     
     
         16 . The method according to  claim 15 , wherein selecting the magnified alphanumeric symbol comprises adding the selected magnified alphanumeric symbol to a word spelled on the display screen, and wherein the method comprises detecting a third gesture made by the part of the body, opposite to the second gesture, and removing one or more symbols from the word responsively to the third gesture. 
     
     
         17 . The method according to  claim 11 , wherein the first gesture comprises a three-dimensional movement by the part of the body of the user. 
     
     
         18 . The method according to  claim 17 , wherein presenting user interface elements comprises displaying a plurality of symbols arranged in at least one arc. 
     
     
         19 . The method according to  claim 18 , wherein displaying the plurality of symbols comprises presenting a set of symbols in a first arc, and wherein increasing the magnification level comprises presenting a magnified subset of the set of symbols in a second arc adjacent to the first arc. 
     
     
         20 . The method according to  claim 18 , wherein detecting the first gesture comprises detecting an arcuate movement of a hand of the user, and associating the arcuate movement with the plurality of symbols in the at least one arc. 
     
     
         21 . The method according to  claim 11 , wherein presenting user interface elements comprises presenting a sequence of textual characters, and wherein increasing the magnification level comprises displaying further characters for addition to the sequence using a language model to select the further characters. 
     
     
         22 . The method according to  claim 21 , wherein detecting the first gesture comprises scrolling forward or backward along the sequence responsively to first movements of a hand of the user in first and second directions along the sequence, and wherein detecting the second gesture comprises selecting the further characters for addition to the sequence responsively to second movements of the hand in at least a third direction perpendicular to the first and second directions. 
     
     
         23 . Apparatus for operating a computerized system, the apparatus comprising:
 a sensing device, which is configured to detect at least a part of a body of a user;   a display screen, which is configured to present user interface elements; and   a processor, which is coupled to the sensing device so as to detect a first gesture made in a three-dimensional space by the part of the body, and which is configured to identify, responsively to the first gesture, an area of the display screen selected by the user and to increase a magnification level of one or more of the user interface elements appearing in the selected area on the display screen, and after increasing the magnification level, to detect a second gesture made by the part of the body so as to select one of the user interface elements that appears in the selected area.   
     
     
         24 . The apparatus according to  claim 23 , wherein the sensing device is further configured to detect a control entity grasped by the part of the body and wherein at least the first gesture is performed using the control entity. 
     
     
         25 . The apparatus according to  claim 23 , and comprising detecting a third gesture made by the part of the body, and decreasing the magnification level responsively to the third gesture. 
     
     
         26 . The apparatus according to  claim 25 , wherein the first gesture and the third gesture comprise circular motions of a hand of the user in opposite, respective directions. 
     
     
         27 . The apparatus according to  claim 23 , wherein the processor is configured to detect the second gesture by actuating a shortcut on the display screen in response to the second gesture. 
     
     
         28 . The apparatus according to  claim 23 , wherein the processor is configured to detect the second gesture by selecting a magnified alphanumeric symbol indicated by the second gesture on the display screen. 
     
     
         29 . The apparatus according to  claim 28 , wherein selecting the magnified alphanumeric symbol comprises adding the selected symbol to a word spelled on the display screen, and wherein the method comprises configuring the processor to detect a third gesture made by the part of the body, opposite to the second gesture, and to remove one or more symbols from the word responsively to the third gesture. 
     
     
         30 . The apparatus according to  claim 23 , wherein the first gesture comprises a three-dimensional movement by the part of the body of the user. 
     
     
         31 . The apparatus according to  claim 30 , wherein the processor is configured to present the user interface elements by displaying a plurality of symbols arranged in at least one arc. 
     
     
         32 . The apparatus according to  claim 31 , wherein displaying the plurality of symbols comprises presenting a set of symbols in a first arc, and wherein the processor is configured to increase the magnification level by presenting a magnified subset of the set of symbols in a second arc adjacent to the first arc. 
     
     
         33 . The apparatus according to  claim 31 , wherein the processor is configured to detect the first gesture by detecting an arcuate movement of a hand of the user, and associating the arcuate movement with the plurality of symbols in the at least one arc. 
     
     
         34 . A computer software product for operating a computerized system, comprising a computer storage medium in which computer program instructions are stored, which instructions, when executed by a computer, cause the computer to present user interface elements on a display screen of the computerized system, to detect a first gesture made in a three-dimensional space by a part of a body of a user, and responsively to the first gesture, to identify an area of the display screen selected by the user and to increase a magnification level of one or more of the user interface elements appearing in the selected area on the display screen, and after increasing the magnification level, to detect a second gesture made by the part of the body of the user so as to select one of the user interface elements that appears in the selected area. 
     
     
         35 . The product according to  claim 34 , wherein the instructions cause the computer to detect a third gesture made by the part of the body, and to decrease the magnification level responsively to the third gesture. 
     
     
         36 . The product according to  claim 35 , wherein the first gesture and the third gesture comprise circular motions of a hand of the user in opposite, respective directions. 
     
     
         37 . The product according to  claim 34 , wherein the instructions cause the computer to detect the second gesture by actuating a shortcut on the display screen in response to the second gesture. 
     
     
         38 . The product according to  claim 34 , wherein the instructions cause the computer to detect the second gesture by selecting a magnified alphanumeric symbol indicated by the second gesture on the display screen. 
     
     
         39 . The product according to  claim 38 , wherein selecting the magnified alphanumeric symbol comprises adding the selected symbol to a word spelled on the display screen, and wherein the instructions cause the computer to detect a third gesture made by the part of the body, opposite to the second gesture, and to remove one or more symbols from the word responsively to the third gesture. 
     
     
         40 . The product according to  claim 34 , wherein the first gesture comprises a three-dimensional movement by the part of the body of the user. 
     
     
         41 . The product according to  claim 40 , wherein the instructions cause the computer to present the user interface elements by displaying a plurality of symbols arranged in at least one arc. 
     
     
         42 . The product according to  claim 41 , wherein the instructions cause the computer to display the plurality of symbols by presenting a set of symbols in a first arc, and wherein the instructions cause the computer to increase the magnification level by presenting a magnified subset of the set of symbols in a second arc adjacent to the first arc. 
     
     
         43 . The product according to  claim 41 , wherein the instructions cause the computer to detect the first gesture by detecting an arcuate movement of a hand of the user, and associating the arcuate movement with the plurality of symbols in the at least one arc. 
     
     
         44 . A method for operating a computerized system, comprising the steps of:
 presenting user interface elements on a display screen of the computerized system;   detecting a gesture made in a three-dimensional space by a part of a body of a user; and   while the user performs the gesture, continuously modifying, responsively to a direction of the gesture, one or more of the user interface elements on the display screen.   
     
     
         45 . The method according to  claim 44 , wherein continuously modifying the one or more of the user interface elements comprises increasing or decreasing a magnification level of at least one of the user interface elements. 
     
     
         46 . The method according to  claim 45 , wherein increasing the magnification level comprises zooming in on a user interface element toward which the gesture is directed. 
     
     
         47 . The method according to  claim 44 , wherein presenting the user interface elements comprises presenting a sequence of textual characters, and wherein continuously modifying the one or more of the user interface elements comprises adding characters to the sequence while scrolling over the sequence responsively to the gesture. 
     
     
         48 . The method according to  claim 47 , wherein adding the characters comprises presenting choices of further characters to add to the sequence, using a language model to determine the choices, and selecting at least one of the choices responsively to the gesture. 
     
     
         49 . The method according to  claim 48 , wherein presenting the choices comprises determining, based on the language model, a respective likelihood of correctness of each of the choices, and displaying the choices so that an effort required by the user to select a given choice is a decreasing function of the likelihood. 
     
     
         50 . The method according to  claim 47 , wherein continuously modifying the one or more of the user interface elements comprises scrolling forward or backward along the sequence responsively to first movements of a hand of the user in first and second directions along the sequence, and selecting further characters for addition to the sequence responsively to second movements of the hand in at least a third direction perpendicular to the first and second directions.