Novel man machine interfaces and applications

Abstract

The invention is aimed at providing affordable methods and apparatus for inputting position, attitude(orientation) or other object characteristic data to computers for the purpose of Computer Aided learning, Teaching, Gaming, Toys, Simulations, Aids to the disabled, Word Processing and other applications. Preferred embodiments of the invention utilize electro-optical sensors, and particularly TV Cameras, providing optically inputted data from specialized datum's on objects and/or natural features of objects. Objects can be both static and in motion, from which individual datum positions and movements can be derived, also with respect to other objects both fixed and moving. Real-time photogrammetry is preferably used to determine relation ships of portions of one or more datums with respect to a plurality of cameras or a single camera processed by a conventional PC.

What is claimed is:  
     
         1 . Apparatus for input by a person of data to a computer having a display comprising 
 One or more Datum means provided on said person, said datum means distinguishable in reflected light    At least one TV Camera having an output    Means for determining from said TV camera output, the position of said datums and/or the orientation of a portion of said person    Means for creating on said display, a representation of at least one object, and;    Means for modifying, manipulating, or positioning said at least one object representation on said screen as a function of the position or orientation of datums or person    
     
     
         2 . Apparatus according to  claim 1  further including light source means for directing light at said member  
     
     
         3 . Apparatus according to  claim 1  wherein at least one of said datums is retroreflective  
     
     
         4 . Apparatus according to  claim 1  wherein at least one of said datums is a natural feature of said member  
     
     
         5 . Apparatus according to  claim 2  wherein said light source is an LED light source  
     
     
         6 . Apparatus according to  claim 2  wherein light from said light source is substantially invisible  
     
     
         7 . Apparatus according to  claim 1  wherein at least one of said datums is distinctive in color  
     
     
         8 . Apparatus according to  claim 1  wherein at least one of said datums is a distinctive shape  
     
     
         9 . Apparatus according to  claim 1  wherein at least two cameras are used  
     
     
         10 . Apparatus according to  claim 9  wherein said cameras provide stereo pair of images of said object  
     
     
         11 . Apparatus according to  claim 9  wherein said cameras look at different sides of said person  
     
     
         12 . Apparatus according to  claim 9  wherein said cameras look at different times at said person  
     
     
         13 . Apparatus according to  claim 1  wherein said cameras are provided with the display  
     
     
         14 . Apparatus according to  claim 1  including further means of affixing a datum  
     
     
         15 . Apparatus according to  claim 1  including further voice input means to said computer  
     
     
         16 . Apparatus according to  claim 1  including further means to allow said camera to see objects associated with said person  
     
     
         17 . Apparatus according to  claim 1  including bandpass filter means associated with at least one of said cameras  
     
     
         18 . A method by which a person may input data to a computer, the method comprising: 
 providing a target on said person    providing a source of light to create an illumination field;    providing at least one TV camera proximate said light source such that the camera can detect reflection of light from said object in said illumination field    detecting radiation reflected from said within the illumination field to create at least one tv image containing an image of said person    determining from said tv image information concerning the position and/or orientation of said target, and    providing a desired input to said computer using said determined information    
     
     
         19 . A method according to  claim 20  wherein said member contains at least one retroreflective datum  
     
     
         20 . A method according to  claim 20  wherein said light source is an LED light source  
     
     
         21 . A method according to  claim 20  wherein said Light source is substantially invisible  
     
     
         22 . A Method for input of information by a person to a computer having a display representing at least one object comprising the steps of 
 Providing a datum associated with said person    Electro-optically determining, the position of at least one datum on said person in 3 dimensions    Providing a representation of at least one computer generated virtual object on said display, and    Using said determined position or orientation data, manipulating said object displayed by said computer to provide a desired visual display or audio response    
     
     
         23 . A method according to  claim 24  wherein at least one of said datums is retroreflective  
     
     
         24 . A method according to  claim 24  wherein said datum is distinctive in color  
     
     
         25 . A method according to  claim 24  wherein said datum is a distinctive shape  
     
     
         26 . A method according to  claim 24  wherein at least two cameras are used  
     
     
         27 . A method according to  claim 24  wherein said cameras provide stereo pair of images of said datum  
     
     
         28 . A method according to  claim 24  wherein said cameras look at different sides of said datum  
     
     
         29 . A method according to  claim 24  wherein said cameras are provided with said display  
     
     
         30 . A method according to  claim 24  including further step of affixing a datum  
     
     
         31 . A method according to  claim 24  wherein at least one of said datums is a natural object feature  
     
     
         32 . A method according to  claim 24  including the further step of recognizing voice input  
     
     
         33 . A method according to  claim 24  including temporary filter means for at least one lens of said cameras  
     
     
         34 . A method according to  claim 24  including the further step of sensing the gray level image of a portion of said user.  
     
     
         35 . A method according to  claim 24  including the further step of changing Sound output as a function of said data  
     
     
         36 . A method according to  claim 24  including the further step of using said display or audio for learning  
     
     
         37 . A method according to  claim 24  including the further step of analyzing movement of said datum  
     
     
         38 . A method according to  claim 24  including the further step of determining the position or orientation of a member  
     
     
         39 . Means for aiding the determination of locations of points on a human, comprising 
 means providing decoration for said human, said means easily visible by a TV camera or other electro-optical device, and    Means for temporarily providing said decoration means on said human    
     
     
         40 . Apparatus according to  claim 39 , wherein said decoration means is retroreflective  
     
     
         41 . Apparatus according to  claim 39 , wherein said decoration is selected from a group comprising rings, bracelets, watches, lipstick, nail polish,  
     
     
         42 . Apparatus according to  claim 39 , wherein said decoration is part of clothing  
     
     
         43 . A Method for producing a display based experience for a user comprising the steps of; 
 Providing a computer    Providing a large screen TV display of size greater than 42 inches diagonal, the display being controlled by said computer    Providing at least one electro-optical sensor having an output    Processing in said computer said sensor output    From said processing, determining the position or orientation of a portion of a person and/or object camera, and using said computer,    Modifying said display to create a response to an action of said person.    
     
     
         44 . A method according to  claim 44  wherein said display is approximately lifesize.  
     
     
         45 . A method according to  claim 44  wherein said user touches or points at virtual objects depicted on said display  
     
     
         46 . A method according to  claim 44  wherein said user pinches, or grips virtual objects depicted on said display  
     
     
         47 . A method according to  claim 44  wherein said display varies as the users view changes  
     
     
         48 . Method for activity involving an object, comprising the steps of  
     
     
         49 . Providing an object  
     
     
         50 . Determining if features can be sensed by a tv camera  
     
     
         51 . Affixing special datums to said object where features are required for best sensing results,  
     
     
         52 . Recording the locations of features and special datums into a data base.  
     
     
         53 . A method according to  claim 48  wherein said special datum is easily affixed by hand  
     
     
         54 . A method according to  claim 48  wherein said special datum is retroreflective  
     
     
         55 . A method according to  claim 48  wherein said special datum is linear  
     
     
         56 . A method according to  claim 48  wherein said special datum is curvilinear  
     
     
         57 . A method of providing a game or other human activity comprising 
 Providing an object    Providing a member attached to said object and movable with respect thereto    Determining the position or orientation, or change therein, of said member with an electro-optical sensing system    From said determined position or orientation, or change therein, determining an input parameter to a computer program, and    Using said program, provide said game or other activity    
     
     
         58 . A method according to  claim 57  wherein said member is movable by said human  
     
     
         59 . A method according to  claim 57  wherein said member moves as a result of the action of a physical variable  
     
     
         60 . A method according to  claim 57  including the additional step of determining the position or orientation of a portion of said human  
     
     
         61 . A method according to  claim 57  Wherein said sensor is comprised of at least one TV camera  
     
     
         62 . A method according to  claim 57 , wherein said position or motion is determined relative to another member or said object