Apparatus and method for detecting a pointer relative to a touch surface

Abstract

An apparatus for detecting a pointer relative to a touch surface includes at least two spaced imaging assemblies having overlapping fields of view encompassing the touch surface. The imaging assemblies see the touch surface in three-dimensions as a perspective view. The imaging assemblies acquire overlapping images from different locations. A processor receives and processes image data generated by at least one of the imaging assemblies to determine the location of the pointer relative to the touch surface.

1 . An interactive input system comprising:
 at least two imaging devices having overlapping fields of view of an input surface, each of said imaging devices comprising a lens having at least one form of distortion; and   processing structure determining at least one distortion parameter of each said lens during calibration.   
   
   
       2 . An interactive input system according to  claim 1  wherein said at least one form of distortion is radial distortion. 
   
   
       3 . An interactive input system according to  claim 1  wherein said at least one form of distortion is decentering distortion. 
   
   
       4 . An interactive input system according to  claim 1  wherein said at least one form of distortion is radial and decentering distortion. 
   
   
       5 . An interactive input system according to  claim 1  wherein the distortion of each said lens is expressed by:
   Δ x =( x−x   0 ) K   1 (( x−x   0 ) 2 +( y−y   0 ) 2 )     Δ y =( y−y   0 ) K   1 (( x   1   −x   0 ) 2 +( y−y   0 ) 2 )   
     where:
 (Δ x , Δ y ) represent radial and decentering distortion of the lens; 
 (x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in a three-dimensional scene; 
 (x 0 , y 0 ) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and 
 K 1  is a lens parameter. 
 
   
   
       6 . An interactive input system according to  claim 1  wherein said processing structure determines said at least one distortion parameter of each said lens during a self-calibration routine. 
   
   
       7 . An interactive input system according to  claim 6  wherein said at least one form of distortion is radial distortion. 
   
   
       8 . An interactive input system according to  claim 6  wherein said at least one form of distortion is decentering distortion. 
   
   
       9 . An interactive input system according to  claim 6  wherein said at least one form of distortion is radial and decentering distortion. 
   
   
       10 . An interactive input system according to  claim 6  wherein the distortion of each said lens is expressed by:
   Δ x =( x−x   0 ) K   1 (( x−x   0 ) 2 +( y−y   0 ) 2 )     Δ y =( y−y   0 ) K   1 (( x   1   −x   0 ) 2 +( y−y   0 ) 2 )   
     where:
 (Δ x , Δ y ) represent radial and decentering distortion of the lens; 
 (x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in a three-dimensional scene; 
 (x 0 , y 0 ) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and 
 K 1  is a lens parameter. 
 
   
   
       11 . An interactive input system according to  claim 1  wherein said input surface is one of planar, curved and non-planar. 
   
   
       12 . An interactive input system according to  claim 11  wherein said processing structure determines said at least one distortion parameter of each said lens during a self-calibration routine. 
   
   
       13 . An interactive input system according to  claim 12  wherein said at least one form of distortion is radial distortion. 
   
   
       14 . An interactive input system according to  claim 12  wherein said at least one form of distortion is decentering distortion. 
   
   
       15 . An interactive input system according to  claim 12  wherein said at least one form of distortion is radial and decentering distortion. 
   
   
       16 . An interactive input system according to  claim 12  wherein the distortion of each said lens is expressed by:
   Δ x =( x−x   0 ) K   1 (( x−x   0 ) 2 +( y−y   0 ) 2 )     Δ y =( y−y   0 ) K   1 (( x   1   −x   0 ) 2 +( y−y   0 ) 2 )   
     where:
 (Δ x , Δ y ) represent radial and decentering distortion of the lens; 
 (x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in a three-dimensional scene; 
 (x 0 , y 0 ) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and 
 K 1  is a lens parameter. 
 
   
   
       17 . An interactive input system according to  claim 1  wherein said input surface is any surface within the overlapping fields of view of said imaging devices. 
   
   
       18 . An interactive input system according to  claim 1  wherein each of said imaging devices sees said input surface in three-dimensions as a perspective view and wherein said processing structure further processes image data generated by at least one of said imaging devices to determine the location of a pointer relative to said input surface. 
   
   
       19 . An interactive input system according to  claim 18  wherein each imaging device is calibrated to establish the relationship between points (X, Y, Z) in its perspective view and points (x, y) in acquired images, each imaging device generating pointer co-ordinate data when a pointer is captured in an acquired image. 
   
   
       20 . An interactive input system according to  claim 19  wherein said processing structure triangulates the pointer co-ordinate data to determine the location of the pointer relative to said input surface. 
   
   
       21 . An interactive input system according to  claim 20  wherein each imaging device is positioned relative to said input surface so that at a minimum the entire periphery of the input surface is within its field of view. 
   
   
       22 . An apparatus according to  claim 21  wherein said input surface is bordered by a bezel. 
   
   
       23 . An interactive input system according to  claim 20  wherein said processing structure determines said at least one distortion parameter of each said lens during a self-calibration routine. 
   
   
       24 . An interactive input system according to  claim 23  wherein said at least one form of distortion is radial distortion. 
   
   
       25 . An interactive input system according to  claim 23  wherein said at least one form of distortion is decentering distortion. 
   
   
       26 . An interactive input system according to  claim 23  wherein said at least one form of distortion is radial and decentering distortion. 
   
   
       27 . An interactive input system according to  claim 23  wherein the distortion of each said lens is expressed by:
   Δ x =( x−x   0 ) K   1 (( x−x   0 ) 2 +( y−y   0 ) 2 )     Δ y =( y−y   0 ) K   1 (( x   1   −x   0 ) 2 +( y−y   0 ) 2 )   
     where:
 (Δ x , Δ y ) represent radial and decentering distortion of the lens; 
 (x, y) are the co-ordinates of a point in an image captured by the imaging device corresponding to a point (X, Y, Z) in the three-dimensional scene; 
 (x 0 , y 0 ) are the co-ordinates of the principal point of the imaging device, the location at which the optical axis of the imaging device meets the focal plane of the imaging device with the optical axis being approximately normal to the focal plane; and 
 K 1  is a lens parameter. 
 
   
   
       28 . An interactive input system according to  claim 23  wherein said input surface is one of planar, curved and non-planar. 
   
   
       29 . An interactive input system according to  claim 20  wherein each imaging device generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image. 
   
   
       30 . An interactive input system according to  claim 29  wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said input surface. 
   
   
       31 . An interactive input system according to  claim 30  wherein said processing structure ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level. 
   
   
       32 . An interactive input system according to  claim 20  wherein the imaging device that detects a pointer in its acquired image first communicates data to the other imaging device to assist that imaging device to detect the pointer in its acquired image. 
   
   
       33 . An interactive input system according to  claim 32  wherein each imaging device also generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image. 
   
   
       34 . An interactive input system according to  claim 33  wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said input surface. 
   
   
       35 . An interactive input system according to  claim 34  wherein said processing structure ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level. 
   
   
       36 . An interactive input system according to  claim 20  wherein each imaging device processes a subset of pixels in each acquired image. 
   
   
       37 . An interactive input system according to  claim 17  wherein said imaging devices are portable. 
   
   
       38 . A camera-based interactive input system comprising:
 a touch surface on which contacts are made using a pointer;   camera devices looking at said touch surface from different vantages and having overlapping fields, said camera devices acquiring images of said touch surface, each of said camera devices having an imperfect lens; and   processing structure receiving and processing said image data to determine the location of said pointer relative to said touch surface via triangulation, said processing structure compensating for image distortion as a result of the imperfect lens of each said camera device.   
   
   
       39 . A camera-based interactive input system according to  claim 38  wherein said processing structure determines at least one distortion parameter of each said lens during a self-calibration routine. 
   
   
       40 . A camera-based interactive input system according to  claim 39  wherein said at least one form of distortion is radial distortion. 
   
   
       41 . A camera-based interactive input system according to  claim 40  wherein said at least one form of distortion is decentering distortion. 
   
   
       42 . A camera-based interactive input system according to  claim 40  wherein said at least one form of distortion is radial and decentering distortion. 
   
   
       43 . A camera-based interactive input system according to  claim 40  wherein the distortion of each said lens is expressed by:
   Δ x =( x−x   0 ) K   1 (( x−x   0 ) 2 +( y−y   0 ) 2 )     Δ y =( y−y   0 ) K   1 (( x   1   −x   0 ) 2 +( y−y   0 ) 2 )   
     where:
 (Δ x , Δ y ) represent radial and decentering distortion of the lens; 
 (x, y) are the co-ordinates of a point in an image captured by the camera device corresponding to a point (X, Y, Z) in a three-dimensional scene; 
 (x 0 , y 0 ) are the co-ordinates of the principal point of the camera device, the location at which the optical axis of the camera device meets the focal plane of the camera device with the optical axis being approximately normal to the focal plane; and 
 K 1  is a lens parameter. 
 
   
   
       44 . A camera-based interactive input system according to  claim 39  wherein said touch surface is one of planar, curved and non-planar. 
   
   
       45 . A camera-based interactive input system according to  claim 39  wherein each camera device generates a certainty value representing the degree of certainty that the camera device has positively identified the pointer in the acquired image. 
   
   
       46 . A camera-based interactive input system according to  claim 45  wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface. 
   
   
       47 . A camera-based interactive input system according to  claim 46  wherein said processing structure ignores pointer co-ordinate data generated by said camera device when the certainty value associated therewith is below a threshold level. 
   
   
       48 . A camera-based interactive input system according to  claim 39  wherein the camera device that detects a pointer in its acquired image first communicates data to the other camera device to assist that camera device to detect the pointer in its acquired image. 
   
   
       49 . A camera-based interactive input system according to  claim 48  wherein each camera device also generates a certainty value representing the degree of certainty that the camera device has positively identified the pointer in the acquired image. 
   
   
       50 . A camera-based interactive input system according to  claim 49  wherein said certainty value is used by said processing structure to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface. 
   
   
       51 . A camera-based interactive input system according to  claim 50  wherein said processing structure ignores pointer co-ordinate data generated by said camera device when the certainty value associated therewith is below a threshold level. 
   
   
       52 . A camera-based interactive input system according to  claim 39  wherein each camera device processes a subset of pixels in each acquired image. 
   
   
       53 . A camera-based interactive input system according to  claim 44  wherein said camera devices are portable.