Method and apparatus for positional error correction in a robotic pool system using a cue-aligned local camera

Abstract

The present invention provides a method and apparatus for accurately positioning a robotic pool-playing device. The system comprises a computer controlled robotic positioning device, such as a gantry robot, that can position a cue over the pool table and place a shot. A global camera is mounted on the ceiling looking down at the table, and the acquired images are transmitted to the computer for analysis to determine the identity and locations of the balls within the table coordinate reference frame. The computer also automatically determines which ball to strike. An aspect of the invention is the use of a local camera, mounted on or near the robotic end-effector in a fixed relationship with the cue, to improve the positioning error of the robotic device prior to placing a shot. By comparing the ball locations perceived from the vantage of the local camera with the known ball locations determined from the global camera image, the invention can calculate the acquired robotic positioning error, which can then be corrected for prior to placing the shot.

1 . A method of playing pool, comprising the steps of:
 a) acquiring a first image of a pool table using a first camera placed above the pool table and positioned that its image plane is substantially parallel to both a playing surface of the pool table and a longitudinal axis of a pool cue and transmitting this image to a computer processing means for analysis, the result of which is a determination of the position and identity of each pool ball on the pool table;   b) said computer processing means planning a shot of a cue ball by the pool cue by calculating a desired position of the pool cue with respect to the cue ball in a pool table frame of reference including a tip position of said pool cue and orientation of the longitudinal axis of the pool cue;   c) said computer processing means instructing a robot connected to said pool cue to position the pool cue to a preferred location from which to place a shot;   d) acquiring a second image of a pool table using a second camera mounted on an end-effector of the robot so that its image plane is substantially perpendicular to both the pool table and said longitudinal axis of the pool cue, the position of said second camera being fixed with respect to said longitudinal axis of the pool cue, transmitting the second image to the computer processing means and determining locations of some of the pool balls within this image within said second camera frame of reference, and wherein if at least three pool balls are visible from both the first and second cameras, then estimating and correcting for an accumulated positional error of the robot and repositioning the pool cue by moving at least one proximal robotic joint which also repositions the second camera; and   e) said computer processing means instructing the robot to place a shot.   
     
     
         2 . (canceled) 
     
     
         3 . (canceled) 
     
     
         4 . The method according to  claim 1  wherein the location of the longitudinal axis of the cue is determined within a coordinate reference frame of the second camera. 
     
     
         5 . The method according to  claim 21  wherein the location of the longitudinal axis of the cue is determined within a coordinate reference frame of the second camera. 
     
     
         6 . The method according to  claim 4  wherein the position of the second camera is fixed with respect to the longitudinal cue axis, and the location of the cue axis within the coordinate reference frame of the second camera is determined in a preprocessing step including capturing an image using a third camera mounted so that its image plane is substantially perpendicular to the longitudinal axis of the cue, and including re-positioning the pool cue with one degree-of-freedom by sliding along the cue axis repeatedly and the center of the cue tip within the third camera image is determined at each extreme cue position, and wherein by comparing the centers of the tip at each extreme position, computing the angle between the cue axis and the third camera whereupon the robotic joints are repositioned so as to improve the perpendicularity of the cue axis with the third camera image plane, and the process iterates, and once the cue is thus positioned so that its axis is accurately perpendicular to the third camera image plane, the relative locations of the third camera and the local camera are then determined by estimating a rigid transformation that relates them, and the cue location is then directly propagated to the local camera coordinate reference frame. 
     
     
         7 . The method according to  claim 5  wherein the position of the second camera is fixed with respect to the longitudinal cue axis, and the location of the longitudinal axis of the cue within the coordinate reference frame of the second camera is determined in a preprocessing step including capturing an image using a third camera mounted so that its image plane is substantially perpendicular to the longitudinal axis of the cue, and including re-positioning the pool cue with one degree-of-freedom by sliding along the cue axis repeatedly and the center of the cue tip within the third camera image is determined at each extreme cue position, and wherein by comparing the centers of the tip at each extreme position, computing the angle between the cue axis and the third camera whereupon the robotic joints are repositioned so as to improve the perpendicularity of the cue axis with the third camera image plane, and the process iterates, and once the cue is thus positioned so that its axis is accurately perpendicular to the third camera image plane, the relative locations of the third camera and the local camera are then determined by estimating a rigid transformation that relates them, and the cue location is then directly propagated to the local camera coordinate reference frame. 
     
     
         8 . The method according to  claim 6  wherein the location of the cue ball and object ball within the coordinate reference frame of the local camera is determined in a preprocessing step, and an estimation and correction of some components of an accumulated positional error of the robot is substantially achieved by matching the current positions of the cue and object balls with those positions that were respectively determined in the preprocessing step. 
     
     
         9 . The method according to  claim 7  wherein the location of the cue ball and object ball within the coordinate reference frame of the local camera is determined in a preprocessing step, and an estimation and correction of some components of an accumulated positional error of the robot is substantially achieved by matching the current positions of the cue and object balls with those positions that were respectively determined in the preprocessing step. 
     
     
         10 . (canceled) 
     
     
         11 . The method according to  claim 22  wherein locations of the landmarks have been predetermined with respect to the first camera frame of reference. 
     
     
         12 . A method of playing pool, comprising the steps of;
 a) acquiring an image of a pool table using a camera mounted on an end-effector of a robot so that its image plane is substantially perpendicular to a longitudinal axis of the pool cue, the position of said camera being fixed with respect to the longitudinal axis of the pool cue, transmitting the image to a computer processing means for analysis, the result of which is a determination of the position and identity of each ball on the pool table within this image within a frame of reference of the camera, and correcting for any errors based upon the relative position of the cue axis and the cue ball;   b) said computer processing means planning a shot of a cue ball by the pool cue by calculating a desired position of the cue with respect to the cue ball; and   c) said computer processing means instructing the robot connected to said pool cue to place a shot.   
     
     
         13 - 20 . (canceled) 
     
     
         21 . A method of playing pool, comprising the steps of:
 a) acquiring a first image of a pool table using a first camera placed above the pool table and positioned that its image plane is substantially parallel to both a playing surface of the pool table and a longitudinal axis of a pool cue and transmitting this image to a computer processing means for analysis, the result of which is a determination of the position and identity of each pool ball on the pool table;   b) said computer processing means planning a shot of a cue ball by the pool cue by calculating a desired position of the pool cue with respect to the cue ball in a pool table frame of reference including a tip position of said pool cue and orientation of the longitudinal axis of the pool cue;   c) said computer processing means instructing a robot connected to said pool cue to position the pool cue to a preferred location from which to place a shot;   d) acquiring a second image of a pool table using a second camera mounted on an end-effector of the robot so that its image plane is substantially perpendicular to both the pool table and said longitudinal axis of the pool cue, the position of said second camera being fixed with respect to said longitudinal axis of the pool cue, transmitting the second image to the computer processing means and determining locations of some of the pool balls within this image within a second camera frame of reference, and wherein if less than three pool balls are visible from either or both of the first and second cameras, then estimating and correcting for some but not necessarily all components of the accumulated positional error of the robot, and repositioning the pool cue by moving at least one proximal robotic joint which also repositions the second camera; and   e) said computer processing means instructing the robot to place a shot.   
     
     
         22 . A method of playing pool, comprising the steps of:
 a) acquiring a first image of a pool table using a first camera placed above the pool table and positioned that its image plane is substantially parallel to both a playing surface of the pool table and a longitudinal axis of a pool cue and transmitting this image to a computer processing means for analysis, the result of which is a determination of the position and identity of each pool ball on the pool table;   b) said computer processing means planning a shot of a cue ball by the pool cue by calculating a desired position of the pool cue with respect to the cue ball in a pool table frame of reference including a tip position of said pool cue and orientation of the longitudinal axis of the pool cue;   c) said computer processing means instructing a robot connected to said pool cue to position the pool cue to a preferred location from which to place a shot; and   d) acquiring a second image of a pool table using a second camera mounted on an end-effector of the robot so that its image plane is substantially perpendicular to both the pool table and said longitudinal axis of the pool cue, the position of said second camera being fixed with respect to said longitudinal axis of the pool cue, transmitting the second image to the computer processing means and determining locations of some landmarks within this image within said second camera frame of reference, and then estimating and correcting for an accumulated positional error of the robot and repositioning the pool cue by moving at least one proximal robotic joint which also repositions the second camera; and   e) said computer processing means instructing the robot to place a shot.