Methods of locating and tracking robotic instruments in robotic surgical systems

Abstract

In one embodiment of the invention, a method is disclosed to locate a robotic instrument in the field of view of a camera. The method includes capturing sequential images in a field of view of a camera. The sequential images are correlated between successive views. The method further includes receiving a kinematic datum to provide an approximate location of the robotic instrument and then analyzing the sequential images in response to the approximate location of the robotic instrument. An additional method for robotic systems is disclosed. Further disclosed is a method for indicating tool entrance into the field of view of a camera.

1 - 25 . (canceled) 
     
     
         26 . A robotic system comprising:
 a first robotic tool having a first tool tip;   a second robotic tool having a second tool tip; and   
       a processor programmed to: fuse kinematics derived position information and image derived position information of the first robotic tool in a state-space model by using a tool tracking system to estimate a first tool tip location while the first tool tip is touching tissue at a first point; fuse kinematics derived position information and image derived position information of the second robotic tool in a state-space model by using the tool tracking system to estimate a second tool tip location while the second tool tip is touching the tissue at a second point; and compare the first tool tip location and the second tool tip location by using the tool tracking system to determine a distance between the first and second points on the tissue. 
     
     
         27 . The robotic system of  claim 26 , wherein
 the tissue comprises a tumor;   the first point is on a first external surface of the tumor;   the second point is on a second external surface of the tumor; and   the comparing determines a diameter of the tumor.   
     
     
         28 . The robotic system of  claim 26 , wherein
 the tissue comprises an organ;   the first point is on a first external surface of the organ;   the second point is on a second external surface of the organ; and   the comparing determines a diameter of the organ.   
     
     
         29 . The robotic system of  claim 26 , wherein the processor is programmed to:
 process sensor data of the first robotic tool by using the tool tracking system to generate the kinematics derived position information of the first robotic tool; and   process sensor data of the second robotic tool by using the tool tracking system to generate the kinematics derived position information of the second robotic tool.   
     
     
         30 . The robotic system of  claim 26 , further comprising:
 an image capture device for capturing a sequence of images of the first robotic tool and the second robotic tool at a surgical site;   wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using the tool tracking system to generate the image derived position information of the first and second robotic tools.   
     
     
         31 . The robotic system of  claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using an image-by-synthesis approach wherein synthesized images of computer models of the first and second robotic tools are compared against images of the first and second robotic tools in the sequence of images. 
     
     
         32 . The robotic system of  claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using a feature matching approach wherein features of the first and second robotic tools are identified in the sequence of images. 
     
     
         33 . The robotic system of  claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using a sequence matching approach wherein objects or features in a sequence of images captured from a view of the image capturing device are matched against objects or features in a sequence of images captured from a different view of the image capturing device. 
     
     
         34 . The robotic system of  claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using a sequence matching approach wherein objects or features in a sequence of images captured from a view of the image capturing device are matched against objects or features in a sequence of synthesized images. 
     
     
         35 . The robotic system of  claim 26 , wherein the processor is programmed to fuse the kinematics derived position information and the image derived position information of the first robotic tool in the state-space model by using a sequential Bayesian approach; and fuse the kinematics derived position information and the image derived position information of the second robotic tool in the state-space model by using the sequential Bayesian approach. 
     
     
         36 . A method for tracking movement of a robotic instrument, the method comprising:
 receiving images of video frames from at least one camera;   receiving kinematics information related to robotic movement of the robotic instrument;   determining mechanical pose information from the kinematics information;   synthesizing model pose information of a computer aided design model of the robotic instrument using the mechanical pose information;   determining video pose information of the robotic instrument by using the synthesized model pose information as a pattern for pattern searching within the images; and   providing a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument, the state-space model to receive raw kinematics information of mechanical pose information and to adaptively fuse the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument.   
     
     
         37 . The method of  claim 36 , wherein the synthesized model pose information of the model of the robotic instrument includes one or more markers of the robotic instrument forming a pattern. 
     
     
         38 . The method of  claim 37 , wherein the one or more markers include artificial markers consisting of a pattern of dots. 
     
     
         39 . The method of  claim 37 , wherein the one or more markers include natural markers represented by consisting of geometry information of the computer aided design model. 
     
     
         40 . A method for tracking movement of a robotic instrument, the method comprising:
 receiving images of video frames from at least one camera;   determining video pose information of the robotic instrument within the images;   estimating uncertainty in the determined video pose information in light of video information in response to view geometry statistics;   receiving kinematics information related to robotic movement of the robotic instrument;   determining mechanical pose information from the kinematics information; and   fusing the mechanical pose information and the video pose information in a state-space model using a covariance matrix configured to compensate for the estimated uncertainty in the video pose information to generate estimated pose information for the robotic instrument.   
     
     
         41 . The method of  claim 40 , wherein the state-space model includes a dynamic model and an observation model respectively including dynamic noise and observation noise that respectively have dynamic and observation Gaussian distributions respectively characterized by dynamic and observation covariance matrices, wherein the observation covariance matrix includes a sub-matrix for vision; and wherein the method further comprises:
 configuring the sub-matrix for vision to compensate for the estimated uncertainty of video information by modifying elements of the sub-matrix so as to adjust a standard deviation for the estimated pose information for the robotic instrument accordingly.   
     
     
         42 . The method of  claim 40 , wherein assuming an independent Gaussian noise model, the view geometry statistics are computed for one or more of digitization error/image resolution, feature/algorithm related image matching error, distance from object to camera, angle between object surface normal and line of sight, illumination, and specularity. 
     
     
         43 . A robotic system comprising:
 a camera;   a robotic instrument; and   one or more processors programmed so as to cooperatively perform the following tasks: receive images of video frames from the camera; receive kinematics information related to robotic movement of the robotic instrument; determine mechanical pose information from the kinematics information; synthesize model pose information of a computer aided design model of the robotic instrument using the mechanical pose information; determine video pose information of the robotic instrument by using the synthesized model pose information as a pattern for pattern searching within the images; and provide a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument, the state-space model to receive raw kinematics information of mechanical pose information and to adaptively fuse the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument.   
     
     
         44 . The robotic system of  claim 43 , wherein the synthesized model pose information of the model of the robotic instrument includes one or more markers of the robotic instrument forming a pattern. 
     
     
         45 . The robotic system of  claim 44 , wherein the one or more markers include artificial markers consisting of a pattern of dots. 
     
     
         46 . The robotic system of  claim 44 , wherein the one or more markers include natural markers represented by consisting of geometry information of the computer aided design model. 
     
     
         47 . A robotic system comprising:
 a camera;   a robotic instrument; and   one or more processors programmed so as to cooperatively perform the following tasks: receive images of video frames from the camera; determine video pose information of the robotic instrument within the images; estimate uncertainty in the determined video pose information in light of video information in response to view geometry statistics; receive kinematics information related to robotic movement of the robotic instrument; determine mechanical pose information from the kinematics information; and fuse the mechanical pose information and the video pose information in a state-space model using a covariance matrix configured to compensate for the estimated uncertainty in the video pose information to generate estimated pose information for the robotic instrument.   
     
     
         48 . The robotic system of  claim 47 , wherein the state-space model includes a dynamic model and an observation model respectively including dynamic noise and observation noise that respectively have dynamic and observation Gaussian distributions respectively characterized by dynamic and observation covariance matrices, wherein the observation covariance matrix includes a sub-matrix for vision; and wherein one of the one or more processors is programmed to: configure the sub-matrix for vision to compensate for the estimated uncertainty of video information by modifying elements of the sub-matrix so as to adjust a standard deviation for the estimated pose information for the robotic instrument accordingly. 
     
     
         49 . The robotic system of  claim 47 , wherein assuming an independent Gaussian noise model, and the view geometry statistics are computed for one or more of digitization error/image resolution, feature/algorithm related image matching error, distance from object to camera, angle between object surface normal and line of sight, illumination, and specularity.