Laparoscopic ultrasound robotic surgical system

Abstract

A LUS robotic surgical system is trainable by a surgeon to automatically move a LUS probe in a desired fashion upon command so that the surgeon does not have to do so manually during a minimally invasive surgical procedure. A sequence of 2D ultrasound image slices captured by the LUS probe according to stored instructions are processable into a 3D ultrasound computer model of an anatomic structure, which may be displayed as a 3D or 2D overlay to a camera view or in a PIP as selected by the surgeon or programmed to assist the surgeon in inspecting an anatomic structure for abnormalities. Virtual fixtures are definable so as to assist the surgeon in accurately guiding a tool to a target on the displayed ultrasound image.

1 . A laparoscopic ultrasound robotic surgical system comprising: 
 a first robotic arm mechanically coupled to an ultrasound probe;    a second robotic arm mechanically coupled to a surgery related device;    a master manipulator;    a control switch having user selectable first and second modes; and    a processor configured to cause the second robotic arm to be locked in position and the first robotic arm to move the ultrasound probe according to user manipulation of the master manipulator when the control switch is in the first mode, and cause the second robotic arm to manipulate the tool according to manipulation of the master manipulator and the first robotic arm to move the ultrasound probe according to stored instructions upon detection of a user command associated with the stored instructions when the control switch is in the second mode.    
     
     
         2 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the surgery related device is a surgical tool.  
     
     
         3 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the user command is a voice command.  
     
     
         4 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the user command derives from user selection of an option provided in a graphical user interface.  
     
     
         5 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the user command derives from a selection indicated by a switch position.  
     
     
         6 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the stored instructions instruct the processor to cause the first robotic arm to move the ultrasound probe so that 2D ultrasound image slices captured by the ultrasound probe while moving provide information for the processor to generate a 3D computer model of at least part of an anatomic structure during a minimally invasive surgical procedure.  
     
     
         7 . The laparoscopic ultrasound robotic surgical system according to  claim 6 , wherein the stored instructions instruct the processor to cause the first robotic arm to move the ultrasound probe along a trajectory of stored points.  
     
     
         8 . The laparoscopic ultrasound robotic surgical system according to  claim 7 , wherein the stored instructions instruct the processor to cause the first robotic arm to repetitively move the ultrasound probe back and forth along the trajectory of stored points.  
     
     
         9 . The laparoscopic ultrasound robotic surgical system according to  claim 7 , wherein the trajectory of stored points include points on a surface of the anatomic structure.  
     
     
         10 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the stored instructions instruct the processor to cause the first robotic arm to move the ultrasound probe so as to be positioned and oriented at a stored position and orientation.  
     
     
         11 . The laparoscopic ultrasound robotic surgical system according to  claim 10 , wherein the stored position and orientation correspond to a clickable thumbnail of an ultrasound image generated by the ultrasound probe at the position and orientation.  
     
     
         12 . The laparoscopic ultrasound robotic surgical system according to  claim 11 , wherein the processor is configured to cause the first robotic arm to move the ultrasound probe to the stored position and orientation upon receiving an indication that a user has clicked the clickable thumbnail.  
     
     
         13 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the surgery related device is a surgical tool, and further comprising: 
 an endoscope for capturing images of a surgical site in a patient;    wherein the processor is configured to display the captured images in a picture section of a display screen and an ultrasound image captured by the ultrasound probe within a picture-in-picture section of the display screen.    
     
     
         14 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the surgery related device is a surgical tool, and further comprising: 
 an endoscope for capturing video images;    wherein the processor is configured to spatially register ultrasound images captured by the ultrasound probe with the video images captured by the endoscope.    
     
     
         15 . The laparoscopic ultrasound robotic surgical system according to  claim 14 , wherein the processor is configured to determine positions and orientations of the ultrasound images relative to a frame of reference associated with the endoscope, and display the ultrasound images so as to overlay the video images at the determined positions and orientations on a display screen.  
     
     
         16 . The laparoscopic ultrasound robotic surgical system according to  claim 15 , wherein the endoscope is a stereoscopic endoscope capturing right and left 2D camera views including captured images of the ultrasound probe, the display screen is a 3D display screen, and the processor is configured to: generate a 3D camera image from the captured right and left 2D camera views, and display the 3D camera images in the 3D display screen with 2D ultrasound image slices captured by the ultrasound probe overlayed in the 3D camera images so as to appear as emanating in proper perspective from the ultrasound probe.  
     
     
         17 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the control switch is placed in the first or second mode by a user voice command.  
     
     
         18 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the control switch is placed in the first or second mode by user selection of an option provided in a graphical user interface.  
     
     
         19 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the control switch is placed in the first or second mode by a user selected switch position.  
     
     
         20 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the ultrasound probe includes a force sensor to sense forces exerted against the ultrasound probe and provides information of the sensed forces to the processor, and the processor reflects information of the sensed forces back to the master manipulator so as to be felt by a user while manipulating the master manipulator.  
     
     
         21 . The laparoscopic ultrasound robotic surgical system according to  claim 1 , wherein the ultrasound probe comprises: 
 an ultrasound sensor;    an elongated shaft; and    a wrist mechanism coupling the ultrasound sensor to the elongated shaft so as to allow pitch and yaw movement of the ultrasound sensor relative to an axis running along a length of the elongated shaft.    
     
     
         22 . The laparoscopic ultrasound robotic surgical system according to  claim 21 , wherein the ultrasound probe further comprises: 
 a first pair of cables coupled to the ultrasound sensor such that the ultrasound sensor moves in a yaw direction by pulling on only one of the first pair of cables and moves in an opposite yaw direction when pulling on only the other of the first pair of cables.    
     
     
         23 . The laparoscopic ultrasound robotic surgical system according to  claim 22 , wherein the ultrasound probe further comprises: 
 a second pair of cables coupled to the ultrasound sensor such that the ultrasound sensor moves in a pitch direction by pulling on only one of the second pair of cables and moves in an opposite pitch direction when pulling on only the other of the second pair of cables.    
     
     
         24 . A method for providing robotic assisted laparoscopic ultrasound, comprising: 
 storing a current ultrasound probe position and orientation upon detection of a start of training indication; and    periodically storing ultrasound probe positions and orientations to define a trajectory of positions and orientations until detection of an end of training indication.    
     
     
         25 . The method according to  claim 24 , wherein the start and end of training indications are generated by user voice commands.  
     
     
         26 . The method according to  claim 24 , wherein the start and end of training indications are generated by user selections using a graphical user interface.  
     
     
         27 . The method according to  claim 24 , wherein the start and end of training indications are generated by positions of one or more switch mechanisms.  
     
     
         28 . The method according to  claim 24 , further comprising: associating the trajectory of positions and orientations with a first voice command.  
     
     
         29 . The method according to  claim 28 , further comprising: causing the ultrasound probe to move along the trajectory of positions and orientations upon detection of the first voice command.  
     
     
         30 . The method according to  claim 28 , further comprising: causing the ultrasound probe to move back and forth along the trajectory of positions and orientations upon detection of the first voice command and until detection of a second voice command.  
     
     
         31 . The method according to  claim 24 , further comprising: associating the trajectory of positions and orientations with a clickable thumbnail displayed on a display screen.  
     
     
         32 . The method according to  claim 31 , further comprising: causing the ultrasound probe to move along the trajectory of positions and orientations upon detection of the clickable thumbnail having been clicked upon and until a stop indication is received.  
     
     
         33 . The method according to  claim 24 , further comprising: associating the trajectory of positions and orientations with a first switch position.  
     
     
         34 . The method according to  claim 33 , further comprising: causing the ultrasound probe to move along the trajectory of positions and orientations upon detection of a switch being in the first switch position and until detection of the switch being in a second switch position.  
     
     
         35 . A method for providing robotic assisted laparoscopic ultrasound, comprising: 
 capturing an ultrasound image using an ultrasound probe disposed at a position and orientation;    storing information of the position and orientation;    generating a clickable thumbnail of the ultrasound image;    associating the stored position and orientation with the clickable thumbnail; and    displaying the clickable thumbnail on a display screen.    
     
     
         36 . The method according to  claim 35 , further comprising: 
 receiving information of a user having clicked on the clickable thumbnail; and    causing the ultrasound probe to be disposed at the position and orientation associated with the clicked upon clickable thumbnail.    
     
     
         37 . A method for providing robotic assisted laparoscopic ultrasound, comprising: 
 displaying an ultrasound view of an anatomic structure in a patient as a registered overlay to a camera view of the anatomic structure;    receiving information of a target marked on the ultrasound view;    determining a path for a tool to travel to the target within the patient; and    generating a virtual fixture to assist in electronically constraining the tool to travel over the determined path.    
     
     
         38 . The method according to  claim 37 , wherein the anatomic structure is cancerous, the target is a lesion marked by a surgeon on the displayed ultrasound view, and the tool is a needle.  
     
     
         39 . The method according to  claim 37 , further comprising: 
 determining a distance of the tool from the target; and    changing a color of the target marked on the ultrasound view so as to indicate the distance.    
     
     
         40 . The method according to  claim 37 , further comprising: 
 determining when a tip of the tool reaches a threshold distance from a surface of the anatomic structure; and    displaying a right angle view of the ultrasound image and the tool upon determining that the tip of the tool has reached the threshold distance in a picture-in-picture window.    
     
     
         41 . The method according to  claim 40 , wherein the ultrasound image is a 3D image generated from a sequence of 2D ultrasound image slices of the anatomic structure.