Auxiliary image display and manipulation on a computer display in a medical robotic system

Abstract

To assist a surgeon performing a medical procedure, auxiliary images generally indicating internal details of an anatomic structure being treated are displayed and manipulated by the surgeon on a computer display screen to supplement primary images generally of an external view of the anatomic structure. A master input device controlling a robotic arm in a first mode may be switched by the surgeon to a second mode in order to function instead as a mouse-like pointing device to facilitate the surgeon performing such auxiliary information display and manipulation.

What is claimed is: 
     
         1 . A method for displaying on a computer display screen an effect of a therapeutic procedure being applied by a therapy instrument to an anatomic structure, comprising:
 generating an auxiliary image indicating the effect of the therapeutic procedure being applied by the therapy instrument to the anatomic structure; and   displaying a primary image of the anatomic structure overlaid with the auxiliary image on the computer display screen during the therapeutic procedure.   
     
     
         2 . The method according to  claim 1 , wherein the therapeutic procedure is performed using a medical robotic system, and the therapy instrument is robotically manipulatable by a surgeon using the medical robotic system to perform the therapeutic procedure. 
     
     
         3 . The method according to  claim 1 , wherein the primary image is captured prior to the therapeutic procedure. 
     
     
         4 . The method according to  claim 3 , wherein the primary image is a pre-operative image generated by ultrasound. 
     
     
         5 . The method according to  claim 3 , wherein the primary image is a pre-operative image generated by magnetic resonance imaging. 
     
     
         6 . The method according to  claim 3 , wherein the primary image is a pre-operative image generated by computed axial tomography. 
     
     
         7 . The method according to  claim 3 , wherein the auxiliary image is a computer model of the therapeutic effect being applied by the therapy instrument during the therapeutic procedure. 
     
     
         8 . The method according to  claim 7 , wherein the computer model is a volumetric shape determined at least partially by the geometry of a therapeutic end of the therapy instrument. 
     
     
         9 . The method according to  claim 7 , wherein the computer model is a volumetric shape determined at least partially by a heat level being applied to the anatomic structure by a therapeutic end of the therapy instrument. 
     
     
         10 . The method according to  claim 7 , wherein the computer model is a volumetric shape determined at least partially by features of surrounding tissue of the anatomic structure being subjected to the therapeutic procedure. 
     
     
         11 . The method according to  claim 1 , wherein the primary image is captured during the therapeutic procedure. 
     
     
         12 . The method according to  claim 11 , wherein the primary image is an intra-operative image captured by a camera unit. 
     
     
         13 . The method according to  claim 12 , wherein the camera unit includes a stereoscopic pair of cameras. 
     
     
         14 . The method according to  claim 12 , wherein the camera unit is included in an endoscope. 
     
     
         15 . The method according to  claim 14 , wherein the endoscope is a laparoscope. 
     
     
         16 . The method according to  claim 11 , wherein the auxiliary image is a computer model of the therapeutic effect being applied by the therapy instrument during the therapeutic procedure. 
     
     
         17 . The method according to  claim 16 , wherein the computer model is a volumetric shape determined at least partially by a shape of a therapeutic end of the therapy instrument. 
     
     
         18 . The method according to  claim 16 , wherein the computer model is a volumetric shape determined at least partially by a heat level being applied to the anatomic structure by a therapeutic end of the therapy instrument. 
     
     
         19 . The method according to  claim 16 , wherein the computer model is a volumetric shape determined at least partially by features of surrounding tissue of the anatomic structure being subjected to the therapeutic procedure. 
     
     
         20 . The method according to  claim 11 , wherein the auxiliary image is an intra-operative image generated by ultrasound. 
     
     
         21 . The method according to  claim 22 , wherein the therapeutic procedure destroys abnormal tissue of the anatomic structure using radio frequency ablation. 
     
     
         22 . The method according to  claim 21 , wherein the abnormal tissue includes diseased tissue. 
     
     
         23 . The method according to  claim 22 , wherein the diseased tissue includes at least one tumor. 
     
     
         24 . The method according to  claim 21 , wherein the abnormal tissue includes damaged tissue. 
     
     
         25 . The method according to  claim 21 , wherein the therapeutic procedure is one of a group consisting of radio frequency ablation, high intensity focused ultrasound, and cauterization. 
     
     
         26 . A method for displaying a selected portion of an auxiliary image of an anatomic structure as an overlay to a primary image of the anatomic structure on a computer display screen, comprising:
 associating a movable window with a pointing device such that the movable window is positionable on the computer display screen using the pointing device;   registering an auxiliary image of an anatomic structure with a primary image of the anatomic structure so as to be at a same position and orientation in a common reference frame; and   displaying the primary image on the computer display screen, and a portion of the registered auxiliary image corresponding to the same screen coordinates as the movable window as an overlay to the primary image in the movable window.   
     
     
         27 . The method according to  claim 26 , wherein the primary image is captured by an image capturing device during a minimally invasive surgical procedure being performed using a medical robotic system, and the image capturing device is robotically manipulatable using the medical robotic system while performing the medical procedure. 
     
     
         28 . The method according to  claim 26 , wherein the movable window appears as a circular lens on the display screen. 
     
     
         29 . The method according to  claim 26 , wherein the movable window appears as a rectangular lens on the display screen. 
     
     
         30 . The method according to  claim 26 , wherein the primary image is a three-dimensional image of the anatomic structure, and the computer display screen is a three-dimensional computer display screen. 
     
     
         31 . The method according to  claim 26 , wherein an entire part of the registered auxiliary image corresponding to the same screen coordinates as the movable window is displayed as an overlay to the primary image in the movable window. 
     
     
         32 . The method according to  claim 26 , wherein the portion of the registered auxiliary image corresponding to the same screen coordinates as the movable window is expanded so as to fit and be displayed as an overlay to the primary image in the movable window so as to appear as a magnified view of the auxiliary image. 
     
     
         33 . The method according to  claim 32 , further comprising:
 receiving a magnification factor selected by a user viewing the computer display screen; and   applying the magnification factor to determine the portion of the registered auxiliary image to be fitted and displayed as an overlay to the primary image in the movable window.   
     
     
         34 . The method according to  claim 26 , wherein the primary and auxiliary images are three-dimensional images of the anatomic structure, and the computer display screen is a three-dimensional computer display screen. 
     
     
         35 . The method according to  claim 26 , wherein the movable window is associated with a user selectable depth of the auxiliary image so that a two-dimensional slice of the auxiliary image corresponding to a depth selected by a user is displayed as an overlay to the primary image in the movable window. 
     
     
         36 . The method according to  claim 26 , wherein the primary image is a pre-operative image generated by magnetic resonance imaging. 
     
     
         37 . The method according to  claim 26 , wherein the primary image is a pre-operative image generated by computed axial tomography. 
     
     
         38 . The method according to  claim 26 , wherein the primary image is an inter-operative image captured by a camera unit. 
     
     
         39 . The method according to  claim 38 , wherein the camera unit is included in an endoscope. 
     
     
         40 . The method according to  claim 38 , wherein the auxiliary image is a pre-operative captured image. 
     
     
         41 . The method according to  claim 40 , wherein the pre-operative captured image is generated by magnetic resonance imaging. 
     
     
         42 . The method according to  claim 40 , wherein the pre-operative captured image is generated by computed axial tomography. 
     
     
         43 . The method according to  claim 40 , wherein the pre-operative captured image is generated by ultrasound. 
     
     
         44 . The method according to  claim 38 , wherein the auxiliary image is an intra-operative captured image. 
     
     
         45 . The method according to  claim 44 , wherein the intra-operative captured image is generated by ultrasound. 
     
     
         46 . The method according to  claim 44 , wherein the intra-operative captured image is generated by a second camera unit. 
     
     
         47 . A medical robotic system comprising:
 an image capturing device for capturing images;   a robotic arm holding the image capturing device;   a computer display screen;   a master input device adapted to be manipulatable by an user in multiple degrees-of-freedom movement; and   a processor configured to control movement of the image capturing device according to user manipulation of the master input device when the master input device is in an image capturing mode, and controlling the displaying of images derived from the captured images on the computer display screen according to user manipulation of the master input device when the master input device is in an image manipulating mode.   
     
     
         48 . The medical robotic system according to  claim 47 , wherein the master input device is configured so as to be manipulatable in six degrees of freedom so that the master input device operates as a three-dimensional mouse when in the image manipulating mode. 
     
     
         49 . The medical robotic system according to  claim 47 , wherein the processor is configured so as to perform a grabbing function on one of the derived images being displayed on the computer display screen when a user activates a control input while a cursor associated with the master input device is being displayed on the derived image, and perform a moving function on the derived image when the user moves the master input device while keeping the control input activated when in the image manipulating mode. 
     
     
         50 . The medical robotic system according to  claim 49 , wherein the processor is further configured to provide haptic feedback to the master input device while performing the moving function on the derived image. 
     
     
         51 . The medical robotic system according to  claim 50 , wherein the haptic feedback is provided by associating a virtual mass and inertial properties to the derived image so that the user would feel a reflected force when the processor is performing the grabbing and moving functions on the derived image in response to user manipulation of the master input device while in the image manipulating mode. 
     
     
         52 . The medical robotic system according to  claim 49 , wherein the image capturing device captures auxiliary images and the processor is configured to cause a primary image captured by a primary image capturing device to be displayed on the computer display screen with at least a portion of one of the derived images overlayed over the primary image. 
     
     
         53 . The medical robotic system according to  claim 52 , wherein the processor is configured to facilitate manually registering the one of the derived images with the primary image by a user performing the grabbing and moving functions on the derived image so as to register the derived image with the primary image as they are both being displayed on the computer display screen when in the image manipulating mode. 
     
     
         54 . The medical robotic system according to  claim 47 , wherein the master input device has a gripper adapted to be squeezed by a hand of a user to function as a control input when the master input device is in the image manipulating mode. 
     
     
         55 . The medical robotic system according to  claim 54 , wherein the processor is configured to adjust a parameter associated with the derived images when the gripper is squeezed and rotated around an axis of the gripper when the master input device is in the image manipulating mode. 
     
     
         56 . The medical robotic system according to  claim 55 , wherein the adjustable parameter is a brightness of the derived image. 
     
     
         57 . The medical robotic system according to  claim 55 , wherein the adjustable parameter is a color of the derived image. 
     
     
         58 . The medical robotic system according to  claim 55 , wherein the adjustable parameter is a level of detail of the derived image. 
     
     
         59 . The medical robotic system according to  claim 58 , wherein the level of detail of the derived image is determined by a level of coarseness of a mesh structure defining the derived image. 
     
     
         60 . The medical robotic system according to  claim 47 , wherein the derived images are three-dimensional volumes generated from the captured images; and the processor is further configured to display one of the three-dimensional volumes and a two-dimensional window on the computer display screen, manipulate a position and orientation of the window on the computer display screen in response to user manipulation of the master input device, and define a cut-plane by an intersection of the window with the three-dimensional volume so as to indicate a two-dimensional slice of the three-dimensional volume. 
     
     
         61 . The medical robotic system according to  claim 60 , wherein the two-dimensional slice is displayed in the window. 
     
     
         62 . The medical robotic system according to  claim 60 , wherein the two-dimensional slice is displayed in a picture-in-picture window of the computer display screen. 
     
     
         63 . The medical robotic system according to  claim 60 , wherein the processor is further configured to display a user selectable number of two-dimensional windows on the computer display screen, individually manipulate positions and orientations of the windows on the computer display screen in response to user manipulation of the master input device, and define cut-planes by intersections of the manipulated windows with the three-dimensional volume so as to indicate corresponding two-dimensional slices of the three-dimensional volume. 
     
     
         64 . The medical robotic system according to  claim 63 , wherein the two-dimensional slices are displayed in corresponding picture-in-picture windows of the computer display screen. 
     
     
         65 . The medical robotic system according to  claim 60 , wherein the processor is configured to display the two-dimensional window on the computer display screen in response to user selection of an item included in a displayed menu on the computer display screen. 
     
     
         66 . The medical robotic system according to  claim 60 , wherein the processor is configured to display the two-dimensional window on the computer display screen in response to user selection of an icon being displayed on the display screen. 
     
     
         67 . The medical robotic system according to  claim 66 , wherein the icon is displayed in a periphery area of the computer display screen, and the processor is further configured to interpret user mouse-type actions of clicking on the icon and dragging the icon away from the periphery area as a user selection of the icon. 
     
     
         68 . The medical robotic system according to  claim 67 , wherein the image capturing device is an ultrasound probe and the derived images are three-dimensional ultrasound images of an anatomic structure that are computer generated from two-dimensional ultrasound slices captured by the ultrasound probe. 
     
     
         69 . The medical robotic system according to  claim 47 , wherein the processor is further configured to display one of the derived images and an eraser image on the computer display screen, manipulate at least a position of the eraser image on the computer display screen in response to user manipulation of the master input device, and erase any portion of one of the derived image being displayed on the computer display screen that is traversed by the eraser image as the eraser image is being manipulated on the computer display screen. 
     
     
         70 . The medical robotic system according to  claim 69 , wherein the processor is configured to erase all detail of the portion of the derived image that is traversed by the eraser image. 
     
     
         71 . The medical robotic system according to  claim 69 , wherein the processor is configured to reduce the detail of the portion of the derived image that is traversed by the eraser image. 
     
     
         72 . The medical robotic system according to  claim 71 , wherein the reduction of detail of the portion of the derived image that is traversed by the eraser image entails reducing the fineness of a mesh structure defining the derived image.