USRE49930EActiveUtility
Methods and systems for computer-aided surgery using intra-operative video acquired by a free moving camera
Est. expiryMar 26, 2035(~8.7 yrs left)· nominal 20-yr term from priority
Inventors:João Pedro De Almeida Barreto
A61B 34/20A61B 34/10A61B 34/25A61B 34/30A61B 90/361G06F 3/0321G06T 7/50G06T 7/62G06T 7/73G06T 7/85G06T 19/006G16H 20/30G16H 20/40G16H 30/20G16H 30/40G16H 50/50H04N 13/221H04N 13/246A61B 2034/2055A61B 2034/2057A61B 2034/2065A61B 2034/301A61B 2034/302A61B 2090/365A61B 2090/3937A61B 2090/3945A61B 2090/3983A61B 2090/3991G06T 2207/30208G06T 2207/30244G06T 2219/004
65
PatentIndex Score
1
Cited by
166
References
102
Claims
Abstract
Disclosed is a concept for computer-assisted procedures of surgery and diagnosis that target rigid, non-deformable anatomical parts such as bone, tissue, or teeth. The disclosure describes attaching small visual markers to instruments and anatomy of interest (e.g. bone surface), with each marker having a printed known pattern for detection and unique identification in images acquired by a free-moving camera, and a geometry that enables estimating its rotation and translation with respect to the camera using solely image processing techniques.
Claims
exact text as granted — not AI-modifiedThe invention claimed is:
1. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT T is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
(viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix). assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates with the 3D reconstruction results and/or overlaid surgical plan;
where the camera in (i) is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f −1 (u; k,ξ)
with f −1 being an inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters.
2. The system of claim 1 wherein the camera is pre-calibrated from a factory or is calibrated in an Operating Room (OR) by acquiring a single image of a known calibration grid or checkerboard pattern that allows estimating intrinsic parameters k, a distortion ξ and a rigid transformation Ĝ that relates coordinates of the calibration grid and coordinates of the camera.
3. The system of claim 2 wherein the instrinsic parameters k and the distortion ξ are automatically updated during operation to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, and variations in zoom.
4. The system of claim 1 wherein the WM in (ii) comprises, depending on a targeted clinical application and a chosen method (vi) of attachment, one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle like object with a flat lateral facet for trans-dermic insertion into the joint or cavity, or a flat button-like object that is pulled inside a joint or cavity by a thread or guide.
5. The system of claim 1 wherein each of the (ii) World Marker, (iii) touch probe, and (iv) surgical instruments are manufactured in metal or plastic, and wherein the Tool Markers can be assembled, secured, printed, or engraved.
6. The system of claim 1 wherein the planar pattern of the visual marker in (ii), (iii), and (iv) comprises any pattern that can be detected and uniquely identified using image processing techniques, and that has recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation r and translation t that map points p in the coordinate system of the pattern into points x in the metric coordinate system of the camera.
7. The system of claim 6 , wherein an initial 3D pose estimation r, t is further refined by determining increments in rotation ξ R and translation ξ t that minimize the following photo-geometric error
ϵ
i
=
∑
u
∈
N
i
[
I
(
w
(
u
;
r
0
+
δ
R
,
t
0
+
δ
t
)
)
-
T
(
u
)
]
2
where T(u) is the pattern template, I(u) is a current frame, N i is a pattern region, and w is an image warping function given by the function composition
w(u; r, t)=f(x; k, ξ)o h(x; r, t)o f −1 (u; k′, ξ′)
with f being the projection function and h denoting a homography map that depends on a relative 3D pose r, t.
8. The system of claim 7 wherein the minimization of the photo-geometric error ε I is performed using direct composition, inverse composition, or efficient second order minimization, and wherein formulation of the photo-geometric error takes into account variable illumination conditions.
9. The system of claim 1 wherein a calibration of the touch-probe of (iii) for finding the coordinates P T of its tip is performed in an operating room (OR) in parallel with a camera calibration, in which case the tool tip is placed in a point in a calibration grid with known coordinates P G and a single calibration image shows both the calibration grid and the TM of the touch-probe such that P T can be determined by
(
P
T
1
)
=
T
^
-
1
G
^
(
P
G
1
)
with {circumflex over (T)} −1 denoting an inverse of a rigid transformation {circumflex over (T)} that encodes a 3D pose r, t of the TM in camera coordinates that is determined from image information.
10. The system of claim 1 wherein an arbitrary point P is reconstructed in global or world coordinates by acquiring an image as described in (vii) that undergoes the following processing steps:
detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM in camera coordinates;
reconstruct a point P that is in contact with the tip of the touch-probe by making
(
P
1
)
=
C
-
1
T
^
(
P
T
1
)
;
and
store 3D coordinates of point P in memory.
11. The system of claim 10 , wherein the system is further configured to execute steps comprising:
reconstructing a contour or a sparse mesh of a surface region, in which case the touch-probe is respectively used to outline the contour or randomly grasp the surface region, while the camera acquires a continuous sequence of images as described in (vii), and the processing steps are executed for each frame of the sequence.
12. The system of claim 11 wherein the 3D reconstruction results are used for measuring distances, areas, or volumes, inferring the shape of curves, regions, or surfaces, or overlying a surgical plan with the patient's anatomy in which case a suitable 3D registration method may be employed.
13. The system of claim 12 wherein 3D data is stored in memory, the 3D data including the reconstruction results of (vii) and the measurements, inferences and surgical plan of (vii), wherein the 3D data is overlaid in the video whenever the WM is in the FOV of the camera, in which case each image undergoes the following processing steps in real-time:
detect, identify, and estimate the 3D pose C of the WM in camera coordinates;
map the 3D data from world coordinates into camera coordinates using C; and
project the 3D data into the image using the camera calibration and projection function f (augmented reality).
14. The system of claim 1 wherein the system is configured to, for each of the one or more surgical instruments of (iv), calibrate the surgical instrument by determining in TM coordinates the position of a point, axis, or CAD model of the surgical instrument that can be either performed in factory at manufacturing time, or in the operating room (OR) before starting the clinical procedure, in which case the camera of (i) and the touch-probe of (iii) are used to reconstruct 3D points in the surgical instrument where the role of the WM is replaced by the TM of the instrument for the reconstruction results to be expressed in a local reference frame of the instrument.
15. The system of claim 1 wherein the assisted execution of the clinical procedure is accomplished by acquiring continuous video according to (ix), the video comprising a plurality of images, with each consecutive image undergoing the following processing steps in real-time:
a. detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
b. detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM of each of the one or more surgical instruments in camera coordinates;
c. compute the 3D pose T of TM in WM coordinates using the following equation:
T=C −1 {circumflex over (T)}
d. map the information of surgical instrument calibration, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T;
e. relate the surgical instrument calibration information with 3D data stored in memory to make measurements and inferences for the purpose of real-time guidance; and
f. display aiding features either by using Augmented Reality (AR), in which case guidance information is overlaid in image, or by animating a Virtual Reality (VR) 3D model.
16. The system of claim 15 wherein the aiding features can take multiple forms including highlighting points, regions or structures in anatomy, measuring distance, thickness or depth, measuring angles between axes or between an axis and a plane, anticipating trajectories of insertion, penetration or cut, and delimiting regions or structures in anatomy for resection or dissection.
17. The system of claim 15 wherein multiple surgical instruments are used simultaneously, in which case steps b to e are run in parallel for each instrument whose TM is visible in the image.
18. The system of claim 15 wherein the one or more surgical instruments of (iv) comprises, or is rigidly attached to, a robot end-effector and the guidance or aided execution of (ix) includes sending commands to the robot for the purpose of controlling the motion and/or action of the one or more surgical instruments.
19. The system of claim 1 , wherein WM is a primary marker, wherein one or more secondary markers WM' are attached to the surface of the rigid anatomical part of interest as described in (vi) for the purpose of increasing a working area or region of operation, in which case all systems functionalities of 3D reconstruction and aided execution are readily available whenever one of the secondary markers WM' is visible in image and the transformation W, that maps WM′ coordinates into global or world coordinates in a reference frame of the primary marker WM, is known.
20. The system of claim 19 wherein the transformation W is determined by acquiring an image where both primary and secondary markers are visible, in which case the transformation is given by
W′=C −1 C′,
with C and C′ being respectively the 3D pose of WM and WM' that are computed from image information.
21. The system of claim 1 wherein the camera of (i) is assembled with a tool such that the camera and the tool become a single rigid body with the position of relevant points or parts of the tool being known in camera coordinates, in which case the location of these points or parts can be determined in global or world coordinates whenever the WM of (ii) is in the FOV of the camera and its 3D pose C can be estimated from image information.
22. The system of claim 21 wherein the camera is assembled with a touch-probe to create an ensemble to replace (iii) in the 3D reconstruction step of (vii), in which case the ensemble is used to pin-point an arbitrary point P while keeping the WM in the FOV of the camera to determine its world coordinates by making
(
P
1
)
=
C
-
1
(
P
C
1
)
,
with P c being the position of the probe tip in camera coordinates and C −1 a 3D pose of the camera in WM coordinates that is estimated from image information.
23. The system of claim 22 wherein a calibration of the ensemble for finding coordinates P c of the touch-probe tip in camera coordinates is performed in an operating room (OR) in parallel with a camera calibration, in which case the tool or instrument tip is placed in a point in a calibration grid with known coordinates P G , and a single calibration image shows the calibration grid such that P c can be determined by
(
P
C
1
)
=
G
^
(
P
G
1
)
with Ĝ denoting a rigid transformation that encodes a 3D pose r, t of a pattern of the calibration grid in camera coordinates t.
24. The system of claim 1 wherein the touch probe of (iii) comprises a laser pointer with a visual marker (TM) attached, for which a line L T defined by a laser beam is known in a reference frame of the visual marker (TM), and wherein the 3D reconstruction step of (vii) is performed in a contactless manner by acquiring an image where WM, TM and a point of incidence of the laser beam with the surface are simultaneously visible, and by processing this image with point P T being given by
P T =L T Λ{circumflex over (T)} −1 (B x )
with Λ denoting the operation of line intersection in 3D, B x being the back-projection line of the point of incidence that is detected in the image, and {circumflex over (T)} −1 (B x ) denoting the line Bx expressed in the local system of coordinates of the visual marker (TM).
25. The system of claim 1 wherein the touch probe of (iii) comprises a time-of-flight device that has a visual marker TM attached and that measures distances λ from an origin S T along a beam direction d T that are known in a reference frame of the visual marker TM, and wherein the 3D reconstruction is performed in a contactless manner by acquiring an image as described in (vii) that undergoes the processing steps with the point P T being given by
P T =S T +λd T .
26. The system of claim 1 wherein the visual markers of (ii), (iii) and (iv) have multiple planar facets with distinct patterns where the location of each pattern is known in a common coordinate system of the markers for the purpose of extending a range of viewing positions and orientations from which the marker can be observed.
27. The system of claim 1 wherein the visual markers of (ii), (iii) and (iv) are non-planar, in which case each marker should comprise n≥3 recognizable points with known coordinates in its local reference frame for enabling 3D pose estimation from images using a Perspective-n-Point algorithm.
28. The system of claim 1 that is used for computer assisted execution of arthroscopic procedures including anterior and/or posterior cruciate ligament reconstruction, resection of femuro-acetabular impingement, or diagnosis and repair of confocal defects in cartilage, in which case the free-moving camera is the arthroscopic camera used for visualizing the articular joint.
29. The system of claim 1 that is used for computer assisted execution of open surgical procedures in orthopedics, including total hip replacement, total knee replacement, unicompartmental knee replacement, shoulder joint replacement, and pedicle-screw placement, in which case a camera is used to observe the operating field.
30. The system of claim 1 that is used for computer assisted execution of prosthodontic procedures including placement of dental implants.
31. A method for computer-aided execution of surgery and diagnosis in anatomical regions comprising rigid, non-deformable parts or tissues, the method comprising:
(i) attaching a visual marker, referred to as the World Marker or WM, in an arbitrary location in the surface of the rigid anatomical part of interest, the visual marker comprising at least one planar facet with a known pattern whose system of coordinates defines a global or world system of coordinates;
(ii) acquiring, by a free-moving camera, intra-operative images and video of an operating field, articular joint, or anatomical cavity;
(iii) using a touch-probe, which is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip PT is known, for pin-pointing points and/or outlining curves in the surface of the rigid anatomical part, while keeping both WM and TM of the probe in the Field-of-View or FOV of the camera to be visible in the acquired images and video;
(iv) receiving, by an image processing system that may correspond to, or may be part of, a computer and/or any other computing device, the images and video acquired in (iii), which are processed for determining the location of the touch-probe and reconstructing in 3D the pin-pointed points and/or outlined curves that are stored in memory in global or world coordinates;
(v) using the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with the patient's anatomy, in which case a 3D registration method may be used for determining the transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to anatomy of interest;
(vi) performing the surgical procedure with the required instruments, where each instrument has at least one visual marker, also referred to as Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which the position of relevant points and parts in the instrument are known, while keeping both WM and TMs of the instruments in the FOV of the camera to be visible in the acquired images and video;
(vii) receiving, by the image processing system, the images and video acquired in (vi), that are processed such that instruments can be located in world coordinates and real-time guidance can be provided by relating these locations with 3D reconstructions results and/or overlaid surgical plan; and
(viii) outputting, by the image processing system, processing results and guidance information to a display, by either overlaying information in the input video using augmented reality (AR) techniques, or by animating models in a virtual reality (VR) environment;
where the camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x =f −1 (u; k, ξ)
with f =1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters.
32. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip P T is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
(viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix). assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
wherein an arbitrary point P is reconstructed in global or world coordinates by acquiring an image as described in (vii) that undergoes the following processing steps:
detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM in camera coordinates;
reconstruct a point P that is in contact with the tip of the touch-probe by making
(
P
1
)
=
C
-
1
T
^
(
P
T
1
)
;
and
store 3D coordinates of point P in a memory.
33. A system for video based computer-aided surgery and diagnosis in anatomical regions of a patient comprising rigid, non-deformable anatomical parts or tissues comprising:
(i). a free-moving camera that is meant to acquire intra-operative images and video of an operating field, articular joint or anatomical cavity;
(ii). a visual marker, referred to as a World Marker or WM, which is an object that can be rigidly attached to a surface, and that comprises at least one planar facet with a known pattern having a system of coordinates that defines a global or world system of coordinates;
(iii). a touch-probe that is a tool or instrument comprising a handgrip and a tip, and that has at least one visual marker, referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates where a position of the tip P T is known;
(iv). one or more surgical instruments for surgical execution, where each surgical instrument has at least one visual marker, also referred to as a Tool Marker or TM, comprising at least one planar facet with a known pattern that defines a local system of coordinates in which a position of relevant points and parts in the instrument are known; and
(v). an image processing system, that may correspond to or may be part of a computer and/or any other computing device, that receives as input images and video acquired by the camera (i), as well as commands that may be entered by a button panel, keyboard, camera buttons, foot switch and/or any other input interface, and that outputs processing results and guidance information to a display;
wherein the system is configured to execute steps comprising:
(vi). attachment of the WM of (ii) in an arbitrary location in a surface of a rigid anatomical part of interest, such that the WM of (ii) and the rigid anatomical part of interest do not move one with respect to the other;
(vii). 3D reconstruction of points in the surface of the rigid anatomical part of interest by using the touch-probe of (iii) to pin-point the points in the surface while keeping both the WM and the TM of the probe in a Field-of-View or FOV of the camera of (i) for determining the location of the touch-probe in the world system of coordinates to create 3D reconstruction results;
(viii). use the 3D reconstruction results to perform measurements, make inferences, or overlay a surgical plan with an anatomical region of the patient, in which case a 3D registration method may be used for determining a transformation that maps local coordinates in a surgical plan into world coordinates in the WM attached to the anatomical part of interest; and
(ix).assisted execution of a clinical procedure by using the surgical instruments of (iv) while keeping both WM and TMs of the surgical instruments in the FOV of the camera of (i) such that the surgical instruments can be located in world coordinates and real-time guidance can be provided by relating surgical instrument locations in world coordinates;
wherein the assisted execution of the clinical procedure is accomplished by acquiring continuous video according to (ix), the video comprising a plurality of images, with each consecutive image undergoing the following processing steps in real-time:
a. detect, identify, and estimate a 3D pose C of the WM in camera coordinates;
b. detect, identify, and estimate a 3D pose {circumflex over (T)} of the TM of each of the one or more surgical instruments in camera coordinates;
c. compute the 3D pose T of TM in WM coordinates using the following equation:
T =C −1 {circumflex over (T)}
d. map the information of surgical instrument calibration, that can be points, axes, or CAD models, into world coordinates using the rigid transformation T;
e. relate the surgical instrument calibration information with 3D data stored in memory to make measurements and inferences for the purpose of real-time guidance; and
f. display aiding features either by using Augmented Reality (AR), in which case guidance information is overlaid in image or by animating a Virtual Reality (VR) 3D model.
34. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a free-moving camera to acquire intra-operative images, video, or images and video of an operating field,
where the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f −1 (u; k, ξ)
with f −1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect to each other, wherein the object comprises a first known pattern that defines a first system of coordinates; one or more surgical instruments, wherein each surgical instrument has at least one tool marker (TM), each TM comprising a respective second known pattern that defines a second system of coordinates in which a position of relevant points and parts of the surgical instrument are known; and an image processing system configured to:
receive the images, video, or images and video acquired by the camera, wherein the images, video, or images and video include the WM, one or more of the TMs, or the WM and one or more of the TMs in a field of view of the camera;
receive a 3D reconstruction of points of the anatomical part in the first coordinate system; and
output information derived from the 3D reconstruction to a display to assist execution of a clinical procedure using the one or more surgical instruments while keeping the WM, one or more of the TMs, or the WM and one or more of the TMs in the field of view of the camera, wherein the output comprises real-time guidance including a respective location of each surgical instrument in the first coordinate system.
35. The system of claim 34, wherein the WM comprises one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle-like object with a flat lateral facet for trans-dermic insertion, or a flat button-like object that is pulled by a thread or guide.
36. The system of claim 34, wherein each pattern comprises recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation and translation that map points in the coordinate system of the pattern into points in the coordinate system of the camera.
37. The system of claim 34 wherein at least one of the known patterns is disposed on a planar surface.
38. The system of claim 34, wherein at least one of the known patterns is disposed on a non-planar surface.
39. The system of claim 34, further comprising a further surgical instrument, wherein the camera is rigidly coupled to the further surgical instrument.
40. The system of claim 39, further comprising:
determining a location of a portion of the further surgical instrument in the first coordinate system according to an image that includes the WM and a known positional relationship between the portion of the further surgical instrument and the camera; wherein the output further comprises the location of the further surgical instrument.
41. The system of claim 34, wherein:
one of the surgical instruments is an end effector of a robotic tool; and the image processing system is further configured to send a command to the robotic tool to control the end effector.
42. The system of claim 41, wherein the camera is rigidly coupled to the end effector.
43. The system of claim 34, further comprising a tablet computer, the tablet computer comprising the image processing system and the display.
44. The system of claim 34, further comprising:
a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device.
45. The system of claim 44, wherein the point location device comprises one or more of:
a touch probe; a laser pointer; or a time-of-flight device.
46. The system of claim 34, further comprising a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device, wherein the image processing system is configured to reconstruct the known points in the first coordinate system by, for at least one image:
determining a 3D pose of the WM in camera coordinates; determining a 3D pose of the point location device TM in camera coordinates; and reconstructing a point captured by the point location device according to the 3D pose of the WM and the 3D pose of the point location device TM.
47. The system of claim 34, wherein the image processing system is configured to determine the locations of the surgical instruments in the first coordinate system by:
determining a 3D pose of the WM in camera coordinates; determining a 3D pose of the surgical instrument TMs in camera coordinates; and determining a 3D pose of the surgical instrument TMs in the first coordinate system according to the 3D pose of the WM in camera coordinates and the 3D pose of the surgical instrument TMs in camera coordinates.
48. The system of claim 34, wherein the camera is calibrated such that image points expressed in pixel coordinates are mapped into image points expressed in distance units according to the projection function, the camera intrinsic parameters, and the camera distortion parameters.
49. The system of claim 48, wherein the image processing system is configured to:
calibrate the camera by acquiring, with the camera, a calibration image of a known calibration grid and determining, based on the calibration image, the camera intrinsic parameters and the camera distortion parameters.
50. The system of claim 49, wherein the image processing system is configured to:
update the camera intrinsic parameters and the camera distortion parameters during the surgical procedure to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, or variations in zoom.
51. The system of claim 49, further comprising:
a point location device for locating points in the rigid anatomical part, wherein a TM is rigidly coupled to the point location device; wherein the calibration image comprises the TM of the point location device and a portion of the point location device, wherein the image processing system is configured to: calibrate the point location device by determining, based on the calibration image, a location of the portion of the point location device according to an inverse of a rigid transformation that encodes a 3D pose of the TM in camera coordinates.
52. The system of claim 34, wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to one or more known points of the anatomical part and according to a location of the WM in the images, video, or images and video.
53. The system of claim 34, wherein the WM or at least one of the TMs has multiple planar facets with multiple known patterns, where the locations of the multiple known patterns are known in a common coordinate system of the WM or the at least one of the TMs.
54. A method for computer-aided execution of surgery and diagnosis in an anatomical region comprising a rigid anatomical part, the method comprising:
attaching a world marker (WM) to a surface of the rigid anatomical part, the WM comprising a known pattern defining a first system of coordinates; acquiring, by a free-moving camera, first intra-operative images, video, or images and video, while keeping the WM in a field of view of the camera,
wherein the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f −1 (u; k, ξ)
with f − being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
receiving a 3D reconstruction of points of the anatomical part in the first coordinate system; during a surgical procedure performed with one or more surgical instruments, wherein each surgical instrument has a respective tool marker (TM), each TM comprising a respective known pattern that defines a respective second system of coordinates in which positions of one or more points on the instrument are known, acquiring, by the camera, second intra-operative images, video, or images and video, while keeping the WM and the surgical instrument TMs in a field of view of the camera; processing, by the image processing system, the second images, video, or images and video to determine locations of the surgical instruments in the first coordinate system; and outputting information derived from the 3D reconstruction to a display to assist execution of a clinical procedure using the one or more surgical instruments while keeping the WM, one or more of the TMs, or the WM and one or more of the TMs in the field of view of the camera, wherein the output comprises real-time guidance including a respective location of each surgical instrument in the first coordinate system.
55. The method of claim 54, wherein the image processing system is configured to determine the locations of the surgical instruments in the first coordinate system by:
determining a 3D pose of the WM in camera coordinates; determining a 3D pose of the surgical instrument TMs in camera coordinates; and determining a 3D pose of the surgical instrument TMs in the first coordinate system according to the 3D pose of the WM in camera coordinates and the 3D pose of the surgical instrument TMs in camera coordinates.
56. The method of claim 54, wherein the camera is calibrated such that image points expressed in pixel coordinates are mapped into image points expressed in distance units according to the projection function, the camera intrinsic parameters, and the camera distortion parameters.
57. The method of claim 56, further comprising:
calibrating the camera by acquiring, with the camera, a calibration image of a known calibration grid and determining, based on the calibration image, the camera intrinsic parameters and the camera distortion parameters.
58. The method of claim 57, further comprising:
updating the camera intrinsic parameters and the camera distortion parameters during the surgical procedure to compensate for one or more of differences in medium, rotation of lens scope with respect to camera-head, or variations in zoom.
59. The method of claim 57, further comprising:
while acquiring the first intra-operative images, locating one or more points in the rigid anatomical part using a point location device, wherein a tool marker (TM) is rigidly coupled to the point location device, wherein the TM comprises a known pattern that defines a third system of coordinates in which a position of a portion of the point location device is known; wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to the one or more located points and according to a location of the WM in the images, video, or images and video; wherein the calibration image comprises the TM of the point location device and the portion of the point location device, the method further comprising:
calibrating the point location device by determining, based on the calibration image, a location of the portion of the point location device according to an inverse of a rigid transformation that encodes a 3D pose of the TM in camera coordinates.
60. The method of claim 54, further comprising:
while acquiring the first intra-operative images, locating one or more points in the rigid anatomical part using a point location device, wherein a TM is rigidly coupled to the point location device, wherein the TM comprises a known pattern that defines a third system of coordinates in which a position of a portion of the point location device is known; wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to the one or more located points and according to a location of the WM in the images, video, or images and video.
61. The method of claim 60, wherein the point location device comprises one or more of:
a touch probe; a laser pointer; or a time-of-flight device.
62. The method of claim 60, wherein the image processing system is configured to reconstruct the located points in the first coordinate system by, for at least one image:
determining a 3D pose of the WM in camera coordinates; determining a 3D pose of the point location device TM in camera coordinates; and reconstructing a point P located by the point location device according to the 3D pose of the WM and the 3D pose of the point location device TM.
63. The method of claim 54, wherein the WM comprises one or more of a screw-like object with a flat head or facet, a nail-like object with a flat head or facet to be fixed by pressure, a needle-like object with a flat lateral facet for trans-dermic insertion, or a flat button-like object that is pulled by a thread or guide.
64. The method of claim 54, wherein each pattern comprises recognizable fiducial points for estimating a plane-to-image homography that, given a camera calibration, can be factorized in a rotation and translation that map points in the coordinate system of the pattern into points in the coordinate system of the camera.
65. The method of claim 54, wherein at least one of the known patterns is disposed on a planar surface.
66. The method of claim 54, wherein at least one of the known patterns is disposed on a non-planar surface.
67. The method of claim 54, further comprising:
determining a location of a portion of a further surgical instrument, the camera rigidly coupled to the further surgical instrument, according to an image that includes the WM and a known positional relationship between the portion of the further surgical instrument and the camera.
68. The method of claim 54, wherein:
one of the surgical instruments is an end effector of a robotic tool; and the image processing system is further configured to send a command to the robotic tool to control the end effector.
69. The method of claim 68, wherein the camera is rigidly coupled to the end effector.
70. The method of claim 54, wherein the image processing system and the display comprise a tablet computer.
71. The method of claim 54, wherein receiving the 3D reconstruction comprises performing the 3D reconstruction according to one or more known points of the anatomical part and according to a location of the WM in the images, video, or images and video.
72. The method of claim 54, wherein the WM or at least one of the TMs has multiple planar facets includes multiple known patterns, where the locations of the multiple known patterns are known in a common coordinate system of the WM or the at least one of the TMs.
73. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a camera to acquire intra-operative images, video, or images and video of an operating field,
where the camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f −1 (u; k, ξ)
with f −1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect each other, wherein the object comprises a first known pattern that defines a first system of coordinates; an instrument comprising a surgical instrument or a point location device, wherein the camera is rigidly coupled to the instrument such that camera has a known positional relationship with a portion of the instrument; an image processing system configured to:
receive the images, video, or images and video acquired by the camera, wherein the images, video, or images and video include the WM in a field of view of the camera;
receive a 3D reconstruction of points of the anatomical part in the first coordinate system; and
output information derived from the 3D reconstruction to a display to assist execution of a clinical procedure using the instrument while keeping the WM in the field of view of the camera, wherein the output comprises a location of the instrument in the first coordinate system.
74. The system of claim 34, wherein the image processing system is further configured to determine the 3D reconstruction of the points of the anatomical part by determining locations of points of the anatomical part.
75. The system of claim 34, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images captured by the free-moving camera.
76. The system of claim 75, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images each comprising the WM and a point, of the anatomical part, to be located.
77. The system of claim 76, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images each comprising the WM and an illumination by light of a light source of the point to be located.
78. The system of claim 74, wherein the image processing system is further configured to determine the 3D reconstruction of the points using a light source and/or a time-of-flight device.
79. The system of claim 78, further comprising:
the light source to illuminate points of the anatomical part for locating the points for 3D reconstruction.
80. The system of claim 78, wherein:
the system further comprises a light source to emit structured light; and the image processing system is further configured to determine a location of a point on the anatomical part by analyzing at least one image of a point of the anatomical part illuminated with the structured light.
81. The system of claim 80, wherein the light source is arranged to emit collimated light.
82. The system of claim 81, wherein the light source is configured to emit a collimated beam of visible light.
83. The system of claim 80, wherein the light source is a laser.
84. The system of claim 74, wherein the image processing system is further configured to determine the 3D reconstruction of the points using a touch probe.
85. The system of claim 84, further comprising:
the touch probe, wherein the touch probe has a TM.
86. The system of claim 34, further comprising:
a second surgical instrument, wherein the second surgical instrument is a guide.
87. The system of claim 86, wherein the guide comprises a TM.
88. The system of claim 34, wherein the image processing system is configured to monitor a location of a portion of a surgical instrument in a field of view of the free-moving camera.
89. The system of claim 88, wherein the image processing system is configured to monitor a location of a distal end of the surgical instrument in the field of view of the free-moving camera.
90. The system of claim 89, wherein the image processing system is configured to monitor a location of a tip of the surgical instrument in a field of view of the free-moving camera.
91. The system of claim 90, wherein the surgical instrument is a touch probe and the tip of the surgical instrument is a tip of the touch probe.
92. The system of claim 88, wherein the surgical instrument is rigidly coupled to the free-moving camera.
93. A system for computer-aided surgery and diagnosis in an anatomical region of a patient comprising a rigid, non-deformable anatomical part, the system comprising:
a free-moving camera to acquire intra-operative images, video, or images and video of an operating field,
where the free-moving camera is calibrated such that image points u expressed in pixel coordinates are mapped into image points x expressed in metric units according to
x=f −1 (u; k, ξ)
with f −1 being the inverse of a projection function f and where k and ξ stand respectively for camera intrinsic and distortion parameters;
a world marker (WM) comprising an object configured to be rigidly coupled to the anatomical part such that the WM and the anatomical part do not move one with respect each other; and an image processing system configured to:
receive the images, video, or images and video acquired by the free-moving camera, wherein the images, video, or images and video include the WM in a field of view of the camera;
receive a 3D reconstruction of points of the anatomical part in a first coordinate system defined using the WM; and
output information derived from the 3D reconstruction for display to assist execution of a clinical procedure using a surgical instrument while keeping the WM in the field of view of the camera, wherein the output comprises guidance including a location of a surgical instrument in the first coordinate system.
94. The system of claim 93, wherein the WM is a planar object.
95. The system of claim 93, wherein the WM comprises a known pattern.
96. The system of claim 93, The system of claim 34, wherein the image processing system is further configured to determine the 3D reconstruction of the points of the anatomical part by determining locations of points of the anatomical part.
97. The system of claim 93, wherein the image processing system is further configured to determine the 3D reconstruction of the points using one or more images captured by the free-moving camera.
98. The system of claim 93, wherein the image processing system is configured to monitor a location of a portion of a surgical instrument in a field of view of the free-moving camera.
99. The system of claim 98, wherein the surgical instrument is rigidly coupled to the free-moving camera.
100. The system of claim 99, wherein the surgical instrument is a touch probe.
101. The system of claim 93, further comprising:
a surgical instrument, wherein the surgical instrument is a guide.
102. The system of claim 101, wherein the guide comprises a tool marker (TM).Join the waitlist — get patent alerts
Track USRE49930E — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.