US11410309B2ActiveUtilityA1
Method, device, and computer program product for deep lesion tracker for monitoring lesions in four-dimensional longitudinal imaging
Est. expiryDec 3, 2040(~14.4 yrs left)· nominal 20-yr term from priority
G06N 3/045G16H 30/40G06N 3/088G16H 50/20G06N 3/09G06N 3/0464G06N 3/0895G06T 2207/30096G06T 2207/20016G06T 2207/10081G06T 7/248G06T 2207/20084G06T 2207/20081G06V 10/50G06V 10/454G06T 2207/20076G06V 10/478G06V 10/457G06T 7/0014G06V 2201/032G06V 2201/031G06V 10/82G06T 5/50G06T 3/0006G06T 3/02
78
PatentIndex Score
1
Cited by
9
References
18
Claims
Abstract
The present disclosure provides a computer-implemented method, a device, and a computer program product for deep lesion tracker. The method includes inputting a search image into a first three-dimensional DenseFPN (feature pyramid network) of an image encoder and inputting a template image into a second three-dimensional DenseFPN of the image encoder to extract image features; encoding anatomy signals of the search image and the template image as Gaussian heatmaps, and inputting the Gaussian heatmap of the template image into a first anatomy signal encoders (ASE) and inputting the Gaussian heatmap of the search image into a second ASE to extract anatomy features; inputting the image features and the anatomy features into a fast cross-correlation layer to generate correspondence maps, and computing a probability map according to the correspondence maps; and performing supervised learning or self-supervised learning to predict a lesion center in the search image.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1. A deep lesion tracker method for medical images, the method comprising:
providing an image pair including a search image and a template image;
inputting the search image into a first three-dimensional DenseFPN (feature pyramid network) of an image encoder, and inputting the template image into a second three-dimensional DenseFPN of the image encoder to extract image features of the search image and the template image in three different scales, wherein the first and second three-dimensional DenseFPNs are configured with shared weights;
encoding anatomy signals of the search image and the template image as Gaussian heatmaps centered at lesion locations, inputting the Gaussian heatmap of the template image into a first anatomy signal encoders (ASE) and inputting the Gaussian heatmap of the search image into a second ASE to extract anatomy features of the search image and the template image in three different scales, wherein the first and the second ASEs are configured with shared weights;
inputting the image features and the anatomy features into a fast cross-correlation layer to generate correspondence maps, and computing a probability map according to the correspondence maps; and
performing supervised learning or self-supervised learning to predict a lesion center in the search image.
2. The method according to claim 1 , wherein encoding the anatomy signals as the Gaussian heatmaps includes:
for the template image, using a location and a size of a template lesion to compute the anatomy signals of the template image; and
for the search image, using an affine-projected location and an affine-projected size of the template lesion to compute the anatomy signals of the search image.
3. The method according to claim 2 , wherein inputting the image features and the anatomy features into the fast cross-correlation layer to generate the correspondence maps includes:
fusing the image features of the template image and the anatomy features of the template image.
4. The method according to claim 3 , wherein after fusing the image features of the template image and the anatomy features of the template image, the method further includes:
defining a cropping function to extract a template kernel K and another template kernel K g , wherein:
a size of the template kernel K g is greater than a size of the template kernel K; and the template kernel K g is decomposed into kernels K g,x , K g,y , and K g,z , along axial, coronal, and sagittal directions, respectively.
5. The method according to claim 4 , wherein a correspondence map is computed by:
M
=
(
K
*
S
)
+
(
∑
i
∈
x
,
y
,
z
K
g
,
i
*
S
)
wherein + denotes element-wise sum, * denotes multiplication, S=ψ(I s )⊙ϕ(G s ), I s , I s is the search image, G s is an anatomy signal map of the search image, ⊙ denotes element-wise multiplication, ψ and ϕ denote network encoders that generate image features and anatomy features, respectively.
6. The method according to claim 5 , wherein after computing the correspondence map, the method further includes:
determining the lesion center in the search image according to the probability map computed based on the correspondence maps.
7. A deep lesion tracker device for medical images comprising:
a memory, containing a computer program stored thereon; and
a processor, coupled with the memory and configured, when the computer program being executed, to perform a method including:
providing an image pair including a search image and a template image;
inputting the search image into a first three-dimensional DenseFPN (feature pyramid network) of an image encoder and inputting the template image into a second three-dimensional DenseFPN of the image encoder to extract image features of the search image and the template image in three different scales, wherein the first and second three-dimensional DenseFPNs are configured with shared weights;
encoding anatomy signals of the search image and the template image as Gaussian heatmaps centered at lesion locations, and inputting the Gaussian heatmap of the template image into a first anatomy signal encoders (ASE) and inputting the Gaussian heatmap of the search image into a second ASE to extract anatomy features of the search image and the template image in three different scales, wherein the first and the second ASEs are configured with shared weights;
inputting the image features and the anatomy features into a fast cross-correlation layer to generate correspondence maps, and computing a probability map according to the correspondence maps; and
performing supervised learning or self-supervised learning to predict a lesion center in the search image.
8. The device according to claim 7 , wherein encoding the anatomy signals as the Gaussian heatmaps includes:
for the template image, using a location and a size of a template lesion to compute the anatomy signals of the template image; and
for the search image, using an affine-projected location and an affine-projected size of the template lesion to compute the anatomy signals of the search image.
9. The device according to claim 8 , wherein inputting the image features and the anatomy features into the fast cross-correlation layer to generate the correspondence maps includes:
fusing the image features of the template image and the anatomy features of the template image.
10. The device according to claim 9 , wherein after fusing the image features of the template image and the anatomy features of the template image, the method further includes:
defining a cropping function to extract a template kernel K and another template kernel K g , wherein:
a size of the template kernel K g is greater than a size of the template kernel K; and the template kernel K g is decomposed into kernels K g,x , K g,y and K g,z , along axial, coronal, and sagittal directions, respectively.
11. The device according to claim 10 , wherein a correspondence map is computed by:
M
=
(
K
*
S
)
+
(
∑
i
∈
x
,
y
,
z
K
g
,
i
*
S
)
wherein + denotes element-wise sum, * denotes multiplication, S=ψ(I s )⊙ϕ(G s ), I s is the search image, G s is an anatomy signal map of the search image, ⊙ denotes element-wise multiplication, and ψ and ϕ denote network extractors that generate image features and anatomy features, respectively.
12. The device according to claim 11 , after computing the correspondence map, the method further includes:
determining the lesion center in the search image according to the probability map computed based on the correspondence maps.
13. A computer program product comprising a non-transitory computer-readable storage medium and program instructions stored therein, the program instructions being configured to be executable by a computer to cause the computer to implement a method comprising:
providing an image pair including a search image and a template image;
inputting the search image into a first three-dimensional DenseFPN (feature pyramid network) of an image encoder and inputting the template image into a second three-dimensional DenseFPN of the image encoder to extract image features of the search image and the template image in three different scales, wherein the first and second three-dimensional DenseFPNs are configured with shared weights;
encoding anatomy signals of the search image and the template image as Gaussian heatmaps centered at lesion locations, and inputting the Gaussian heatmap of the template image into a first anatomy signal encoders (ASE) and inputting the Gaussian heatmap of the search image into a second ASE to extract anatomy features of the search image and the template image in three different scales, wherein the first and the second ASEs are configured with shared weights;
inputting the image features and the anatomy features into a fast cross-correlation layer to generate correspondence maps, and computing a probability map according to the correspondence maps; and
performing supervised learning or self-supervised learning to predict a lesion center in the search image.
14. The computer program product according to claim 13 , wherein encoding the anatomy signals as the Gaussian heatmaps includes:
for the template image, using a location and a size of a template lesion to compute the anatomy signals of the template image; and
for the search image, using an affine-projected location and an affine-projected size of the template lesion to compute the anatomy signals of the search image.
15. The computer program product according to claim 14 , wherein inputting the image features and the anatomy features into the fast cross-correlation layer to generate the correspondence maps includes:
fusing the image features of the template image and the anatomy features of the template image.
16. The computer program product according to claim 15 , wherein after fusing the image features of the template image and the anatomy features of the template image, the method further includes:
defining a cropping function to extract a template kernel K and another template kernel K g , wherein:
a size of the template kernel K g is greater than a size of the template kernel K; and the template kernel K g is decomposed into kernels K g,x , K g,y and K g,z , along axial, coronal, and sagittal directions, respectively.
17. The computer program product according to claim 16 , wherein a correspondence map is computed by:
M
=
(
K
*
S
)
+
(
∑
i
∈
x
,
y
,
z
K
g
,
i
*
S
)
wherein + denotes element-wise sum, * denotes multiplication, S=ψ(I s )⊙ϕ(G s ), I s is the search image, G s is an anatomy signal map of the search image, ⊙ denotes element-wise multiplication, and ψ and ϕ denote network encoders that generate image features and anatomy features, respectively.
18. The computer program product according to claim 17 , wherein after computing the correspondence map, the method further includes:
determining the lesion center in the search image according to the probability map computed based on the correspondence maps.Join the waitlist — get patent alerts
Track US11410309B2 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.