US2019286921A1PendingUtilityA1
Structured Prediction Crosswalk Generation
Est. expiryMar 14, 2038(~11.6 yrs left)· nominal 20-yr term from priority
G06T 2207/20081G06T 7/73G06T 2207/10024G06T 2207/10028G06T 2207/30184G06T 2207/20084G06T 2207/10032G06F 16/587G06V 10/82G06V 10/764G06V 20/588G06F 16/909G06F 16/55G06F 16/29G06T 7/70G06T 2207/30256G06K 9/00798
45
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
A method includes receiving image data associated with an image of a roadway including a crosswalk, generating a plurality of different characteristics of the image based on the image data, determining a position of the crosswalk on the roadway based on the plurality of different characteristics, the position including a first boundary and a second boundary of the crosswalk in the roadway, and providing map data associated with a map of the roadway, the map data including the position of the crosswalk on the roadway in the map. The plurality of different characteristics include a classification of one or more elements of the image, a segmentation of the one or more elements of the image, and one or more angles of the one or more elements of the image with respect to a line in the roadway.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1 . A computer-implemented method, comprising:
receiving, with a computing system comprising one or more processors, image data associated with an image of a roadway including a crosswalk; generating, with the computing system, a plurality of different characteristics of the image based on the image data, wherein the plurality of different characteristics include a classification of one or more elements of the image, a segmentation of the one or more elements of the image, and one or more angles of the one or more elements of the image with respect to a line in the roadway; determining, with the computing system, a position of the crosswalk on the roadway based on the plurality of different characteristics, wherein the position includes a first boundary and a second boundary of the crosswalk in the roadway; and providing, with the computing system, map data associated with a map of the roadway, wherein the map data includes the position of the crosswalk on the roadway in the map.
2 . The computer-implemented method of claim 1 , wherein determining the position of the crosswalk on the roadway based on the plurality of different characteristics further comprises:
determining, based on the one or more angles and the image data, a first likelihood of the one or more elements being on the first boundary or the second boundary; determining, based on the one or more angles and the image data, a second likelihood of the one or more elements being between the first boundary and the second boundary; and determining, based on the first likelihood and the second likelihood, whether the position of the crosswalk on the roadway includes the one or more elements.
3 . The computer-implemented method of claim 1 , wherein the position of the crosswalk on the roadway is determined according to the following function:
max
χ
1
,
χ
2
,
β
λ
I
(
φ
seg
,
,
β
(
χ
2
)
-
φ
seg
,
,
β
(
χ
1
)
)
+
(
1
-
λ
I
)
(
φ
dt
,
,
β
(
χ
2
)
+
φ
dt
,
,
β
(
χ
1
)
)
,
wherein ϕ seg is a segmented output feature map including the segmentation of the one or more elements in the image, ϕ dt is a classification output feature map including the classification of the one or more elements in the image, χ 1 and χ 2 are points on the line l in the roadway, β is the one or more angles of the one or more elements of the image with respect to the line l, and λ 1 is a weight that balances ϕ seg and ϕ dt .
4 . The computer-implemented method of claim 1 , wherein the segmentation of the one or more elements of the image includes a segmentation image including a plurality of elements inside the first boundary and the second boundary and a plurality of elements outside the first boundary and second boundary.
5 . The computer-implemented method of claim 1 , wherein the classification of the one or more elements of the image includes a distance map including a plurality of elements associated with at least one of the first boundary and the second boundary, the distance map including at least one distance to the at least one of the first boundary and the second boundary for at least one element of the plurality of elements within a threshold distance of the at least one of the first boundary and the second boundary.
6 . The computer-implemented method of claim 1 , wherein the line includes at least a portion of a centerline of the roadway.
7 . The computer-implemented method of claim 1 , wherein the one or more angles of the one or more elements comprise a dilated representation of a boundary angle including an x-value and an y-value with respect to the line.
8 . The computer-implemented method of claim 1 , further comprising:
training, with the computing system, a convolutional neural network by modifying parameters to optimize a loss function of the convolutional neural network that depends on ground truth labels and one or more predictions of the plurality of different characteristics, wherein the loss function comprises a sum of a segmentation loss, an alignment loss, and a boundary loss; and processing, with the computing system, the image data associated with the image of the roadway including the crosswalk using the trained convolutional neural network, to generate a plurality of images associated with the plurality of different characteristics of the image.
9 . A computing system, comprising:
one or more processors programmed and/or configured to:
receive image data associated with an image of a roadway including a crosswalk;
generate a plurality of different characteristics of the image based on the image data, wherein the plurality of different characteristics include a classification of one or more elements of the image, a segmentation of the one or more elements of the image, and one or more angles of the one or more elements of the image with respect to a line in the roadway;
determine a position of the crosswalk on the roadway based on the plurality of different characteristics, wherein the position includes a first boundary and a second boundary of the crosswalk in the roadway; and
provide map data associated with a map of the roadway, wherein the map data includes the position of the crosswalk on the roadway in the map.
10 . The computing system of claim 9 , wherein the one or more processors are further programmed or configured to determine the position of the crosswalk on the roadway based on the plurality of different characteristics by:
determining, based on the one or more angles and the image data, a first likelihood of the one or more elements being on the first boundary or the second boundary; determining, based on the one or more angles and the image data, a second likelihood of the one or more elements being between the first boundary and the second boundary; and determining, based on the first likelihood and the second likelihood, whether the position of the crosswalk on the roadway includes the one or more elements.
11 . The computing system of claim 9 , wherein the position of the crosswalk on the roadway is determined according to the following function:
max
χ
1
,
χ
2
,
β
λ
I
(
φ
seg
,
,
β
(
χ
2
)
-
φ
seg
,
,
β
(
χ
1
)
)
+
(
1
-
λ
I
)
(
φ
dt
,
,
β
(
χ
2
)
+
φ
dt
,
,
β
(
χ
1
)
)
,
wherein ϕ seg is a segmented output feature map including the segmentation of the one or more elements in the image, ϕ dt is a classification output feature map including the classification of the one or more elements in the image, χ 1 and χ 2 are points on the line l in the roadway, β is the one or more angles of the one or more elements of the image with respect to the line l, and λ 1 is a weight that balances ϕ seg and ϕ dt .
12 . The computing system of claim 9 , wherein the segmentation of the one or more elements of the image includes a segmentation image including a plurality of elements inside the first boundary and the second boundary and a plurality of elements outside the first boundary and second boundary.
13 . The computing system of claim 9 , wherein the classification of the one or more elements of the image includes a distance map including a plurality of elements associated with at least one of the first boundary and the second boundary, the distance map including at least one distance to the at least one of the first boundary and the second boundary for at least one element of the plurality of elements within a threshold distance of the at least one of the first boundary and the second boundary.
14 . The computing system of claim 9 , wherein the line includes at least a portion of a centerline of the roadway.
15 . The computing system of claim 9 , wherein the one or more angles of the one or more elements comprise a dilated representation of a boundary angle including an x value and a y value with respect to the line.
16 . The computing system of claim 9 , wherein the one or more processors are further programmed and/or configured to:
train a convolutional neural network by modifying parameters to optimize a loss function of the convolutional neural network that depends on ground truth labels and one or more predictions of the plurality of different characteristics, wherein the loss function comprises a sum of a segmentation loss, an alignment loss, and a boundary loss; and process the image data associated with the image of the roadway including the crosswalk using the trained convolutional neural network, to generate a plurality of images associated with the plurality of different characteristics of the image.
17 . A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
receive image data associated with an image of a roadway including a crosswalk; generate a plurality of different characteristics of the image based on the image data, wherein the plurality of different characteristics include a classification of one or more elements of the image, a segmentation of the one or more elements of the image, and one or more angles of the one or more elements of the image with respect to a line in the roadway; determine a position of the crosswalk on the roadway based on the plurality of different characteristics, wherein the position includes a first boundary and a second boundary of the crosswalk in the roadway; and provide map data associated with a map of the roadway, wherein the map data includes the position of the crosswalk on the roadway in the map.
18 . The computer program product of claim 17 , wherein the instructions cause the at least one processor to determine the position of the crosswalk on the roadway based on the plurality of different characteristics by:
determining, based on the one or more angles and the image data, a first likelihood of the one or more elements being on the first boundary or the second boundary; determining, based on the one or more angles and the image data, a second likelihood of the one or more elements being between the first boundary and the second boundary; and determining, based on the first likelihood and the second likelihood, whether the position of the crosswalk on the roadway includes the one or more elements
19 . The computer program product of claim 17 , wherein the position of the crosswalk on the roadway is determined according to the following function:
max
χ
1
,
χ
2
,
β
λ
I
(
φ
seg
,
,
β
(
χ
2
)
-
φ
seg
,
,
β
(
χ
1
)
)
+
(
1
-
λ
I
)
(
φ
dt
,
,
β
(
χ
2
)
+
φ
dt
,
,
β
(
χ
1
)
)
,
wherein ϕ seg is a segmented output feature map including the segmentation of the one or more elements in the image, ϕ dt is a classification output feature map including the classification of the one or more elements in the image, χ 1 and χ 2 are points on the line l in the roadway, β is the one or more angles of the one or more elements of the image with respect to the line l, and λ I is a weight that balances ϕ seg and ϕ dt .
20 . The computer program product of claim 17 , wherein the line includes at least a portion of a centerline of the roadway.Join the waitlist — get patent alerts
Track US2019286921A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.