US2024152753A1PendingUtilityA1
Method and apparatus with training of batch norm parameter
Est. expiryNov 7, 2042(~16.3 yrs left)· nominal 20-yr term from priority
G06N 3/084G06N 3/04G06N 3/08G06N 3/045
61
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
Disclosed is a processor implemented method that includes calculating a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer, calculating a final loss using a regularization loss term determined based on the quantization error for each channel, and updating a batch norm parameter of the first layer in a direction to decrease the final loss.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1 . A processor-implemented method, the method comprising:
calculating a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer; calculating a final loss using a regularization loss determined based on the quantization error for each channel; and updating a batch norm parameter of the first layer in a direction to decrease the final loss.
2 . The method of claim 1 , wherein the calculating of the quantization error for each channel further comprises quantifying, for each of the channels, the quantization error as an inverse of a signal-to-quantization-noise ratio (SQNR) for a corresponding scale.
3 . The method of claim 1 , wherein the calculating of the final loss comprises calculating the regularization loss by calculating an average of the quantization error for each channel.
4 . The method of claim 1 , wherein the calculating of the final loss comprises calculating the final loss by summing the regularization loss and a cross-entropy loss.
5 . The method of claim 1 , wherein the updating of the batch norm parameter comprises deriving the batch norm parameter in a direction reduces a value of the regularization loss by performing stochastic gradient descent using the final loss.
6 . The method of claim 1 , wherein the updating of the batch norm parameter comprises fixing values of all parameters of the neural network other than the batch norm parameter when updating the batch norm parameter.
7 . The method of claim 1 , wherein the updating of the batch norm parameter comprises training a quantization scale of the first layer to reduce the final loss.
8 . The method of claim 1 , wherein the updating of the batch norm parameter comprises measuring performance of the neural network having the updated batch norm parameter.
9 . The method of claim 1 , further comprising:
calculating the regularization loss—according to:
L
reg
l
=
1
❘
"\[LeftBracketingBar]"
c
❘
"\[RightBracketingBar]"
∑
j
∈
C
1
/
S
Q
N
R
(
f
l
j
,
α
l
+
1
)
,
S
Q
N
R
(
X
,
α
)
=
E
[
X
2
]
E
[
X
-
Q
(
X
,
α
)
]
wherein L reg l denotes the regularization loss, SQNR denotes a signal-to-quantization-noise ratio (SQNR), C denotes a channel, f and X denote the activation data, l denotes a layer, and a denotes a quantization scale.
10 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .
11 . An apparatus, the apparatus comprising:
one or more processors configured to execute instructions; and a memory storing the instructions, wherein execution of the instructions by the one or more processors configure the one or more processors to:
calculate a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer;
calculate a final loss using a regularization loss term based on the quantization error for each channel; and
update a batch norm parameter of the first layer in a direction to decrease the final loss.
12 . The apparatus of claim 11 , wherein the one or more processors are configured to, when calculating the quantization error for each channel, quantify, for each of the channels, the quantization error as an inverse of a signal-to-quantization-noise ratio (SQNR) for a corresponding scale.
13 . The apparatus of claim 11 , wherein the one or more processors are configured to, when calculating the final loss, calculate the regularization loss by calculating an average of the quantization error for each channel.
14 . The apparatus of claim 11 , wherein the one or more processors are configured to, when calculating the final loss, calculate the final loss by summing the regularization loss and a cross-entropy loss.
15 . The apparatus of claim 11 , wherein the one or more processors are configured to, when updating the batch norm parameter, derive the batch norm parameter in a direction that reduces a value of the regularization loss by performing stochastic gradient descent using the final loss.
16 . The apparatus of claim 11 , wherein the one or more processors are configured to, when updating the batch norm parameter, fix values of all parameters of the neural network other than the batch norm parameter.
17 . The apparatus of claim 11 , wherein the one or more processors are configured to, when updating the batch norm parameter, train a quantization scale of the first layer to reduce the final loss.
18 . The apparatus of claim 11 , wherein the one or more processors are configured to, when updating the batch norm parameter, measure performance of the neural network having the updated batch norm parameter.
19 . The apparatus of claim 11 , wherein the one or more processors are configured to calculate the regularization loss according to
L
reg
l
=
1
❘
"\[LeftBracketingBar]"
c
❘
"\[RightBracketingBar]"
∑
j
∈
C
1
/
S
Q
N
R
(
f
l
j
,
α
l
+
1
)
,
S
Q
N
R
(
X
,
α
)
=
E
[
X
2
]
E
[
X
-
Q
(
X
,
α
)
]
wherein L reg l denotes the regularization loss, SQNR denotes a signal-to-quantization-noise ratio (SQNR), C denotes a channel, f and X denote the activation data, l denotes a layer, and α denotes a quantization scale.Join the waitlist — get patent alerts
Track US2024152753A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.