US2004091047A1PendingUtilityA1
Method and apparatus for nonlinear multiple motion model and moving boundary extraction
Est. expiryNov 11, 2022(expired)· nominal 20-yr term from priority
H04N 19/17G06T 2207/10016H04N 19/105G06T 2207/20021H04N 19/139H04N 19/553H04N 19/543G06T 2207/20016G06T 7/223H04N 19/521G06T 7/207G06T 7/215H04N 7/18G06T 7/20H04N 7/12
48
PatentIndex Score
0
Cited by
0
References
0
Claims
Abstract
A method and apparatus for nonlinear multiple motion model and moving boundary extraction are disclosed. In one embodiment, an input image is received, the input image is partitioned into regions/blocks, and the new multiple motion model is applied to each region to extract the motions and associated moving boundaries.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1 . A method for motion estimation comprising:
receiving an input image; receiving a set of reference images; partitioning said input image into regions; and for each region:
extracting one or more motions; and
extracting one or more associated moving boundaries.
2 . The method of claim 1 wherein said set of reference images is selected from the group consisting of one or more past frames, one or more future frames, and one or more past frames and one or more future frames.
3 . The method of claim 2 wherein extracting said one or more motions and extracting said one or more associated moving boundaries involves a multiple motion model with a nonlinear coupling between space and time variables, and a boundary model directly coupled to said one or more motions.
4 . The method of claim 3 wherein said extracting one or more associated moving boundaries is determined implicitly from a region competition model, which is driven by reducing a prediction error for a boundary selection.
5 . The method of claim 3 wherein an estimate for an associated moving boundary for 2 motions is based on duality of occlusion relative to a past frame and a future frame.
6 . The method of claim 3 wherein said time variable allows for a combined motion estimation with respect to a past frame and a future frame in order to capture 2 motions.
7 . The method of claim 3 wherein said time variable allows for a combined motion estimation with respect to a set of past and future frames in order to capture more than 2 motions with non-intersecting boundaries.
8 . The method of claim 3 wherein a nonlinear function of said time variable, denoted t 1 , is coupled to pixel positions, denoted (x,y), and is used to control and refine an estimate of said one or more associated moving boundaries, and has a form:
t 1 =F ( s )
where s is a function of the pixel position (i.e., s=B(x,y)) within one of said regions.
9 . The method of claim 8 wherein a pixel assignment to a reference frame, and hence a motion class/state, is determined by coupling the method of claim 8 to one or more motion fields, wherein said coupling results in a nonlinear coupled model which evolves according to a driving force to reduce prediction error, and said extracted moving boundaries, and said motion states are derived from this prediction error minimization, and wherein F(s) from claim 8 in this nonlinear coupled model saturates to reference time values away from particular said associated moving boundaries, and is steeper near particular said associated moving boundaries.
10 . The method of claim 9 wherein for the case of 2 motions, using a past frame and a future frame as reference images, said particular said associated moving boundaries are determined when substantially s=−0.5.
11 . The method of claim 9 wherein said motion model has a form for 2 motions of:
t 1 =F ( s ) s=B ( x,y ) x 1 =v 1 x ( x,y )+( v 2 x ( x,y ))( t 1 +1) y 1 =v 1 y ( x,y )+( v 2 y ( x,y ))( t 1 +1)
where v i x,y is a motion vector map for an i motion, and where F,B are a time referencing and boundary model, respectively.
12 . The method of claim 11 , wherein said boundary model is a smooth function of said pixel positions and has a form:
s=B ( x,y )= gx+hy+αx 2 +βy 2 +i
where {g,h,α,β,i} are parameters of the boundary model.
13 . The method of claim 11 , where said motion model for each motion may be any standard model.
14 . The method of claim 13 wherein said any standard model is an affine model having a form:
{right arrow over (v)} i ( x,y )=( a i x +b i y+c i ,d i x+e i y+f i )
where {a i ,b i ,c i ,d i ,e i ,f i } are parameters of the affine model.
15 . The method of claim 9 wherein said motion model has a form for extraction of more than 2 motions with non-intersecting boundaries of:
x
→
′
=
∑
i
=
1
M
g
i
(
t
′
)
g
i
(
t
i
ref
)
v
→
i
(
x
,
y
)
g
i
≠
1
(
t
′
)
=
∏
j
=
1
,
,
(
j
≠
i
)
M
(
t
′
-
t
j
ref
)
g
1
(
t
′
)
=
1
t
′
=
F
(
{
s
j
}
)
s
j
=
B
j
(
x
,
y
)
where M is the number of motions in said regions, and {t i ref } label time values of said set of reference images used; for example t i ref =−1 (past), t i ref =0 (future), t i ref =−2 (2 frames deep in past), etc., and equations s i =B j (x,y) are boundary models for j=1,2, . . . M−1 boundaries.
16 . The method of claim 15 , wherein said boundary model has smooth functions of pixel coordinates of a form:
s j =B j ( x,y )= g j x+h j y+α j x 2 +β j y 2 +i j
where {g j ,h j ,α j ,β j ,i j } are parameters of the j boundary model.
17 . The method of claim 15 , wherein said motion model for each motion may be any standard model
18 . The method of claim 17 wherein said any standard model is an affine model having a form of:
{right arrow over (v)} i ( x,y )=( a i x+b i y+c i ,d i x+e i y+f i )
where {a i ,b i ,c i ,d i ,e i ,f i } are parameters of the affine model.
19 . The method of claim 15 , wherein said time variable for more than 2 motions is of a form:
t 1 =F ({ s j },{w j },{t i ref })
where {w j } are a set of width parameters for said one or more associated moving boundaries.
20 . The method of claim 11 , wherein a nonlinear time referencing equation for extraction of 2 motions has a form:
t
′
=
F
(
s
)
=
(
tanh
(
(
s
+
0.5
)
/
w
)
-
1
)
2
where w is a width parameter for the moving boundary.
21 . The method of claim 19 , wherein said parameters {w j }, which control widths of said one or more associated boundaries, itself varies dynamically in said motion model, such that a system may select the best boundary widths {w j } according to a minimum prediction error.
22 . The method of claim 19 , wherein said parameters {w j } are initially fixed and small during a first estimation of parameters for said motion model, and then successively reduced in re-estimation stages of said parameters for said motion model.
23 . The method of claim 11 , wherein said motion and boundary/interface model parameters characterizing the quantities {{right arrow over (v)} i ,B,F} for the said model are determined from a steepest descent algorithm to minimize prediction error, and use multiple resolution layers and projection of said parameters from one layer to a next layer.
24 . The method of claim 15 , wherein said motion and boundary/interface model parameters characterizing the quantities {{right arrow over (v)} i ,B i ,F}for the said model are determined from a steepest descent algorithm to minimize prediction error, and use multiple resolution layers and projection of said parameters from one layer to a next layer.
25 . The method of claim 4 wherein said motion model has a form for 2 motions of:
t 1 =F ( s ) s=B ( x,y ) x 1 =v 1 x ( x,y )+( v 2 x ( x,y ))( t 1 +1) y 1 =v 1 y ( x,y )+( v 2 y ( x,y ))( t 1 +1)
where {right arrow over (v)} i x,y is a motion vector map for an i motion, and where F,B are the time referencing and boundary model, respectively.
26 . The method of claim 25 , wherein said boundary model is a smooth function of said pixel positions and has a form:
s=B ( x,y )= gx+hy+α 2 +βy 2 +i
where {g,h,α,β,i} are parameters of the boundary model.
27 . The method of claim 25 , where said motion model for each motion may be any standard model.
28 . The method of claim 27 wherein said any standard model is an affine model having a form:
{right arrow over (v)} i ( x,y )=( a i x+b i y+c i ,d i x+e i y+f i )
where {a i ,b i ,c i ,d i ,e i ,f i } are parameters of the affine model.
29 . The method of claim 4 wherein said motion model has a form for extraction of more than 2 motions with non-intersecting boundaries of:
x
→
′
=
∑
i
=
1
M
g
i
(
t
′
)
g
i
(
t
i
ref
)
v
→
i
(
x
,
y
)
g
i
≠
1
(
t
′
)
=
∏
j
=
1
,
…
,
(
j
≠
i
)
M
(
t
′
-
t
j
ref
)
g
1
(
t
′
)
=
1
t
′
=
F
(
{
s
j
}
)
s
j
=
B
j
(
x
,
y
)
where M is the number of motions in said regions, and {t i ref } label time values of said set of reference images used; for example t i ref =−1 (past), t i ref =0 (future), t i ref =−2 (2 frames deep in past), etc., and equations s j B j (x,y) are boundary models for j=1,2, . . . M−1 boundaries, and F({s j }) is the nonlinear time-referencing equation.
30 . The method of claim 29 , wherein said boundary model has smooth functions of pixel coordinates of a form:
s j =B j ( x,y )= g j x+h j y +α i x 2 +β j y 2 +i j
where {g j ,h j ,α j ,β j ,f j } are parameters of the j boundary model.
31 . The method of claim 29 , wherein said motion model for each motion may be any standard model.
32 . The method of claim 31 wherein said any standard model is an affine model having a form of:
{right arrow over (v)} i ( x,y )=( a i x+b i y+c i ,d i x+e i y+f i )
where {a i ,b i ,c i ,d i ,e i ,f i } are parameters of the affine model.
33 . The method of claim 29 , wherein said time variable for more than 2 motions is of a form:
t 1 =F ({ s j },{w j },{t i ref })
where {w j } are a set of width parameters for said one or more associated moving boundaries.
34 . The method of claim 25 , wherein a nonlinear time referencing equation for extraction of 2 motions has a form:
t
′
=
F
(
s
)
=
(
tanh
(
(
s
+
0.5
)
/
w
)
-
1
)
2
where w is a width parameter for the moving boundary.
35 . The method of claim 33 , wherein said parameters {w j }, which control widths of said one or more associated moving boundaries, itself varies dynamically in said motion model, such that a system may select a best boundary width w according to a minimum prediction error.
36 . The method of claim 33 , wherein said parameters {w j } are initially fixed and small during a first estimation of parameters for said motion model, and then successively reduced in re-estimation stages of said parameters for said motion model.
37 . The method of claim 25 , wherein said motion and boundary/interface model parameters characterizing the quantities {{right arrow over (v)} i ,B,F} for the said model are determined from a steepest descent algorithm to minimize prediction error, and use multiple resolution layers and projection of said parameters from one layer to a next layer.
38 . The method of claim 29 , wherein said motion and boundary/interface model parameters characterizing the quantities {{right arrow over (v)} j ,B j ,F}for the said model are determined from a steepest descent algorithm to minimize prediction error, and use multiple resolution layers and projection of said parameters from one layer to a next layer.
39 . The method of claim 2 wherein the number of said extracted one or more motions is no greater than the number of reference image frames.
40 . The method of claim 1 wherein the number of said extracted one or more associated moving boundaries is no greater than the number of said set of reference image frames.
41 . A processing system comprising a processor coupled to a memory, which when executing a set of instructions performs the method of claim 1 .
42 . A machine-readable medium having stored thereon instructions, which when executed performs the method of claim 1 .
43 . An apparatus for motion estimation comprising:
means for receiving an input image; means for receiving a set of reference images; means for partitioning said input image into regions; and for each region:
means for extracting one or more motions; and
means for extracting one or more associated moving boundaries.
44 . The apparatus of claim 43 wherein said set of reference images is selected from the group consisting of one or more past frames, one or more future frames, and one or more past frames and one or more future frames.
45 . The apparatus of claim 44 wherein said means for extracting one or more motions and said means for extracting one or more associated moving boundaries involves a multiple motion model with a nonlinear coupling between space and time variables, and a boundary model directly coupled to said one or more motions.
46 . The method of claim 45 wherein said motion model has a form for 2 motions of:
t=F ( s ) s=B( x,y ) x 1 =v 1 x ( x,y )+( v 2 x ( x,y ))( t 1 +1) y 1 v 1 y ( x,y )+( v 2 y ( x,y ))( t 1 +1)
where v i x,y is a motion vector map for an i motion, and where F,B are a time referencing and boundary model, respectively.
47 . The method of claim 45 wherein said motion model has a form for extraction of more than 2 motions with non-intersecting boundaries of:
x
→
l
=
∑
i
=
1
m
g
i
(
t
l
)
g
i
(
t
i
ref
)
v
→
i
(
x
,
y
)
g
i
≠
1
(
t
l
)
=
∏
j
=
1
,
,
M
(
j
≠
i
)
(
t
l
-
t
j
ref
)
g
1
(
t
l
)
=
1
t
l
=
F
(
{
s
j
}
)
s
j
=
B
j
(
x
,
y
)
where M is the number of motions in said regions, and {t i ref } label time values of said set of reference images used; for example t 1 ref =−1 (past), t 1 ref =0 (future), t 1 ref =−2 (2 frames deep in past), etc., and equations s j ==B j (x,y) are boundary models for j=1,2, . . . M−1 boundaries, and F({s j }) is a nonlinear time-referencing equation.
48 . A machine-readable medium having stored thereon information representing the apparatus of claim 43 .
49 . An apparatus for improvement of a standard motion segmentation comprising:
means for receiving an input image; means for receiving a set of reference images; means for partitioning said input image into regions; and for each region:
means for extracting one or more motions; and
means for extracting one or more associated moving boundaries.
50 . The apparatus of claim 49 wherein said set of reference images is selected from the group consisting of one or more past frames, one or more future frames, and one or more past frames and one or more future frames.
51 . The apparatus of claim 50 wherein said means for extracting one or more motions and said means for extracting one or more associated moving boundaries involves a multiple motion model with a nonlinear coupling between space and time variables, and a boundary model directly coupled to said one or more motions.
52 . The method of claim 51 wherein said motion model has a form for 2 motions of:
t 1 =F ( s ) s=B ( x,y ) x 1 =v 1 x ( x,y )+( v 2 x ( x,y ))( t 1 +1) y 1 =v 1 y ( x,y )+( v 2 y ( x,y ))( t 1 +1)
where v i x,y is a motion vector map for an i motion, and where F,B are a time referencing and boundary model, respectively.
53 . The method of claim 51 wherein said motion model has a form for extraction of more than 2 motions with non-intersecting boundaries of:
x
→
l
=
∑
i
=
1
m
g
i
(
t
l
)
g
i
(
t
i
ref
)
v
→
i
(
x
,
y
)
g
i
≠
1
(
t
l
)
=
∏
j
=
1
,
,
M
(
j
≠
i
)
(
t
l
-
t
j
ref
)
g
1
(
t
l
)
=
1
t
l
=
F
(
{
s
j
}
)
s
j
=
B
j
(
x
,
y
)
where M is the number of motions in said regions, and {t i ref } label time values of said set of reference images used; for example t i ref =−1 (past), t i ref =0 (future), t i ref =−2 (2 frames deep in past) etc., and equations s j =B j (x,y) are boundary models for j=1,2, . . . , M−1 boundaries, and F({s j }) is a nonlinear time-referencing equation.
54 . A machine-readable medium having stored thereon information representing the apparatus of claim 49 .
55 . An apparatus for preprocessing an image for a non-rigid boundary tracker comprising:
means for receiving an input image; means for receiving a set of reference images; means for partitioning said input image into regions; and for each region:
means for extracting one or more motions; and
means for extracting one or more associated moving boundaries.
56 . The apparatus of claim 55 wherein said set of reference images is selected from the group consisting of one or more past frames, one or more future frames, and one or more past frames and one or more future frames.
57 . The apparatus of claim 56 wherein said means for extracting one or more motions and said means for extracting one or more associated moving boundaries involves a motion model with a nonlinear coupling between space and time variables, and a boundary model directly coupled to said one or more motions.
58 . The method of claim 57 wherein said motion model has a form for 2 motions of:
t 1 =F ( s ) s=B ( x,y ) x 1 =v 1 x ( x,y )+( v 2 x ( x,y ))( t 1 +1) y 1 =v 1 y ( x,y )+( v 2 y ( x,y ))( t 1 +1)
where v i x,y is a motion vector map for an i motion, and where F,B are a time referencing and boundary model, respectively.
59 . The method of claim 57 wherein said motion model has a form for extraction of more than 2 motions with non-intersecting boundaries of:
x
→
l
=
∑
i
=
1
m
g
i
(
t
l
)
g
i
(
t
i
ref
)
v
→
i
(
x
,
y
)
g
i
≠
1
(
t
l
)
=
∏
j
=
1
,
,
M
(
j
≠
i
)
(
t
l
-
t
j
ref
)
g
1
(
t
l
)
=
1
t
l
=
F
(
{
s
j
}
)
s
j
=
B
j
(
x
,
y
)
where M is the number of motions in said regions, and {t i ref } label time values of said set of reference images used; for example t i ref =−1 (past), t i ref =0 (future), t i ref =−2 (2 frames deep in past), etc., and equations s j =B j (x,y) are boundary models for j=1,2, . . . M−1 boundaries, and F({s j }) is a nonlinear time-referencing equation.
60 . A machine-readable medium having stored thereon information representing the apparatus of claim 55.Join the waitlist — get patent alerts
Track US2004091047A1 — get alerts on status changes and closely related new filings.
We store only your email — no account needed. See our privacy policy.