Kanade Lucas Tomasi Tracker

Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Kanade
Kanade
-
-
Lucas
Lucas
-
-
Tomasi
Tomasi
(KLT)
(KLT)
Feature Tracker
Feature Tracker
Computer Vision Lab.
Computer Vision Lab.
Jae Kyu Suhr
Jae Kyu Suhr
Introduction
Introduction
Motivation Motivation
Image sequence
Feature point detection
Feature point tracking
Tracking result
3D Reconstruction results 3D Reconstruction results
Problem Statement
Problem Statement
Problem Statement Problem Statement
I, J : 1
st
and 2
nd
grayscaledimages
I(x,y) : gray value of I at [x y]
T
Let u=[u
x
u
y
]
T
be a point on the 1
st
image I
The goal is to find v on J , where I(u) and J (v) are
similar. (v= u+d = [u
x
+d
x
u
y
+d
y
]
T
)
d=[d
x
d
y
]
T
is the image velocity at u
or the optical flow at u.
Problem Statement Problem Statement
Definition of the residual function (d).

x
and
y
integration window size parameter
integration window size (2
x
+1)(2
y
+1).
(u
x
, u
y
)
w
x
w
y
I J
( - )
2
Pyramid Implementation
Pyramid Implementation
Why pyramid representation? Why pyramid representation?
Standard KLT algorithm can deal with small pixel displacement.
Solution for this is a pyramidal implementation.
Let I
0
=I be the 0
th
level image
The pyramid representation is built recursively.
I
0
I
1
I
2
I
3
I
Lm
(Lm: 2~4)
I
L-1
: the image at level L-1
I
L
: the image at level L
Pyramidal feature tracking Pyramidal feature tracking
A given point u in I, find its corresponding location v=u+d.
Corresponding point of u(u
0
) on the pyramidal image I
L
is u
L
Simple overall pyramid tracking algorithm
d
Lm
is computed at the pyramid level L
m
d
Lm-1
is computed
with an initial guess of d
Lm
at L
m-1
This continues up to the level 0
Let g
L
=[g
L
x
g
L
y
]
T
be an initial guess at level L
( g
L
is available from level L
m
to level L+1 )
Residual pixel displacement vector d
L
=[d
L
x
d
L
y
]
T
that minimizes the new image matching error function
L
Window size (2
x
+1)(2
y
+1) is constant for all pyramid
g
L
is used to pre-translate the image patch in 2
nd
image J
d
L
is small and therefore easy to compute using KLT algorithm
Assume that d
L
is computed, new initial guess g
L-1
at level L-1
Initial guess for the deepest level L
m
The final optical flow solution d
Standard KLT algorithm
Standard KLT algorithm
Standard KLT optical flow computation Standard KLT optical flow computation
Goal is to find d
L
that minimizes the matching function
L
.
Same operation is performed for all levels L, drop the superscript L
and define the new images A and B
Let us change the displacement vector and the point vector
Goal is to find that minimizes the matching function
To find the optimum
Let us substitute by its 1
st
order Taylor expansion
about the point
frame difference Image gradient
( G must be invertible).
Standard LK is valid only the pixel displacement is small.
(Because of the first order Taylor approximation)
Therefore, iterative version of LK is necessary.
Iterative KLT algorithm
Iterative KLT algorithm
Iterative KLT optical flow computation Iterative KLT optical flow computation
k: iteration index
: initial guess from the previous iteration 1,2,,k-1

B
k
: new translated image according to
The goal is to compute that minimize
One step LK optical flow computation is
I
x
, I
y
are computed only once at the beginning of the iteration.
G (22matrix) remains constant throughout the iteration loop.
Only I
k
needs to be recomputed at each iteration
Once is computed, a new pixel displacement guess
The iteration goes on until is smaller than a threshold
or reached at the maximum number of iteration.
(5 are enough to reach convergence)
At 1
st
iteration, the initial guess
Assuming that K iterations are necessary to reach convergence,
the final solution for the optical flow vector
This overall procedure is repeated at all levels L-1, L-2, , 0
Feature selection Feature selection
Goal is to find the location of v on J corresponding to u on I.
How to detect u on I
Find the point u whose G matrix is non-singular
The minimum eigenvalueof G must larger than a threshold.
1. Compute the G matrix and its minimum eigenvalue
m
at every pixel
in the image I.
2. Call
max
the maximum value of
m
over the whole image.
3. Retain the image pixels that have a
m
value larger than a threshold.
4. Retain the local maximum pixels (a pixel is kept if its
m
value is
larger than that of any other pixel in its 33 neighborhood).
5. Keep the subset of those pixels so that the minimum distance
between any pair of pixels is larger than a given threshold distance.
Declaring a feature Declaring a feature Lost Lost
A feature point falls outside of the image.
An image patch around the tracked point varies too much
between image I and image J .
(A cost function (SSD) is larger than a threshold).
Flow chart Flow chart
Input 1
st
image
Select feature points
Input 2
nd
image
Track feature points
Delete lost feature points
Save feature points
Save tracked points
Replace 1
st
image with 2
nd
image
Replace 2
st
image with next image
Select feature points
to replace lost feature points
Save feature points
References References
Bruce D. Lucas and Takeo Kanade. An Iterative Image Registration
Technique with an Application to Stereo Vision. IJ CAI, pages 674-679,
1981.
J . Shi and C. Tomasi, "Good Features to Track," CVPR'94
J ean-Yves Bouguet, Pyramidal Implementation of the Lucas Kanade
Feature Tracker Description of the algorithm, Intel Corporation
Microprocessor Research Labs.
http://www.ces.clemson.edu/~stb/klt/ C++code
KLT: An Implementation of the Kanade-Lucas-Tomasi Feature Tracker
OpenCV: CalcOpticalFlowLK, CalcOpticalFlowPyrLK
http://vision.ucla.edu//MASKS/labs.html Matlabcode
An Invitation to 3D Vision
Thank you
Thank you
Q & A
Q & A

Kanade Lucas Tomasi Tracker

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Kanade Lucas Tomasi Tracker

Uploaded by

Copyright:

Available Formats

Computer Vision (EEE6503) Fall 2009, Yonsei Univ.

: initial guess from the previous iteration 1,2,,k-1

You might also like