Tracking with 3D Data

M. Usman Butt

10 July, 2012
Department of Electrical and Computer Engineering

Introduction
Digital video cameras
o Single camera
o Multiple cameras
Stereo cameras

3D information
Segmentation
Illumination
Shadow
Occlusion

More than 2 cameras

Objectives
To propose a novel solution for robust object
detection and tracking using 3D (depth) information
and other features, if necessary, such as texture,
edges etc.
To design and develop a program implementing the
scheme.
To investigate and identify the problems for realtime implementation of the scheme for object
detection and tracking.
To develop the scheme on an FPGA or a GPU
platform for real-time implementation.
To evaluate and validate the performance of the
system by comparing with existing state-of-the-art
systems.

Previous Work
Stereo with:
o Intensity and binary template
(Beymer et al., Frame-rate, 1999)

o Colour and pattern

(Darell et al., IJCV, 2000)

o Colour and motion

(Yang et al., IJIST, 2005)

o Intensity, 3D world point clouds (plan view)

(Bahadori et al., Appl. Intelligence, 2007)

o Sparse set of feature points and kernel-based

clustering
(Cai et al., Pattern Recognition, 2010)

My Approach
Only 3D information from a high resolution disparity map
generated by implementation of Symmetric Dynamic
Programming Stereo (SDPS) algorithm on FPGA
Object detection from SDPS disparity map
Tracking using the Kalman filter
Assumptions
A suitable background disparity map is available
Objects of interest can be any rigid (e.g. vehicles) or non-rigid (e.g.
human) object, but object sizes must lie in a pre-defined range
Each tracked object first appears as an isolated object and moves
upright
Object speed is less than 10 m s-1 (Olympic sprinter) for human
tracking
The ground plane is basically flat and fixed
Cameras are tilted at fixed angle to the ground plane

Symmetric Dynamic Programming

Stereo (SDPS)
Matches scan line by scan line
Produces disparity map seen by a
Cyclopan eye
Has a nice mathematical property
o Any change in disparity along a
scan line is accompanied by one
monocular point for each unit
change in disparity
d d+x: d+1, d+2, , d+x are ML or MR pixels

Objects separated from their

background are outlined with bands
of ML or MR points

Symmetric Dynamic Programming

Stereo (SDPS)
An objects 3D structure is represented by several
contours
A key problem
o Determine the contour best outlining the object
d-x

Sphere

Major Drawback
o Streaking artefacts

Disparity map

Contours

Stereo Hardware System

SDPS algorithm implemented on FPGA as well
as GPU
1 Mpixels at 30 fps
Disparity range 0-127

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

SDPS
Algorithm
(FPGA or
GPU)

Disp
map

Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Residual
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
k

Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Predicted
disp map

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Background subtraction from

disparity map

p(x,y) = 0, if p(x,y) <= pref(x,y)

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Residual
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
k

Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Predicted
disp map

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Predicted
disp map

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Predicted
disp map

Object Detection and Tracking

Morphological Open Operation(MOO)
Before MOO

After MOO

MOO

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Predicted
disp map

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Predicted
disp map

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Predicted
disp map

Object Detection Algorithm

Compare the candidate contours at adjacent
disparities and determine the binocularly visible true
contours representing the object outline using rules:
If the centroid of Bd,j, lies inside Bd-1,k, then Bd,j is
enclosed by Bd-1,k, then
If the heights of both the contours are same within
the tolerance, (hd,j hd-1,k < h) and
The internal contour area ratio, Ad,j/Ad-1,k > 0.9, then
Bd,j is an object outline
Retain the enclosed contours as 3D structure of the
object

d-x

Contours

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

Predicted
disp map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Tracki
ng

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

Predicted
disp map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Tracki
ng

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Convert BV
o
contour
n
s
contours &
c
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

Predicted
disp map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Tracki
ng

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Contour
Convert BV
o
Templat
n
contours &
c
e
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

Predicted
disp map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Tracki
ng

Object Detection Algorithm

Detection of objects already in the
scene
FOR EACH OBJECT IN PREV. FRAME
Determine search window size for best match of object template
Search in the window around the predicted position in disparity map
FOR p = dk- d to dk+ d
FOR uk = uk- u to uk+ u
FOR vk = vk- v to vk+ v
vis = count pix in disp map with disparity <= p+dmax
occ= count pix in disp map with disparity > p+dmax
Save the best match (vis+occ) position
IF f (vis/(vis+occ)) >= matching_thresh1, then
Template matched, update the template
ELSE
IF f > matching_thresh2 then,
Template partially matched (occluded)
Amalgamate vis and occ region to update the template
ELSE
Preserve the template
Convert all (u, v, d) (X, Y, Z) object location, update the KF parameters

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Contour
Convert BV
o
Templat
n
contours &
c
e
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

Predicted
disp map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Tracki
ng

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Contour
Convert BV
o
Templat
n
contours &
c
e
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

Predicted
disp map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Tracki
ng

Object Detection and Tracking

For each frame k=
1,2,3,
Ref Disp
map

CFo
V

Rcamera

R
image

L
image
Lcamera

Disp
map
SDPS
Algorithm
(FPGA)
Occl
map

Threshold from ds to
dmin

Calculate Boundin List of

g Box candidat
contour
List
of
MOO
e
Thresholdi
attributes
Determine
contours
Background
&
contours
ng disp
&
BV (object)
=1
subtraction
Contour
map
Select the
contours
Binary
Residual > 1
Generation
candidate
Image
disp
contours
map
3D
Object
model
List of BV
Residual
Calibration
contours
Convert
disp
data
into 2D
map
model
Detection of isolated and new
Residual
objects
disp
map
RW
Dimensi
on
Match
c
Contour
Convert BV
o
Templat
n
contours &
c
e
List of
a
attributes
t
Matched
into 3D (RW)
List of BV
BV
contours
RW
in RW
contours
Positio
n
Detection existing
k

objects
Initializ
e&
update
position
Kalman
Filter

List of
Predicted
contours
in RW

Construct
Predicted
(Translated
Objects) disp
map

Predicted
disp map

List of
Predicted
contours in
prev frame
(k-1)
Previo
us list
=
current
list

Tracki
ng

Tracking
Kalman Filter predicts the object position in the next frame
Adequate for simple linear model
Faster computationally simple
Predicted data helps to
o Identify heads cut off due to SDPS streaks
o Find missing objects (occluded or buried in noise) in the frame

Search in a small area around the predicted position for best match with
the template

Reference Tracking Point

Centroid (Xc, Yc, Zc)
o An obvious choice but, for articulated objects
(person), centroid shifts as the object deforms
Feet (Xf, Yf, Zf)
o Ground contact point changes significantly as
people walk
o Usually, feet cut off by background subtraction
Head (Xh, Yh, Zh)
o Almost always visible

Reference Point Choice

X,m

Y,m

Z,m

Centroi
d

0.04

0.09

0.18

Feet

0.08

0.07

0.15

Head

0.05

0.04

0.14

Ref. Tracking Point

(Xc, Yh, Zh)

Results
Scenario A

Results
Scenario A
Obsvd
Track

Pred
Track

X = 0.04

Y = 0.04

Z = 0.18

Results
Scenario B

Results
Scenario B
XA = 0.04
XB = 0.06

YA = 0.02
YB = 0.04

ZA = 0.12
ZB = 0.16

Results
Scenario C

Results
Scenario C
-2

A(Pred)

B(Pred)

A(X)

A(Pred_X)

B(X)

B(Pred_X)

-1

C(Pred)

0.2
0.1

Deviation(X) m 0.0

0
Xm

-0.1

-0.2

3
2
2

Zm

a)

A(Y)
B(Y)
C(Y)

A(Pred_Y)
B(Pred_Y)
C(Pred_Y)

Y m1

d
)B

Deviation(Y)

1
0

Frame #
-2

-2

-1

-1

0
Xm

30
25

A(Z )

b)

B(Z )

Deviation(Z)

Z m

15
10

Frame #

c)

0.2
0.1
m 0.0
-0.1
-0.2

e)

B(Pred_Z )

20

1.0
0.5
m 0.0
-0.5
-1.0

Frame #

f)

XA = 0.03

30

XB = 0.01
XC= 0.01

20
10

A
B
C

g)
Y Devition
15 YA = 0.03
YB = 0.03
10

XC = 0.03 B
C

0
-0.6-0.4-0.2 0.0 0.2 0.4 0.6

Frame #
A(Pred_Z )

X Devition
40

0
-0.6-0.4-0.2 0.0 0.2 0.4 0.6

Frame #

Frame #

h)
Z Devition
4

ZA = 0.19

ZB = 0.16 A

2
1

ZC = 0.13

B
C

0
-0.6-0.4-0.2 0.0 0.2 0.4 0.6

j)

Results

Results
Scenario D

Results
Scenario D
X = 0.04

Y = 0.03

Z = 0.18

Results
A Further Check For Precision
Height (Y),m
Actual

Measured

Subject A

1.70

1.64

0.05

Subject B

1.73

1.7

0.04

Subject C

1.77

1.59

0.21

Subject D

1.56

1.52

0.03

Results
Performance
Time, s
Background
Detection
subtraction
Scenario A
(Single
Person)

Scenario B
(Two People)

Scenario C
(Three People)

Tracking
(KF)

Total

0.4

11

0.03

12

0.5

32

0.08

33

0.6

56

0.12

57

Research Progress to Date

Designed and developed isolated object detection and tracking technique
from depth map
Developed and integrated Kalman Filter (KF) into the tracking scheme
Acquired video sequences for several outdoor scenarios with the help of
CITR group to test the developed technique
Implemented and tested the technique on the video sequences with
isolated objects
Based on analysis of results of isolated object tracking, improved the
detection and tracking technique to handle the occlusion problem
Tested the improved version on the video sequences with the presence of
occlusion
Attended one of the Doctoral Skills Programme Induction Days
Submitted full thesis proposal
Published a paper:
M. U. Butt and J. Morris, Precise Tracking using High Resolution
Real-time Stereo, Image and Vision Computing New Zealand
(IVCNZ2011), Auckland, New Zealand, IVCNZ, 2011, pp. 143-148.

Future Work

Identification of bottlenecks for real-time implementation

Parallelization of the algorithm for real-time implementation
Study of architecture of different hardware platforms (GPU
and FPGA) for suitability of parallelization
Development of algorithm on suitable platform (FPGA or
GPU) for real-time processing
Testing of real-time performance of the algorithm
Further improvement in the algorithm based on result
analysis
Evaluation of the algorithm