Pedesterian and Vehicle Tracking with

Adaptive Background Subtraction and

Adaptive Object Matching By Using Simple
Object Features
zlem Morkaya, Serdar Korukolu
Abstract This study focuses on systems that are tracking and classifying humans and vehicles in the case of occlusion with
the use of adaptive background subtraction and adaptive object matching. Our system uses adaptive background subtraction to
tackle with once moving then stopped objects, added to background; and once stable then moving objects, considered as
foreground. Noise components in frames result of background subtraction are cleared via morphological operations. Adaptive
object matching uses one-to-one or one- to-many matching options consecutively in the case of normal or occlusion states.
Addition to this, adaptive matching provides a clear solution for the problem of object matching before and after occlusion.
Nearest neighboring and size ratio features are basically used as matching criteria. Finally, tracked objects are classified as
pedestrian or vehicle with regard to their object features.
Index Terms Image Processing and Computer Vision, Motion, Scene Analysis, Segmentation, Tracking.

1 INTRODUCTION
UR application tracks and classifies objects into two
distinct classes; pedestrian or vehicle. System tracks
objects and resolves occlusion with the use of adap-
tive background subtraction and adaptive object match-
ing. Our software heavily depends on object tracking
concepts. Object tracking is an appealing study of field
with wide range of use. Automated surveillance, traffic
monitoring, motion based recognition, human computer
interaction [1] are significant usage areas of object track-
ing. Object tracking has close relationship with human
and vehicle tracking. In the literature, there exist many
applications related with human tracking [5], [6], [12],
[14], [15], [16], [21]; vehicle tracking [7], [8], [11], [13], [17],
[19] and traffic monitoring [9], [10], [18], [22]. Although
there are plenty of works in object tracking field, no single
algorithm can find a complete solution for all cases. Be-
cause object tracking has challenging issues such as noise,
complex object motion, complex object shapes, partial and
full occlusions of objects and illumination changes [1].
[29] emphasizes that object tracking has four main steps,
that can be listed as detection, tracking, object representa-
tion and recognize object behavior. [1] separates object
tracking into three steps: These are detection of interest-
ing moving objects, tracking of objects from frame to
frame, and analysis of object tracks to recognize their be-
havior. In this paper, motion detection and object tracking
steps of video analysis will be argued.
Motion detection is a prior step for object tracking. Ex-
tracted regions outputs of motion detection become in-
puts for tracking. Background model of the video frames
is built by motion detection phase. Success of object track-
ing especially/mainly depends on the success of the
background modeling. [20] classifies commonly used
background modeling techniques and partitions them
into two main groups; recursive and non-recursive.
Frame differencing, median filter, linear predictive filter,
non-parametric model are samples of non-recursive me-
thods while approximated median filter, Kalman filter
and mixture of Gaussian filters are samples of recursive
methods. In the literature there exist a number of surveys
related to background subtraction techniques [25], [1],
[28]. Characteristics of a good background model are
listed by [23] as follows:
- Immunity to noise,
- Insensitivity to illumination changes,
- Adaptability to the changes in the initial back-
ground,
- Insensitivity to the motion from clutter,
- Causing no deformations on foreground,
- Need as little post-processing operations as possi-
ble,
- Fast and reliable,
- Easy to implement.

Point detectors, segmentation and background model-
ing and supervised classifiers are listed as methods of
detection of moving objects by [1]. Static camera videos
heavily use background subtraction as a way of back-

- . Morkaya is with the Scientific and Technological Research Council of
Turkey (TBTAK- EEEAG), Kavaklidere, Ankara, 06100, Turkey
- S. Korukoglu is with Computer Engineering Department, Ege University,
Bornova, Izmir,35100, Turkey


O
JOURNAL OF COMPUTING, VOLUME 3, ISSUE 12, DECEMBER 2011, ISSN 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 71

ground modeling. Background subtraction provides a
simple and efficient way of motion detection and it takes
advantage of colors of moving areas being changed and
also colors of stable areas which are static. [2] mentions
that background subtraction is the best compromise be-
tween performance and reliability. Our system also uses
adaptive version of background subtraction to handle
changes of the background more accurately. [26] Com-
pares three dynamic background subtraction methods
and indicates that Selective Update using Temporal Me-
dian method is the best of the three methods. Finally [27]
implements an adaptive and probabilistic method for
background subtraction.
Object tracking step is implemented after detection of
moving regions of video frames. The aims of this step are
to determine the trajectory of moving objects and to as-
sign valid labels to moving regions that provide corres-
pondence of moving objects among video frames [1]. Oc-
clusion handling is a significant object tracking problem
and there exist many applications in the literature to solve
this problem [30], [31], [32], [33], [34]. Our application
tackles with occlusion problem with the use of one-to-one
and one-to-many matching approaches.
This paper is organized as follows: In section 2 adap-
tive background subtraction and in section 3 adaptive
object matching algorithms will be discussed. In section 4
classification method of objects into two classes; pede-
strian or vehicle, will be explained. Our algorithm was
tested with PETS 2000 video but instead of using original
video size (768x576) our algorithm used a smaller size
video (192x144) for the sake of performance requirements.
Finally, in section 5 experimental results and conclusions
of our work will be given.
2 OBJECT DETECTION WITH ADAPTIVE
BACKGROUND SUBTRACTION
Proposed system uses adaptive background subtrac-
tion method to detect moving objects in videos. Back-
ground is continuously updated because of illumination
changes. Another reason of using adaptive background
subtraction method is to handle new objects added to
background. Once moving then stopped objects are add-
ed to the background after three seconds following the
end of their movement. Similarly initially stable then
moving objects are added to foreground. Usage of static
background cannot handle such situations. Fig. 1 shows
an example of the adaptive background. In Fig. 1(a) first
background of the video is illustrated. In Fig. 1.(b) up-
dated version of the first background is shown. In Fig.
1(b) a red circle is drawn around the newly added car to
the background. Red circle emphasizes the change of the
updated background.
Our application uses two backgrounds; the first one is
updated for every pixel regardless of its grayscale value
with a ratio 0.986:0.014. The second background is up-
dated with a ratio 0.98:0.02 only if the difference between
grayscale value of current frame pixel and background
pixel is greater than or equal to 15. These background
updating criteria are formulated in (1) and (2).
Both backgrounds are initialized with median filtering
of frames belonging to first 4 seconds of video. Every
frame has two background difference images. If the dif-
ference between grayscale values of current frame pixel
with the first background is greater than or equal to 15,
then current frame pixel is marked as candidate fore-
ground pixel for the first background. Same threshold is
also used for second background. If objects of these back-
ground difference images have less than 25 pixels, they
are considered as noise and are removed. Closing opera-
tion is applied on both background difference images of
current frame with a matrix of 3x3. After then intersection
(AND operation) of both difference images determines
final foreground pixel of current frame. Finally closing
operation with a matrix of 7x7 and then opening opera-
tion with a matrix of 3x3 are applied on image obtained
by intersection. These closing operations are used with
the purpose of combining separated parts of foreground
object because of thresholding operations during back-
ground subtraction. Aim of using final opening operation
is to remove unnecessary combination of objects because
of previous closing operations.

)
`

+ +
= + -
=
) 014 . 0 ) 1 ( ( ) 986 . 0 ) ( (
) 1 (
1
1 1
1
t P t P
t P P
P
C BG
BG BG
BG
(1)

= + <
+ +
= + >
=
) (
) 1 ( 15
) 02 . 0 ) 1 ( ( ) 98 . 0 ) ( (
) 1 ( 15
2
2 2
2
2 2
2
t P
t P P P
t P t P
t P P P
P
BG
BG C BG
C BG
BG C BG
BG
(2)
3 ADAPTIVE OBJECT MATCHING
This paper proposes adaptive object matching to ac-
complish tracking of objects among video frames. Adap-
tive object matching uses one-to-one and one-to-many
matching approaches. If there exists no occlusion of ob-
jects, one-to-one matching approach is chosen. In the case
of occlusion one-to-many matching approach is used.
Every object that newly enters the video scene is called
main object and a unique number is assigned to it. If the
object in the frame is not assigned as a follower of any
objects in the previous frame then this object is regarded
as a main object. Numbers assigned to main objects start
from one and increased by one as new objects enter the

Fig. 1. (a) First background (b) Updated version of the first background
JOURNAL OF COMPUTING, VOLUME 3, ISSUE 12, DECEMBER 2011, ISSN 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 72

video scene. Proposed system maintains a 3D matrix
named object_relation whose depth equals to a con-
stant number 6. Number of rows of the matrix equals to
count of main objects of the video and number of col-
umns equals to number of video frames. Fig. 2 shows the
structure of object_relation matrix. System assigns a
single row of these matrices for each main object. All fol-
lower objects of main objects among next video frames
are stored in cells of the belonging row of the main object.





Row and column indices determine a depth block con-
taining six cells. These cells store six features of follower
objects. Cells and their contents are explained below.

- The first cell stores the number assigned to follower
object among other objects in the same frame.
- The second cell stores the x coordinate value of the
frame belonging to the centroid point of the follow-
er object.
- The third cell stores the y coordinate value of the
frame belonging to the centroid point of the follow-
er object.
- The fourth cell stores 1 if the follower object is in
the occlusion state or it stores -1 if it is not in the
occlusion state.
- The fifth cell stores width value of the follower ob-
ject.
- The sixth cell stores height value of the follower ob-
ject.

Any moving object which has more than seventy
frames is considered as worth to track so object move-
ments which last less than seventy frames are not tracked.
This is applied because of eliminating movements result-
ing from illumination changes and noise components.
Nearest neighboring is the most commonly used
matching criterion [3]. In addition to nearest neighboring,
our application also uses other criteria to accomplish suc-
cessful matching. These criteria are listed below.

- Distances between centroids of matched objects in
the t. and t+1. frames must be less than twice of the
diagonal of the bounding box of the matched object
in the t. frame.
- In the case of occlusion, same occluded object num-
ber is assigned to both objects which take part of the
occlusion. In this case, one-to-many matching ap-
proach is implemented.
- If the bounding box of the follower object in t+1.
frame touches the edges of the frame then it is as-
sumed that this object left the scene.

When two objects begin to occlude themselves, that is
called merging in the literature [4], [24]. In the merging
state same object is assigned as a follower object by two
objects in the previous frame. Our system uses one-to-
many matching approach to handle occlusion. Occlusion
terminates if two objects are separated from each other
and that is called splitting in the literature [4], [24]. Sys-
tem proposed controls the splitting of occluded objects by
examining the region which is obtained by enlarging ten
pixels from the edges of the bounding box of the occluded
object. If this region contains an extra object rather than





Fig 3. (a) Occluded object in 876.frame (b) Splitted objects in 877.
frame





Fig. 4. (a): Objects of 865. frame (occlusion) (b): Objects of 1230.
frame (occlusion)





Fig. 5. (a): Objects of 875.frame (after occlusion) (b): Objects of
1248.frame (after occlusion)

Fig 2. Structure of object_relation matrice
JOURNAL OF COMPUTING, VOLUME 3, ISSUE 12, DECEMBER 2011, ISSN 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 73

the occluded object then it is decided that occluded object
is split and occlusion of the objects is terminated. Fig. 3
shows splitting of the occluded object. In Fig. 4 objects in
occlusion state, their current paths and bounding boxes
are depicted. When occlusion state terminates system and
it also terminates the use of one-to many matching ap-
proach, instead one-to-one matching concept is applied to
handle splitting phase. Objects before occlusion must be
matched with objects after occlusion. Trajectory directions
of objects before and after occlusion are compared and
objects are matched according to the similarity of their
trajectory directions. In Fig. 5 objects and their paths are
illustrated after completion of the occlusion. Identifica-
tion of bounding boxes and border of moving objects dur-
ing occlusion phase is achieved by interpolating of objects
dimensions before and after occlusion state. Fifth and
sixth cells of depth block of object_relation matrix store
width and height value of objects. This helps to calculate
borders and bounding boxes of objects during occlusion
state. When object tracking phase completes then objects
are classified with the use of simple object features.
4 OBJECT CLASSIFICATION
Significant difference of our algorithm between other
object tracking methods is classification of objects into
two classes; pedesterian or vehicle. As it is seen in Fig. 4,
Fig. 5 and Fig. 6 bounding box of pedesterians are drawn
with red color while bounding boxes of vehicles are
drawn by blue color in addition to that, letter "P" for
pedesterians and letter "V" for vehicles are added to video
frames. Aspect ratio of bounding box (width/height) is
used as a discrimative feature to accomplish classification.
If aspect ratio of bounding box is greater than or equal to
1 then object is classified as a vehicle and if aspect ratio is
smaller than 1 object is classified as a pedesterian. In Fig. 6
results of classification of our application is presented. This
is formulated in (3).


P 1
V 1
<
>
Height
Width
Height
Width
BB
BB
BB
BB
(3)
5 EXPERIMENTAL RESULTS AND CONCLUSION

In Fig. 6 trajectories of moving objects in PETS' 2000
video are depicted. These results overlap with the results
declared by [4], [35]. Our algorithm also classifies vehicles
and pedestrians, and result being obtained by classifica-
tion for PETS' 2000 video is completely accurate. In PETS'
2000 video there exist three pedestrians and three ve-
hicles. Results being calculated by our algorithm are same
and there is no false positive.
Our application achieves tracking and classification of
moving objects even if occlusion of objects exists. Our
algorithm uses adaptive background subtraction method
to detect moving regions and adaptive object matching to
track objects among video frames. Our application is not
only an object tracker but it also classifies object using
simple object features.
This software can be very beneficial if it is used as a
prior phase of a driver assistance system or a traffic rules
obedience control system. Our future target is to develop
a tool which controls obedience of traffic rules. If such a
system is implemented, this application will be a basic
part of the system. Traffic rules obedience will be con-
trolled with regard to types of moving objects in the traf-
fic. As a result classification of moving objects in traffic is
a must for such systems.

REFERENCES
[1] Yilmaz, A., Javed, O., and Shah, M. 2006. Object tracking: A survey.
ACM Comput. Surv. 38, 4, Article 13 (Dec. 2006), 45 pages. DOI =
10.1145/1177352.1177355
http://doi.acm.org/10.1145/1177352.1177355
[2] P. Spagnolo, T.D Orazio *, M. Leo, A. Distante, Moving object segmen-
tation by background subtraction and temporal analysis, Image and Vi-
sion Computing, 2006
[3] Y. Bar-Shalom, T.E. Forthmann, Tracking and data association. In Ma-












Fig. 6. (a): (a) First vehicle (300. frame), (b) Second vehicle (650.
frame), (c) Third vehicle (950. frame), (d) First pedesterain 1380.
frame, (e) Second pedesterian (1050. frame), (f) Third pedesterian
(1453. frame)
JOURNAL OF COMPUTING, VOLUME 3, ISSUE 12, DECEMBER 2011, ISSN 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 74

thematics in Science and Engineering, volume 179. Academic Press Inc.,
1988.
[4] Cina Motamed Motion detection and tracking using belief indicators
for an automatic visual-surveillance system, Image Vision and Compu-
ting June, 2005
[5] Tao Zhao, Ram Nevatia IEEE Transactions on Pattern Analysis and
Machine Intelligence, Tracking Multiple Humans in Complex Situa-
tions, Vol 26, No 9, September 2004, p 1208-1221
[6] Tao Zhao, Ram Nevatia, Bo Wu IEEE Transactions on Pattern Analysis
and Machine Intelligence, Segmentation and Tracking of Multiple Hu-
mans in Crowded Environments, Vol 30, No 7, July 2008, p 1198-1211
[7] Jos Melo, Andrew Naftel, Alexandre Bernardino, Jos Santos-Victor,
Detection and Classification of Highway Lanes Using Vehicle Motion
Trajectories, IEEE Transactions on Intelligent Transportation Systems,
Vol. 7 ,No. 2 June 2006 p 188-200
[8] Lingqun Wang, Shizhu Pan, Yingping Zheng, Moving Vehicle Track-
ing Based on Unscented Kalman Filter Algorithm, 2009 World Con-
gress on Computer Science and Information Engineering
[9] P.F. Alcantarilla, M.A. Sotelo, L.M. Bergasa Automatic Daytime Road
Traffic Control and Monitoring System, Proceedings of the 11th Inter-
national IEEE Conference on Intelligent Transportation Systems Beijing,
China, October 12-15, 2008
[10] Fan-feng Meng, Zhen-shen Qu, Qing-shuang Zeng, Li Li, Traffic Object
Tracking Based on Increased-step Motion History Image, Proceedings
of the IEEE International Conference on Automation and Logistics Au-
gust 18 - 21, 2007, Jinan, China
[11] Changsoo Jeong, Alice C. Parker, 3D Tree-Structured Object Tracking
for Autonomous Ground Vehicles, Fourth Canadian Conference on
Computer and Robot Vision(CRV'07),2007
[12] Prahlad Kilambi, Evan Ribnick, Ajay J. Joshi, Osama Masoud, Nikolaos
Papanikolopoulos, Estimating pedestrian counts in groups, Computer
Vision and Image Understanding 110 (2008) 4359
[13] Roya Rad, Mansour Jamzad, Real time classification and tracking of
multiple vehicles in highways, Pattern Recognition Letters 26 (2005)
15971607, Elsevier
[14] R. Bodor, A. Drenner, D. Fehr, O. Masoud, N. Papanikolopoulos, View-
independent human motion classification using image-based recon-
struction, Image and Vision Computing 27 (2009) 11941206
[15] Dirk Ormoneit, Michael J. Black, Trevor Hastie, Hedvig Kjellstrm,
Representing cyclic human motion using functional analysis, Image
and Vision Computing 23 (2005) 12641276
[16] Wei-Lwun Lu, Kenji Okuma, James J. Little Tracking and recognizing
actions of multiple hockey players using the boosted particle filter, Im-
age and Vision Computing 27 (2009) 189205
[17] Fatemeh Karimi Nejadasl, Ben G.H. Gorte, Serge P. Hoogendoorn,
Optical flow based vehicle tracking strengthened by statistical deci-
sions, ISPRS Journal of Photogrammetry & Remote Sensing 61 (2006)
159169
[18] Peter Reinartz , Marie Lachaise, Elisabeth Schmeer, Thomas Krauss,
Hartmut Runge, Traffic monitoring with serial images from airborne
cameras, ISPRS Journal of Photogrammetry & Remote Sensing 61
(2006) 149158
[19] Derek R. Magee Tracking multiple vehicles using foreground, back-
ground and motion models, Image and Vision Computing 22 (2004)
143155
[20] Sen-Ching S.Cheung, Chandrika Kamath Robust techniques for back-
ground subtraction in urban traffic video, Visual Communications and
Image Processing 2004. Proceedings of the SPIE, Volume 5308, pp. 881-
892 (2004).
[21] Ismail Haritaoglu, David Harwood, Larry S. Davis, W4: Real-Time
Surveillance of People and Their Activities, IEEE, August, 2000
[22] Jen-Chao Tai*, Shung-Tsang Tseng, Ching-Po Lin, Kai-Tai Song, Real-
time image tracking for automatic traffic monitoring and enforcement
applications, Image and Vision Computing 22 (2004) 485501
[23] Mustafa Oral, Umut Deniz, Centre of mass model A novel approach
to background modelling for segmentation of moving objects, Image
and Vision Computing 25 (2007) 13651376
[24] Lin Zhu, Jie Zhou, Jingyan Song, Tracking Multiple objects through
occlusion with online sampling and position estimation, Pattern Recog-
nition 41 (2008) 2447 2460
[25] McIvor A. M.,Background subtraction techniques, In Proc. of Image
and Vision Computing, Auckland, New Zealand, 2000.",
[26] Bijan Shoushtarian, Helmut E. Bez, A practical adaptive approach for
dynamic background subtraction using an invariant colour model and
object tracking, Pattern Recognition Letters 26 (2005) 526
[27] Chris Stauffer W.E.L Grimson, Adaptive background mixture models
for real-time tracking, Proc. of CVPR 1999, pp. 246-252.
[28] Donovan H. Parks and Sidney S. Fels, Evaluation of Background Sub-
traction Algorithms with Post-processing, Proceedings of the 2008 IEEE
Fifth International Conference on Advanced Video and Signal Based
Surveillance Pages 192-199, 2008
[29] Zalili Musa and Junzo Watada, Video Tracking System: A Survey ,
ICIC Express Letters ICIC International, Volume 2, Number 1, March
2008 pp. 65-72,
[30] Andrew Senior*, Arun Hampapur, Ying-Li Tian, Lisa Brown, Sharath
Pankanti, Ruud Bolle, Appearance models for occlusion handling, Im-
age and Vision Computing 24 (2006) 12331243
[31] Eduardo Parrilla_, Damian Ginestar, Jose Luis Hueso, Jaime Riera,
Juan Ramon Torregrosa, Handling occlusion in optical flow algorithms
for object tracking, Computers and Mathematics with Applications 56
(2008) 733742
[32] D. Greenhill, J. Renno, J. Orwell, G.A. Jones, Occlusion analysis: Learn-
ing and utilising depth maps in object tracking , Image and Vision
Computing 26 (2008) 430441
[33] Lin Zhu, Jie Zhou, Jingyan Song, Tracking multiple objects through
occlusion with online sampling and position estimation, Pattern Recog-
nition 41 (2008) 2447 2460
[34] Andrew N. Stein *,1, Martial Hebert, Local detection of occlusion
boundaries in video, Image and Vision Computing 27 (2009) 514522
[35] Lucio Marcenaro, Franco Oberti,Carlo Regazzoni, Short-memory Shape
Models for Ground-plane Predictive Object Tracking, Proceedings 1st
Int. Workshop on PETS , Grenoble, France, March 31, 2000 (50-56 pag-
es)

zlem Morkaya received Phd Degree from Ege University Comput-
er Engineering Department in May 2011. She received MS Degree
from Ege University International Computer Institute in 2005 and BS
Degree from Ege University Computer Engineering Department in
2002. She is currently working at the Scientific and Technological
Research Council of Turkey (TBTAK), Ankara, Turkey. Her re-
search interests are image processing, object tracking and object
recognition.

Serdar Korukolu is a Full-time professor of Computer Engineering
Department at Ege University, Izmir, Turkey. He received his B.S.
degree in Industrial Engineering, M.Sc. in Applied Statistics and
Ph.D. in Computer Engineering from Ege University, Izmir, Turkey.
He was in Reading University of England as a visiting research fel-
low in 1985.

JOURNAL OF COMPUTING, VOLUME 3, ISSUE 12, DECEMBER 2011, ISSN 2151-9617
https://sites.google.com/site/journalofcomputing
WWW.JOURNALOFCOMPUTING.ORG 75