Pattern Recognition Letters 34 (2013) 21352143

Contents lists available at ScienceDirect

Pattern Recognition Letters

journal homepage: www.elsevier.com/locate/patrec

Localizing people in multi-view environment using height map

reconstruction in real-time
kos Kiss a,b,, Tams Szirnyi a
a
Distributed Events Analysis Laboratory, Institute for Computer Science and Control (MTA SZTAKI), Kende u. 13-17, H-1111 Budapest, Hungary
b
Department of Control Engineering and Information Technology, Budapest University of Technology and Economics, Magyar tudsok krtja 2., H-1117 Budapest, Hungary

a r t i c l e i n f o a b s t r a c t

Article history: In this article we address the problem of visual people localization, based on the detection of their feet.
Received 13 February 2013 Localization is based on searching cone intersections. The altitude of location is also retrieved, which
Available online 14 August 2013 eliminates the need of planar ground which is a common restriction in the related literature. We found
Communicated by D. Coeurjolly that positions can be computed accurately, and despite a large number of false positives, the height map
of the scene can be reconstructed with small error. Precision of the detector can be increased given the
Keywords: height map, so that results of our method are comparable to state of the art methods in case of planar
Multi-view localization
ground, but adding the ability to handle arbitrary ground. Our algorithm is capable of real-time operation,
3D position
Projection
based on two optimizations: decreasing the number of cones, and approximating intersection bodies.
Real-time processing Cones are back-projections of ellipses in images covering feet regions. Moreover, most demanding steps
are parallelizable, and distributable due to lack of data dependencies.
2013 Elsevier B.V. All rights reserved.

1. Introduction
Single camera object detection can rely on the visual appear-
ance of the object, or can utilize its dynamic properties. Appear-
ance models can incorporate some kind of descriptors, like color,
shape, and texture. However, sometimes the temporal behavior
of the object is more characteristic, for example, in case of gait
(Jung and Nixon, 2013).
Non-typical and occluding objects are hard to detect, multiple
cameras are often used to overcome these limitations. Using more
viewpoints the possibility of clear visibility increases, and by gain-
ing spatial information the localization of the objects also becomes
possible.
In such cases, consistency of object pixels among views will sig-
nal objects in 3D space. There are many methods for nding object
pixels. Using stereo cameras, the disparity map can highlight/seg-
ment foreground, or using wide-baseline stereo imaging,
image-wise foreground detection can be carried out (Benedek
and Szirnyi, 2008). In this latter case, eliminating shadows and
reections is a hard problem in itself (Benedek and Szirnyi,
2007; Prati et al., 2003).
The resulting set of object pixels can be further segmented, and
consistency checking might be reduced to likely correspondent
segments. This might be done using some descriptors (e.g. color
Mittal and Davis, 2001, 2002), however color calibration is neces-
sary due to different sensor properties and different illumination
conditions (Jeong and Jaynes, 2008).
Foreground masks can be projected to a ground plane, overlap-
ping pixels mark consistent regions (Iwase and Saito, 2004).
Robustness can be improved using multiple parallel planes (Khan
and Shah, 2009). Furthermore, looking for certain patterns in the
projection plane increases reliability (Utasi and Benedek, 2011).
In many works the authors assume that the homography and
the ground plane is known for carrying out the projections. In
Havasi and Szlavik (2011) homography parameters are estimated
from co-motion statistics from multimodal input videos, eliminat-
ing the need of human supervision.
The projection of whole foreground masks is computationally
expensive, but ltering pixels can reduce complexity. In some
works, points associated with feet are searched, reducing fore-
ground masks from arbitrary blobs to points and lines (Kim and
Davis, 2006; Iwase and Saito, 2004).
Another gain of ltering foreground points is that depending
on the geometry feet are less occluded than whole bodies, elim-
inating a great source of errors. Reducing occlusion is especially
important in dense crowds, for example, by using top-view cam-
eras (Eshel and Moses, 2010).

Corresponding author at: Distributed Events Analysis Laboratory, Institute for

Computer Science and Control (MTA SZTAKI), Kende u. 13-17, Budapest, H-1111
Hungary. Tel.: +36 703230688.
E-mail addresses: kiss.akos@sztaki.mta.hu (. Kiss), sziranyi.tamas@sztaki.mta.
hu (T. Szirnyi).
2. Overview
Our goal was to design an algorithm for localizing people in a
multiview environment where the geometry of the ground is

0167-8655/$ - see front matter 2013 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.patrec.2013.08.007
2136 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143
arbitrary, which problem is not addressed in the eld. Moreover
we aimed at reaching real-time processing with possibly many
views to make it available for surveillance systems.
We utilize the fact that foreground pixels of a view correspond
to lines in 3D scene space through projective back-projection
and lines coming from different views intersect in scene space at
surface points of the object. Detecting such intersections however
would require too many computations, as the number of line pairs
np is proportional to
n
np sx sy r fg 2 1 highly affects the appearance of shadows. Consequently, the selec-
2
where sx ; sy is the resolution of the camera image, r fg is the ratio of
foreground pixels in the image and n is the number of views. We
found that it is not the number of views that can cause bottleneck
problems, but the increasing number of foreground pixels, either
due to high resolutions or noisy foreground masks. Therefore we
1. lter the foreground mask to nd pixels relevant to detecting
people positions, and
2. cluster spatially coherent pixels into one primitive instead of
handling them separately.
Some aspects of this approach were briey introduced in Kiss
and Szirnyi (2013), in this work we give more details, explain
selection of algorithm parameters and show comparative test re-
sults. We lter pixels possibly corresponding to feet, and replace
these pixels with encompassing ellipses. These ellipses can be
back-projected to cones in scene space, thus the detection of inter-
secting cones can replace the detection of many line pair intersec-
tions. Our approach has several advantages in means of both
precision and speed:
 for determining cone parameters, undistortion may be carried
out with few computations leading to accurate parameters,
 the number of cones is proportional to the number of objects
regardless of image resolution, which makes our cone matching
approach scalable as opposed to pixel matching,
 ground does not have to be at (unlike when using homogra-
phies in Utasi and Benedek (2011), Khan and Shah (2009,
2006), Iwase and Saito (2004) and Berclaz et al. (2006)),
 we can compute a synergy map without presumption about
subjects heights (which is required in Utasi and Benedek
(2011) and Khan and Shah (2009, 2006))
However, the presented approach has some requirements and
in some cases can have some drawbacks:
 Precise calibration is required for reliable estimation of
cone parameters (both intrinsic and extrinsic parameters
of the camera). A small error in cone or camera parameters
can slightly change the direction of the cone, which can
cause large errors at locations far from the camera.
 Precise synchronization of the cameras is required, because
the positions of feet may change fast (comparable to the
size of feet in 0.1 s).
 Our algorithm may fail on incorrect foreground masks. As
we will show in Section 3.2, if the foreground mask is incor-
rect, the extracted ellipses will not cover the feet regions
precisely, or will not even intersect any foot.
We show the applied preprocessing steps for forming ellipses
from foreground masks in Section 3. Section 4 introduces the steps
of forming cones in scene space and matching these cones.
Section 5 describes feet localization from cone matches as well
as details of height map reconstruction. We show results and
comparison to state of the art methods in Section 6 and conclude
our work in Section 7.
3. Preprocessing
Foreground detection can be corrupted by shadows, reections
or changes in illumination conditions. To minimize such errors, we
use a foreground detection method capable of eliminating shadows
and reections (described in Benedek and Szirnyi (2008)).
Different color spaces characterize colors differently, which
tion of the color space highly impacts performance of shadow
detection (Benedek and Szirnyi, 2007). We evaluated a number
of color spaces and use XYZ as it leads to best overall performance
of our algorithmhowever in certain situations the foreground
mask is still corrupted by shadows, as we can see in Fig. 2(a).
Digital cameras auto white balance function is aimed at adopt-
ing to changes in illumination. However, large objects entering or
exiting the eld of view may affect the measurement of illumina-
tion parameters. This results in changing chromaticity of the out-
put image, even if lightning conditions remain the same.
Consequently pixel values will not t the static background model
and the foreground mask becomes invalid.
In many digital cameras this feature cannot be turned off at all.
To overcome this issue, we have to neglect its effect. We do this by
extending the background model with white balance adaptation: if
we observe a tendency in relative changes of values for a color
channel, we conclude that the gain of the channel changed, and up-
date parameters of the static background accordingly. We com-
puted the tendency of change T c for channel c only on
background pixels x 2 B to discard scene changes:
P Ic;x
x2B ~Ic;x
Tc 2
jBj
(
~Ic;x T c 2 0:97 . . . 1:03
~I0 3
c;x
~Ic;x =T c otherwise;
where Ic;x is value for channel c at position x; ~I is the static back-
ground parameter, ~I0 is the updated parameter. This method leads
to a more reliable foreground detection according to our visual val-
idation (see Fig. 1(a) for example).
We lter out small areas from the foreground mask to suppress
noise. In our tests, we eliminated areas covering less than 7 pixels,
as we found foreground blobs due to noise mostly did not exceed
this size. For other videos, this threshold might be adjusted accord-
ing to video quality. Note that this step is not necessary for our
method to work, it only speeds up further processing.
3.1. Filtering foreground pixels
We assume that cameras are in upright position so that pixels of
feet are bottom pixels of vertical lines in the foreground mask. We
consider these pixels as candidates for feet, calling them candidate
pixels. A sample of extracted candidate pixels can be seen in
Fig. 1(b).
Note that using different setups, the vanishing point for vertical
direction is known from calibration, so candidate pixel ltering can
also be carried out.
We cluster the candidate pixel set by connecting pixels closer
than 3px (choice of parameter is based on experiments, see
Fig. 5(d)). This is necessary since candidate pixels are often not
connected because of noise and steep edges. Note that these do
not necessarily change when using higher resolutions, so this
threshold can be used for different image sizes, however, adjust-
ment might be required for different image quality.
 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143 2137

Fig. 1. Steps of extracting feet.

After clustering, we obtain a set of pixel clusters. We model Feet can appear in different orientations, therefore the ellipse
each cluster with an ellipse according to the moments of the pixel can also be rotated. Rotating around origin with a we get:
coordinates. This makes our algorithm robust to image resolution,     
x0 c s x
as increasing resolution results in more candidate pixels, but the 0
7
same number of clustersthus reducing the number of 3D primi- y s c y
tives, which drastically speeds up pairwise matching. Only the Z Z Z
steps of preprocessing are affected by the resolution. x02 c2 x2 s 2 y2 8

3.2. Forming ellipses Z Z Z

y02 s2 x2 c 2 y2 9
We want to model possible position of a foot in space with a
cone. Intersection of a cone and a planethe image plane in our Z Z Z 
caseis an ellipse lying on the plane. For this reason, we consider x0 y0 sc x2  y2 10
a cluster to be a sampling of an elliptical area. For an ellipse with
major and minor radii a and b parallel to axis, we know: R
2 x0 y0
tg2a R R 11
Z y02  x02
a3 bp
x2 4 Z Z Z Z
4 abp 2 2
Z 3 x02 y02 x2 y2 a b 12
2 ab p 4
y 5
4 R Z Z
Z
x0 y0 abp 2 2
xy 0 6 x2  y2 a  b 13
sc 4
2138 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143
Z
1 abp; 14
where s sina; c cosa. Now we can compute major and minor ra-
dii and rotation. We reject too small and upright ellipses. Minimal
size can be adjusted for different setups and image resolution. In
our experiments ellipses covering less than 7 candidate pixels were
rejected, this ensured feet far from the camera are still detected
(choice of parameter is based on evaluation, see Fig. 5(d)). We de-
ned the maximal angle as 45 . Foot direction does not depend
on setup or resolution, so this threshold should work in any setup.
Sample results can be seen in Fig. 1(c).
4. Detecting correspondences
We model candidate pixel clusters with ellipses in every view.
We then back-project ellipses in all images to cones in scene
spacethe 3D reference space. Cones corresponding to a foot will
intersect close to its location.
However, cones corresponding to different feet can intersect,
and candidate pixels can appear on arms or on foreground artifacts.
Both can lead to false detections, Fig. 2(b) shows examples.
Intersection of cones is a complex bodyexcept for some
extraordinary situations. Determining this body would be difcult,
but it is not even necessary. We use simple heuristics to detect
cone intersections which we call cone matching.
4.1. Forming cones
We describe a cone with a vertex C and three orthogonal vectors
u; v and w, where w is the unit length direction vector of the axis
and u; v are direction vectors of major and minor axes (Fig. 3(a)).
Bevel angles are determined by the length of u; v vectors. A cone
consists of points p where
 2
p  Cu p  Cv 2 6 p  Cw2 ; p  Cw P 0 15 v 0 v jCPclose j
Fig. 2. Sources of errors.
From the extrinsic parameters of the cameras we know the 3D
position of points of the image plane as well as the optical center O.
Thus computation of w is straightforward. We determine major
and minor axes directions by vector products to ensure the orthog-
onality of vectors:
v ui  w 16
uwv 17
Finally, u and v are scaled according to major and minor bevel
angles so that (15) stands.
4.2. Cone matching
The intersection of cones is a complex body, especially when
bevel angles differ for major and minor axes. Our experiments
showed that we do not need the exact intersection, an approximate
solution is sufcient for searching cone matches. We simplify com-
putations in two ways:
1. Generally a foot covers a small area of the image, so cones have
small bevel angles. Consequently in a relatively small space seg-
ment they can be replaced by cylinders. We assume that an
intersection will take place near the point where the two cone
axes are closest, so we compute the major and minor radii at
!
this offset (CP close in Fig. 3(b)).
2. Computing the exact intersection of cylinders is still a hard
problem, and the intersection body is still complex. We only
seek for an optimal point p in space, subject to minimal distance
from cylinder axesconsidering different major and minor radii.
!
After computing the jCP close j distance, we scale u and v vectors
to match major and minor cylinder radii:
!
u0 ujCPclose j
18
!
 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143 2139

Fig. 3. Steps of extracting feet.

Now, the cylinder equation becomes cylinder axes (the error of the approximate solution) we also assign
 2 a weight to matches.
2
p  Cu0 p  Cv 0 6 1 19
4.3. Indexing cones
The distance from axis left side of (19) is a linear function of
p, enabling us to write a linear equation system expressing p is on
In stereo vision, epipolar constraints are often used to reduce
both axes (index refers to cylinders):
the search space for nding correspondences (Rodrigues and
 2  2 Fernandes, 2004; Mittal and Davis, 2001). The corresponding lines
p  C 1 u01 p  C 1 v 01 0
in two images refer to a plane on both optical centers (O1 O2 ). The
 2  2 20
lines are characterized by only a rotation around this O1 O2 axis,
p  C 2 u02 p  C 2 v 02 0;
so a point in space could be indexed by this rotation angle.
Cones are extent, their index could be the angle interval they
which is equivalent to
touch. However due to the small number of primitives and the very
2 3 0 1 low computational time, applying indexing would not affect over-
u0T
1 u01 C 1
6 v 0T 7 B v0 C C all performance, and therefore we do not utilize epipolar
6 1 7 B 1 1C
6 0T 7p B 0 C 21 constraints.
4 u2 5 @ u2 C 2 A
v 0T2 v 0 2 C2
5. Localizing feet
Ap b 22
5.1. Detection
Of course, axes will practically never intersect, we can only
compute an approximate solution. Solving subject to least square One match itself can come from a false correspondence. How-
error is straightforward and practical in our case. It is straightfor- ever, for two matches, the foot has to be visible in at least three
ward, because it is a distance from the axes considering different views, which is not guaranteed in a dense crowd. Matches from
major and minor radii, and it is practical, because this optimization random correspondences tend to have low weights, so detecting
is easy to carry outfor example using pseudo inverse. a foot from a single match is possible by thresholding the matchs
If p is outside both cylinders, we conclude that the cones are not weight.
intersecting, otherwise they intersect and p is the position of the We merge matches close in space to form one detection. We
intersection which we call a match. Using the distances of also assign weights to such match sets, computed from the weights
2140 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143
Fig. 4. Sample results of localization and determining height map.
of the matches so that the possibility of detection highly increases
with the number of matches.
We merge matches over a grid on a horizontal plane. As feet do
not appear above each other in a cell of the grid we do not mix
matches from different altitudes. We collect matches in overlap-
ping 2  2 cell blocks, then compute the weights of match sets in
every block. Using overlapping blocks is necessary to obtain a
smooth weight map over the grid.
Every local maximum in this 2D weight map with a weight
above a given threshold is considered a detected foot. This thresh-
old is a parameter of our algorithm. A higher threshold leads to in-
creased precision along with decreased recall. The 3D location of
the foot is computed as the barycenter of matches in the block,
the resulting z coordinate is the altitude (it is parallel to upright
direction).
5.2. Height map
As we mentioned before, and showed in Fig. 2(b), false corre-
spondences can occur due to incorrect foreground detection or ran-
dom intersections of non-related cones. Especially in a dense
crowd, many false matches appear. However the altitude of these
detections is quite random, and they usually occur far from the
ground surface - either above or below.
Consequently these false positives appear as a noise. To sup-
press this noise, we collect all feet locations from sample videos
and lter dense regions in scene space. This is easily done using
statistical ltering by assuming normal distribution. The resulting
dense points will be height points of ground surface.
The phrase height map can be somewhat misleading, because at
the border of surfaces in different altitudes, it is possible to have
height points above each other, however in our tests this never
occurred. Thus we kept the height map expression.
After extracting the height map, we ignore detections far from
any height point, leading to a much more reliable method. We
speed up height point searching by accumulating detections in a
3D grid, which makes periodic reruns possible. This enables us to
rene the height map in time, as more and more detections occur.
A sample height map of our non planar test case is shown in
Fig. 4(a). There are certain areas where few detections took place,
which led to an incomplete height map.
6. Experiments
We tested our algorithm on a commonly used test sequence for
multiview detection, the EPFL terrace sequence (EPFL, 2011) (sam-
ple results can be seen in Fig. 4(b)), and our own test videos (SZTAKI
sequence).
 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143 2141

Fig. 5. Evaluation of height point calculation and detection.

We made test videos of a scene where three planar surfaces
were present at different altitudes. This was required to demon-
strate the capabilities of our method as available multiview test
sets present planar ground. We recorded videos from 4 views with
different kinds of consumer digital cameras with video capture
function. Our algorithm performed well on this dataset despite
the different camera parameters, distortion and image quality.
Our approach for detecting foot has advantages and disadvan-
tages as well. Feet are always near the ground, which makes height
map detection possible and helps rejecting false positives. Also, for
certain camera setups, feet are less likely to be occluded.
On the other hand, in case of detecting foot positions, precise
time synchronization is necessary, as a small time skew can cor-
rupt detection. In Fig. 2(c), we can see the moving leg is bended
in one view while straight in the other. Hardly noticeable in the
small image, however the distance in space is greater than the size
of the foot itself.
6.1. Height map reconstruction
Our algorithm reconstructed the height maps for both EPFL and
SZTAKI sequences with high precision in both cases.
6.1.1. EPFL sequence
In case of the EPFL sequence, we measured that the recon-
structed ground surface is a at plane with a small error:
l 1:6 cm (mean), r 1:7 cm (std. deviation). Fig. 5(a) shows
the histogram of altitude values.
Assuming noise-like error, altitude values on a at ground
should follow normal distribution, however we observed a differ-
ent distribution. A possible source of systematic error is an incor-
rect plane normal, so we computed the optimal normal vector by
tting a plane on height points. The optimal normal vector was
very close to the upright direction specied by camera calibration.
It diverged by only 0:36 , resulting in a slightly smaller error:
2142 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143
Table 1
Numerical evaluation of two main aspects: accuracy and running time.
Surface Height l r
(a) Statistical information on surfaces found in scene.
Floor 0 cm 1.1 cm
Box 50 cm 50.2 cm
Table top 73 cm 74.4 cm
Test set Foreground detection Candidate ltering
(b) Average processing time of each step for all 4 views (single threaded implementation, 2.4 GHz Core 2 Quad CPU). Foreground detection, ltering and cone forming can be done
parallel (per camera), cone matching and detection requires all cone information to be present.
EPFL 51.2 ms 3.37 ms
SZTAKI 32.5 ms 2.88 ms
Fig. 6. Comparing our results to a SoA method.
l 1:6 cm;r 1:5 cm. However, using this normal led to a signif-
icantly better distribution (Fig. 5(a), right histogram).
6.1.2. SZTAKI sequence
In case of our sequence, all three surfaces were found, and their
altitude was determined with high accuracy. Measurements are
summarized in Table 1(a). Fig. 6(b) shows the histogram of altitude
for points on ground surface.
There are 3 outlier points (above 6 cm height) for the oor.
These appear on the edge of the area of interest, as can be seen
in Fig. 4(a). This is due to sagging people which lead to false can-
didate pixels on image borders. Without these outlier points, mean
and r for oor becomes 0.9 cm and 1.1 cm respectively.
6.2. Localizing people
We tested our method using manually created ground truth
information of feet positions. We implemented an application to
create ground truth, in which we have to mark a foot in at least
2 views and its position is determined by solving a linear equation
similar to (21). If we mark a foot in more views, the number of
rows of A and length of b increases.
We collected feet positions frame by frame, person by person. 1
or 2 legs can be specied for a person, because sometimes only one
foot is visible from more views.
For the evaluation, we dened a region of interest (ROI), a rect-
angle on the ground plane and matched locations to ground truth
inside this area. In case of the EPFL dataset, this rectangle is spec-
ied along with the dataset, for our test case this area was chosen
so that every part is visible from at least 3 views.
We accepted a location as true positive (or detected foot), if it
was not further than 25 cm from a ground truth foot position. This
is approximately the length of a foot. Afterwards we eliminated
people and positions outside the ROI, and computed false negatives
# of points min max
1.7 cm 131 0.3 cm 11.2 cm
0.6 cm 6 49.1 cm 50.8 cm
1.2 cm 23 70.3 cm 75.9 cm
P P
Cone forming (per camera) Cone matching Detection (per system)
63 ls 13.7 ms 879l s 28l s 907l s
1.99 ms 9.35 ms 600l s 18l s 618l s
and false positives from the remaining people and locations
accordingly.
Other multiview pedestrian localization methods localize the
whole bodies unlike our method, which localizes feet. Conse-
quently, a different evaluation criteria had to be used: we consid-
ered a person detected, if either of his legs was detected. With this
denition, our results become comparable to other results.
The performance of our algorithm depends on some parame-
ters. As mentioned in Section 5.1, a detection threshold is applied
to the weight map. The trade-off between precision and recall is
balanced with this threshold. To evaluate our method, we ran tests
with different threshold values, resulting in precision-recall curves.
Results are shown in Fig. 5(c).
Two other parameters were introduced in Section 3: maximal
distance for candidate pixel clustering and minimal cluster size.
We measured achievable maximum of minprecision; recall (in
function of the detection threshold) and averaged over the data-
sets, Fig. 5(d) shows results. We found best overall performance
at 3px clustering distance and minimal cluster size of 7. Increased
minimal cluster size might increase precision, but leads to ltering
out small feet, which decreases achievable recall.
6.2.1. Comparison to SoA methods
We compared our method to two SoA methods referred to as
POM (Fleuret et al., 2008) and 3DMPP (Utasi and Benedek, 2011).
Comparison is not straightforward, as these methods localize
bodies projected to a ground plane, in contrast we localize feet in
space. We consider someone detected if any foot is detected.
Fig. 6(a) shows results found in Utasi and Benedek (2011) for
POM and 3DMPP evaluated on EPFL and PETS (PETS, 2009) data-
sets compared to our measurements. We evaluated our method
on the same dataset, but we found that PETS did not meet the
requirements of our algorithm (in means of precise calibration
and synchronization), so we could only use the EPFL dataset.
 . Kiss, T. Szirnyi / Pattern Recognition Letters 34 (2013) 21352143
We used similar amount of test data: 179 test frames (every
25th frame from the test sequence) with 661 objects appearing
in the ground truth data with at least one leg inside the ROI,
which was the same as in Utasi and Benedek (2011). Results show
that our method is comparable to these SoA methods in case of at
ground surface.
We also compared results on our SZTAKI dataset to show the
effects of a non-planar ground surface. For this we chose POM as
it was easy to generate input for its available implementation.
For our dataset, foreground detection was challenging due to
reective ground and poor video quality. Consequently both
methods performed worse in this case, however our algorithm
performed noticeably better (see Fig. 6(b)).
6.3. Running time
We measured the average running time of the processing steps,
with results shown in Table 1(b). Every step highly depends on the
input data, thus the processing time is not linear in the number of
pixels in the images. In these cases, the video resolution was
360  288 for EPFL and 320  240 for SZTAKI sequences.
Results show that our method is capable of real-time processing
for both EPFL and SZTAKI sequences even with single threaded
implementation (including foreground detection). However, paral-
lel processing is also possible with some restrictions.
Several steps of the processing run on single images, therefore
can be parallelized. Clearly, foreground detection, ltering, ellipse
and cone forming is done image-wiseno information exchange
is required between different views at all. Consequently, process-
ing individual images can run on separate threads or even be dis-
tributed to different computerspossibly smart camerasdue to
lack of data dependency.
Cone matching and localization requires all cone information to
be present. Still, these steps could be parallelized but only in a non-
trivial manner. Therefore, due to their short processing time these
steps are preferred to run sequentially.
In case of the above mentioned SoA methods, mean processing
time of foreground masks is in the order of 0:5 s for POM, and 1 s
for 3DMPP, using unoptimized code on similar processor (data pro-
vided by author of Utasi and Benedek (2011)). However in our case,
processing the foreground took less than 5 ms.
Moreover, many methods implicate data dependencies which
make distributed computing impossiblelike projecting masks in
3DMPP, or iterative probability computation in POM.
7. Conclusion
We proposed a multiview-localization algorithm that extracts
3D position of people using multiple calibrated and synchronized
views. In our case, unlike other algorithms, non-planar ground sur-
face can be present. This is done by modeling possible feet loca-
tions with 3D primitives, cones in scene space and searching for
intersections of these cones.
For good precision, the height map of the ground surface should
be known. Our method can compute the height map on the y,
reaching high precision after a startup time.
2143
After height map reconstruction we measured precision and re-
call values comparable to SoA methods on a commonly used data-
set. Our algorithm worked well also on our test videos we made to
demonstrate the capabilities of handling non-planar ground
surface.
In the future we plan to examine tracking people by their lean-
ing leg positions (Havasi et al., 2007).
Acknowledgements
The authors thank Franois Fleuret for making the implementa-
tion of POM available, and kos Utasi for sharing his experiments
of evaluating POM and 3DMPP algorithms. This work has been sup-
ported by the Hungarian Scientic Research Fund grant OTKA
#106374.
References
Benedek, C., Szirnyi, T., 2007. Study on color space selection for detecting cast
shadows in video surveillance. International Journal of Imaging Systems and
Technology 17 (3), 190201.
Benedek, C., Szirnyi, T., 2008. Bayesian foreground and shadow detection in
uncertain frame rate surveillance videos. IEEE Transaction on Image Processing
17 (4), 608621.
Berclaz, J., Fleuret, F., Fua, P. 2006. Robust people tracking with global trajectory
optimization. In: IEEE CVPR, pp. 744750.
EPFL. 2011. Multi-camera pedestrian videos. http://www.cvlab.ep.ch/data/pom/
Eshel, R., Moses, Y., 2010. Tracking in a dense crowd using multiple cameras.
International Journal of Computer Vision 88, 129143.
Fleuret, F., Berclaz, J., Lengagne, R., Fua, P., 2008. Multicamera people tracking with a
probabilistic occupancy map. IEEE Transactions on Pattern Analysis and
Machine Intelligence 30 (2), 267282.
Havasi, L., Szlavik, Z. 2011. A method for object localization in a multiview
multimodal camera system. In: CVPRW, pp. 96103.
Havasi, L., Szlvik, Z., Szirnyi, T., 2007. Detection of gait characteristics for scene
registration in video surveillance system. IEEE Transaction on Image Processing
16 (2), 503510.
Iwase, S., Saito, H. 2004. Parallel tracking of all soccer players by integrating
detected positions in multiple view images. In: IEEE ICPR, pp. 751754.
Jeong, K., Jaynes, C., 2008. Object matching in disjoint cameras using a color transfer
approach. Machine Vision and Applications 19, 443455.
Jung, S.-U., Nixon, M.S., 2013. Heel strike detection based on human walking
movement for surveillance analysis. Pattern Recognition Letters 34 (8), 895
902.
Khan, S., Shah, M. 2006. A multiview approach to tracking people in crowded scenes
using a planar homography constraint. In: ECCV 2006, Lecture Notes in
Computer Science, pp. 133146.
Khan, S., Shah, M., 2009. Tracking multiple occluding people by localizing on
multiple scene planes. PAMI 31 (3), 505519.
Kim, K., Davis, L.S. 2006. Multi-camera tracking and segmentation of occluded
people on ground plane using search-guided particle ltering. In: ECCV, pp. 98
109.
Kiss, Szirnyi, T. 2013. Multi-view people detection on arbitrary ground in real-
time. In: VISAPP, pp. 675680.
Mittal, A., Davis, L. 2001. Unied multi-camera detection and tracking using region-
matching. In: IEEE Multi-Object Tracking, pp. 310.
Mittal, A., Davis, L. 2002. M2tracker: a multi-view approach to segmenting and
tracking people in a cluttered scene using region-based stereo. In: ECCV 2002,
pp. 1833.
PETS. 2009. Performance evaluation of tracking and surveillance.
Prati, A., Mikic, I., Trivedi, M.M., Cucchiara, R., 2003. Detecting moving shadows:
algorithms and evaluation. IEEE Transactions on PAMI 25, 918923.
Rodrigues, R., Fernandes, A.R. 2004. Accelerated epipolar geometry computation for
3D reconstruction using projective texturing. In: SCCG 04, pp. 200206.
Utasi, ., Benedek, C. 2011. A 3-D marked point process model for multi-view
people detection. In: IEEE CVPR, pp. 33853392.