Laser Scanner and Camera Fusion

for Automatic Obstacle Classication

in ADAS Application
Aurelio Ponz1(&), C.H. Rodrguez-Garavito2, Fernando Garca1,
Philip Lenz3, Christoph Stiller3, and J.M. Armingol1
1

Intelligent Systems Lab, Universidad Carlos III de Madrid, Butarque 15,

Legans, Madrid, Spain
{apv,fegarcia,armingol}@ing.uc3m.es
2
Automation Engineering Department, Universidad de La Salle, Bogot,
Colombia
cerodriguez@unisalle.edu.co
Institut fr Mess- Und Regelungstechnik, Karlsruher Institut fr Technologie,
Karlsuhe, Germany
{lenz,stiller}@kit.edu

Abstract. Reliability and accuracy are key in state of the art Driving Assistance
Systems and Autonomous Driving applications. These applications make use of
sensor fusion for trustable obstacle detection and classication in any meteorological and illumination condition. Laser scanner and camera are widely used
as sensors to fuse because of its complementary capabilities. This paper presents
some novel techniques for automatic and unattended data alignment between
sensors, and Articial Intelligence techniques are used to use laser point clouds
not only for obstacle detection but also for classication.. Information fusion
with classication information from both laser scanner and camera improves
overall system reliability.
Keywords: ADAS
Computer vision
Lidar
system
Laser
Stereo camera
Sensor fusion

Advanced driving assistance

1 Introduction
Trafc accidents are among the most important avoidable risks for human life. About 1.2
million people die and 50 million people are disabled every year in the world as a
consequence of trafc accidents [16]. Advanced Driving Assistance Systems (ADAS) can
reduce the number and severity of trafc accidents by using Computer Vision
(CV) technologies, Articial Intelligence (AI) and Laser. ADAS use widely laser scanners
and cameras to detect and classify obstacles on the road. These sensors are complementary, as the lasers ability to detect obstacles regardless of the light quality and to select
Regions of Interest (ROI) for camera classication, improves remarkably the speed and
accuracy of the Computer Vision classication in the images from the camera.
The present work has been developed using the Intelligent Vehicle based on Visual
Information 2.0 (IVVI 2.0), as seen in Fig. 1, the Intelligent Systems Labs research
platform, designed to develop and test ADAS technologies [10].
Springer International Publishing Switzerland 2015
M. Helfert et al. (Eds.): Smartgreens 2015 and Vehits 2015, CCIS 579, pp. 237249, 2015.
DOI: 10.1007/978-3-319-27753-0_13

238

A. Ponz et al.

The article is divided in the following sections:

Section 2 provides scientic context of the state of the art in the related domain.
Section 3 is a general description of the system. Section 4 describes the method for
laser point cloud clustering, which is the initial part of obstacle detection. Section 5
outlines the data alignment process, essential for a correct data association between the
camera and the laser system. Section 6 depicts the strategy for obstacle classication
using laser point clouds and images, with a Support Vector Machine (SVM). Finally,
Sect. 7 presents conclusions for the present work.

2 Related Work
The work described in the present paper covers several elds with interesting state of
the art. Regarding the automatic and unattended data alignment phase in our system,
[8] proposed a method for calibration using a chessboard pattern, [13] proposed a
method for automatic camera and laser calibration, based on point cloud reconstruction
of the road surface. Other approaches such as [7] and [6] projects the features into a 2D
plane to minimize the distance among the features in the different sensors. [9] presents
a CAD model based calibration system for inter-sensor matching. Similar approach
based on triangular model is presented in [1] and based on circular models in [3].
Once the different systems are aligned, next step is obstacle detection and classication. Data fusion detection approaches can be divided in centralized and decentralized schemes. First perform a single detection based on the information provided by
the different sensors. Some examples of decentralized schemes can be found in [11]
and [12] with different algorithms to combine the features from computer vision and
laser scanner, such as Nave Bayes, GMMC, NN, FLDA. Decentralized schemes
implements detection and classication on each sensor independently and a further
fusion stage combines the detection according to the certainty provided by each sensor.
[15] provides high level fusion based on multidimensional features for laser scanner
and Histogram of Oriented Gradients (HOG) for computer vision. [4] provides
pedestrian detection based on pedestrians leg model for laser scanner and HOG features for computer vision for distributed pedestrian detection and danger evaluation. [5]
takes advantage of advance fusion techniques (i.e. Joint Probabilistic Data Association
Filter) to enhance decentralised laser and computer vision pedestrian detection.

3 General Description
The presented work uses sensor fusion between laser scanner and computer vision for
obstacle detection and classication in automotive applications.
This work is included in the IVVI 2.0 project (Fig. 1). IVVI 2.0 is the second
platform for development and research of ADAS technologies of the Intelligent Systems Lab, at Universidad Carlos III de Madrid.
In the presented application, a Sick LDMRS 4-layer Laser Scanner and a trinocular
camera are used. Laser scanner for primary obstacle detection and later for classication, and stereo capability from the trinocular camera is used for point cloud ground

Laser Scanner and Camera Fusion

239

Fig. 1. IVVI 2.0 research platform.

representation and data alignment parameters estimation; later one of the cameras from
the trinocular camera is used as a monocular camera for image capturing.
The laser scanner generates a point cloud in which the system extracts the obstacles
as clusters of points. These clusters are used both for ROI generation in the images and
as information for obstacle classication (Fig. 6). The extracted ROIs in the image are
processed for obstacle classication using AI methods applied to Computer Vision.
The last step in the process performs further information fusion between laser and
camera for a nal obstacle classication based on machine learning.
A database with manually labeled images and point clouds is used for SVM
training and testing in the classication process.

4 Point Cloud Clustering for Laser Detection

The rst step in our system is the obstacle detection using laser generated point clouds.
This is the most reliable sensor in our system, as it is not affected by illumination
conditions but only by some meteorological conditions.
The four layer laser sensor obtains a point cloud representing some of the reality in
front of the vehicle. Obstacles are part of this reality and can be located as local
concentrations of points in the point cloud that can be mathematically categorized as
clusters.
Several clustering techniques have been studied in order to obtain the highest and
most reliable amount of information from the point cloud. It is important to note that
obstacles to be detected will be represented by very few points in the point cloud,
typically from four points to not much more than fty depending on the distance to the
vehicle, due to laser limitations. Most of the clustering strategies already available are
designed for highly populated point clouds, obtained from high resolution multilayer
laser scanners or stereo cameras, and do not adapt well to our outdoor, sparse point
clouds offering limited information.

240

A. Ponz et al.

Our SICK LD-MRS laser scanner offers some scanning frequencies with different
angular resolution. The smallest frequency, 12.5 Hz, with variable angular resolution
between 0.125 in front of the vehicle, 0.25 between the 10 and the 30 and 0.5
between 30 and 50 (60 in the right side of the scene) as seen in Fig. 2. This
conguration increases the ability for long range detection in front of the vehicle, where
obstacles tend to be further. For automotive applications, lower resolutions in the sides
are acceptable, as the obstacles of our concern are closer than in the front and will be
represented by many points even at lower resolutions.

Fig. 2. Angular resolutions by sector in the laser scanner.

Distances between measured points in Y explain the need for adaptation of cluster
threshold according to the distance from the obstacle to the laser, in order to obtain the
most populated possible clusters.
The meaning of the values in Table 1 are:
Table 1. Distances between measured points at angular resolutions of 0.125.
Dist (m) Y_S Y_G1 Y_G2 X_Layer
10
0.014 0.029 0.007 0.139
25
0.035 0.074 0.019 0.349
50
0.069 0.148 0.039 0.698
100
0.139 0.296 0.078 1.396

Laser Scanner and Camera Fusion

241

Y_S is the width of the measured point, Y_G1 is the distance between measured
points in one measurement plane, Y_G2 is the distance between measured points
between two laser pulses, and X_Layer is the height of the measured point [14].
Further sections deploy the different techniques developed for advanced automatic
cluster detection.
4.1

Adapted Euclidean Distance and Geometrically Constrained Clusters

In this approach, a classical Euclidean distance clustering strategy has been adopted,
modulated by several parameters in order to modify the clustering behavior, such as
distance from the sensor to the obstacle, geometrical constraints, allowed number of
points in every cluster, etc.
Additionally, some parameters used in the clustering process such as maximum
distance between candidate points, are modied according to the shapes detected in the
point cloud near to the cluster, to improve oblique obstacle detections.
An alternative strategy has been tested, using Mahalanobis distance, as the normalized Euclidean distance from the clusters centroid to candidate points. This method
tends to obtain compact clusters and ignores increasingly further points belonging to
oblique obstacles. Taking into account that our scenario will produce small clusters,
that is the reason why it has been discarded.
In this approach, clusters are dened as the set of points separated a certain distance, which varies as a function of several parameters, plus some points that does not
meet the distance requirements, but some geometric constraints, such as belonging to
the same line in the space than some of the points in the cluster.
The strategy is dened as an iterative addition of points to the cluster with the
following steps:
First point in the point cloud is taken as the rst point in the cluster.
All the other points in the point cloud are checked to have a distance smaller than
the cluster threshold ClusterTh

where BaseTh is a parameter experimentally determined as the base threshold. DistCorr

(x) is a function of the x coordinate which ensures that no distance smaller than the

242

A. Ponz et al.

minimum physically possible distance will be required, as seen in Eq. 1, and depending
on the different angular resolutions seen in Fig. 2. DistCorr(x) is computed as the
minimum distance possible between two consecutive points in z and y coordinates. ay
Represents the angle between two consecutive laser reads in horizontal (y axis) and az is
the angle between two consecutive laser reads in vertical (z axis) (Fig. 3).

Fig. 3. IVVI 2.0 research platform with axis represented in the image: X = laser-obstacle
distance, Z = detection height, Y = horizontal deviation from the laser.

All the points in the point cloud are checked for cluster inclusion. The same
iteration is performed for every point added to the cluster until all cross checks are
performed. Then, points close to the obstacle but not belonging to the cluster are
included into a temporary new point cloud together with the obtained cluster, and then
lines are searched in the new cluster using RANSAC. If lines are found containing a
determined minimum of points belonging to the original cluster and points not
belonging to it, then these points are added to the cluster. This strategy has proven to be
effective for oblique obstacles.
Figure 4 shows the result of the algorithm. Red dots are the cluster created by
Euclidean Adapted distance. Blue dots are the points close to the cluster but not
belonging to it. Yellow lines are 3D lines found by RANSAC, including points from
the original cluster and points from the extended cluster.
Upon completion of cluster extraction, it is checked against the parameters ClusterTolerance for maximum width of cluster in meters, and minClusterSize and
maxClusterSize for minimum and maximum number of points, respectively. These
parameters are also a function of the distance to the obstacle.
The strategy is addressed to obtain the most populated clusters possible, taking into
account that we are using a low resolution multilayer laser. The threshold distance must
be adapted to the distance x from the laser sensor to the obstacle, as the distance
between consecutive laser points grows with x. Due to laser construction limitations,

Laser Scanner and Camera Fusion

243

Fig. 4. Extended cluster using geometrical constraints.

the minimum distance detected in y and z in consecutive points will be greater than the
initial threshold if not adapted following Eq. (1).

4.2

Ground Detection and Removal from Point Cloud

As outlined in Sect. 5, our system can compute the plane corresponding to the road
surface, so it is possible to remove ground plane points from the list of detected
clusters. Figure 5 shows the result of the algorithm, ignoring as cluster candidates all
the points located in the ground plane obtained with RANSAC.

Fig. 5. Cluster removal in ground plane.

244

A. Ponz et al.

Fig. 6. Obstacle detection based on cluster computation.

5 Data Alignment
Our system is based in data fusion between several sensors, based on different physical
phenomena. Thus each of these sensors has its own system of reference, and extrinsic
parameters between sensors system of reference must be estimated in order to perform
the data alignment.
To achieve the necessary alignment, rotation and translation between sensors must
be estimated. Some methods have already been proposed by other authors, involving
chessboards or specic patterns [8] detectable by all of the sensors involved in the
fusion [6]. This is cumbersome and requires driver implication or some help from
others, needs specic and stationary environment and to be performed manually again
in case of change of orientation or translation between sensors.
Our approach estimates the extrinsic parameters of all the sensors involved, and
calibration between them, assuming flat surface in front of the vehicle, thus sensors
height and two rotation angles can already be determined. For the third angle computation, any identiable obstacle located in the road surface can be used.
Applying the M-estimator-Sample-Consensus (MSAC) algorithm for plane detection in the point clouds obtained from the stereo camera and from the laser scanner, the
most populated planes are found from the clouds in the form
pX : axc byc czc d 0

which can be written in the Hessian form

*

n:p h
p X : ~

Laser Scanner and Camera Fusion

245

where ~
n is the vector normal to the road plane, and the relation between this vector and
the camera and laser rotation angles can be computed as in [13].
Once all the calibration parameters, i.e. roll, pitch, yaw and x,y,z translations
between sensors have been computed, the system is able to translate from laser
coordinates into camera coordinates in the image for obstacle classication using
Computer Vision.
The conversion between laser and image coordinate systems can be performed as in
Eq. (4)

where T represents the translation vector and R the rotation matrix between sensors.

6 Obstacle Classication Using Laser and Image Information

Fusion
Obstacle classication in this work can be performed with single sensor information or
using sensor fusion information. The automated environment developed will allow
direct comparison of results and fast training improvements.

6.1

SVM Classication

Classication is performed using the SVM implementation from the Computer Vision
OpenCV library. SVM algorithm was developed by Vapnik & Cortes [2] and is widely
used in machine learning as a classication method. In the present work, a database of
manually labelled images and clusters obtained from a capture with the IVVI platform
is used to execute a supervised learning process. Image database also includes images
from sources other than the IVVI. After the training process, the SVM structures are
stored and used for classication of images and clusters as seen in Figs. 8 and 9.

246

6.2

A. Ponz et al.

Laser Scanner Feature Vector

Clusters detected in laser scanner generated point clouds are used to determine a
Region of Interest in the image where we can perform obstacle classication applying
Computer Vision and Articial Intelligence techniques, but can also be used for
obstacle classication without image support [11].
Clusters are converted into a mesh structure by Delaunay triangulation in order to
reconstruct the shape of the obstacle and to extract relevant features according to the 3d
shape of the cluster, as seen in Fig. 7. These obstacles are detected by the system as
clusters, which have some characteristics suitable for further SVM training following
the process outlined in Fig. 8. Clusters obtained from the test sequences are stored and
manually labeled using the corresponding images for training. These clusters are
manually labeled as frontal view, back view, side view, frontal oblique view and back
oblique view.

Fig. 7. Mesh representation of a cluster.

Fig. 8. SVM learning process for clusters: training and classication.

Laser Scanner and Camera Fusion

247

Previous works [11] have considered 2D point clouds for classication, but the
present work is intended to extract features from a 3D point cloud, in an effort to
maximize the use of the available information. The features considered are described in
Table 2.
Table 2. Features considered for cluster classication.
Concentration: Normalized mean distance to the centroid 3D
Y-Z concentration: normalized mean distance to the centroid excluding x
X-Z concentration: normalized mean distance to the centroid excluding y
X-Y concentration: normalized mean distance to the centroid excluding z
Planicity: normalized mean distance to the most populated plane found in the cluster
Sphericity: normalized mean distance to the most populated sphere
Cubicity: measures how far are the planes containing the mesh triangles from being the same
plane or from being perpendicular
Triangularity: measures the uniformity of the triangles composing the mesh bye the relation
between sides lengths
Average deviation from the median in x, y, z

Fig. 9. SVM learning process for images: Training and classication.

6.3

Computer Vision Feature Vector

As pointed before, obstacles found in the Laser Scanner Point Cloud as clusters are
used to determine a ROI in the image suitable for applying SVM for obstacle classication in images.
An image database has been created using both the ROIs obtained from obstacles
detected by laser and images from public databases, from any point of view.

248

A. Ponz et al.

These images have been manually labeled as frontal view, back view, side view,
frontal oblique view and back oblique view. Later, Histogram of Oriented Gradients
(HOG) features are extracted from every image (Fig. 10), and SVM training is performed
following the process outlined in Fig. 9, in order to obtain the SVM classier [17].

Fig. 10. Original image and representation of HOG features.

6.4

Information Fusion

In poor illumination conditions, when the camera offers no help, laser scanner obstacle
detection and classication can still be used, but the real advantage of the sensor fusion
resides in the combination of the information obtained from several sensors to obtain a
result which is greater than the mere sum of the individual contributions. [5]. Information fusion will be performed with SVM and results will be compared with individual sensor classications.

7 Preliminary Results and Discussion

The presented work is currently under development but its results have been preliminarily tested, showing better gures using sensor fusion for classication than single
sensor classication, as presumed, and very promising expectations. Future tests will
present complete quantitative results and comparisons.
Acknowledgements. This Work Was Supported by the Spanish Government through the
CICYT Project (TRA2013-48314-C3-1-R).

References
1. Debattisti, S., Mazzei, L., Panciroli, M.: Automated extrinsic laser and camera
inter-calibration using triangular targets. In: 2013 Intelligent Vehicles Symposium (IV),
pp. 696701. IEEE (2013)
2. Cortes, C., Vapnik, V.: Support vector network. Mach. Learn. 20, 125 (1995)

Laser Scanner and Camera Fusion

249

3. Fremont, V., Bonnifait, P.: Extrinsic calibration between a multi-layer lidar and a camera. In:
2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent
Systems (2008)
4. Garca, F., Jimnez, F., Naranjo, J.E., Zato, J.G., Aparicio, F., Armingol, J.M., de la
Escalera, A.: Environment perception based on LIDAR sensors for real road applications.
Robotica 30, 185193 (2012)
5. Garca, F., Garca, J., Ponz, A., de la Escalera, A., Armingol, J.M.: Context aided pedestrian
detection for danger estimation based on laser scanner and computer vision. Expert Syst.
Appl. 41(15), 66466661 (2014)
6. Kwak, K., Huber, D.F., Badino, H., Kanade, T.: Extrinsic calibration of a single line
scanning lidar and a camera. In: IEEE/RSJ International Conference on Intelligent Robots
and Systems, pp. 32833289 (2011)
7. Li, Y., Ruichek, Y., Cappelle, D.: 3D triangulation based extrinsic calibration between a
stereo vision system and a LIDAR. In: 14th International IEEE Conference on Intelligent
Transpprtation Systems, pp. 797802 (2011)
8. Li, Y., Liu, Y., Dong, L., Cai, X.: An algorithm for extrinsic parameters calibration of a
camera and a laser range nder using line features. In: IEEE/RSJ International Conference
on Intelligent Robots and Systems (2007)
9. Lisca, G., Jeong, P.J.P., Nedevschi, S.: Automatic one step extrinsic calibration of a multi
layer laser scanner relative to a stereo camera. In: 2010 IEEE International Conference on
Intelligent Computer Communication and Processing (ICCP) (2010)
10. Martn, D., Garca, F., Musleh, B., Olmeda, D., Marn, P., Ponz, A., Rodrguez, C.H.,
Al-Kaff, A., de la Escalera, A., Armingol, J.M.: IVVI 2.0: an intelligent vehicle based on
computational perception. Expert Syst. Appl. 41, 79277944 (2014)
11. Premebida, C., Ludwig, O., Nunes, U.: LIDAR and vision-based pedestrian detection
system. J. Field Robot. 26, 696711 (2009)
12. Premebida, C., Ludwig, O., Silva, M., Nunes, U.: A cascade classier applied in pedestrian
detection using laser and image-based features. In: Transportation, pp. 11531159 (2010)
13. Rodrguez-Garavito, C.H., Ponz, A., Garca, F., Martn, D., de la Escalera, A., Armingol, J.
M.: Automatic laser and camera extrinsic calibration for data fusion using road plane (2014)
14. Sick, LD-MRS Manual. SICK AG Waldkirch, Reute, Germany (2009)
15. Spinello, L., Siegwart, R.: Human detection using multimodal and multidimensional
features. In: IEEE International Conference on Robotics and Automation, pp. 32643269
(2008). doi:10.1109/ROBOT.2009.4543708
16. WHO, Global status report on road safety. Time for action. WHO library
cataloguing-in-publication data, World Health Organization, Geneva, Switzerland (2009).
ISBN 978-9-241563-84-0
17. Zezhi, C., Pears, N., Freeman, M., Austin, J.: Road vehicle classication using support
vector machines. 2009 IEEE Int. Conf. Intell. Comput. Intell. Syst. ICIS 2009 4, 214218
(2009). doi:10.1109/ICICISYS.2009.5357707