2016 International Conference on Emerging Technological Trends [ICETT]
Automated Road Distress Detection
Lucy Powell
Department of Electronics and Communication
Engineering, Amal Jyothi College of Engineering,
Kanjirapally, Kottayam, India
lucypowelljoseph@gmail.com
Abstract— Potholes, cracks and patches are a few types of
road surface distresses which are to be assessed India. Presently,
road distress data assessment is reported to be done through
distress data collection and processing the collected data. The
data analysis and collection are to be automated these days, since
manual assessment is expensive, time consuming and slows down
the road maintenance management. The presence of shadows has
been responsible for reducing the reliability of the system due to
wrong detection results. Therefore, shadow detection and
removal is important for improving performance. Many
techniques have been proposed over the years, but shadow
detection still remains an extremely challenging problem,
particularly from a single image. In this paper, a robust method
for automated detection and assessment of potholes, cracks and
patches from images of Indian local roads is proposed, where
effect of shadow is rectified. For testing its performance, the
proposed method has been implemented using MATLAB. The
results are evaluated through accuracy and precision recall
metrics and compared with the methods presented by earlier
researchers as well as current practices in the field.
Keywords—shadow removal; surface distress; thresholding
I. INTRODUCTION
Of all the means of transport, roadway carries almost 70%
of the volume. Roads are not just important for human
conveyance, but also transport of heavy loads which include
all forms of matter. Road maintenance, hence, is a great
concern to the Public Works Department or similar
departments of every country. Road maintenance is essential
in order to pre-serve the road in its originally constructed
condition, protect adjacent resources and ensure user safety,
and provide efficient and convenient travel along the route.
Unfortunately, often maintenance is neglected or improperly
performed resulting in rapid deterioration of the road and
eventual failure from both climatic and vehicle use impacts.
Therefore it is impossible to build and use a road that requires
no maintenance. In order to plan for road maintenance needs,
it is important to keep a complete set of built plans [2] and
records of all maintenance operations and observations. The
road surveys carried out are biased to the fact that lighting
conditions vary during and the results of images taken at one
time are different from those taken at another. Hence
generalization is a difficult task. Moreover as far as Indian
roads are concerned, either sides of the roads are planted with
trees which cast shadow on the roads almost the whole day.
The challenge in implementation of a fully automatic system
for surface distress detection is that at times these shadows are
978-1-5090-3751-3/16/$31.00 ©2016 IEEE
Prof. KG Satheeshkumar
Department of Electronics and Communication
Engineering, Amal Jyothi College of Engineering,
Kanjirapally, Kottayam, India
kgsatheeshkumar@amaljyothi.ac.in
wrongly detected as distresses and gets added up to the survey
results.
The aim of the system introduced here, the first of its kind
to look into the issues of shadow removal in road surveys, is to
automatically detect road distresses and patches on the road, in
images captured from the pavement surface. The input to the
system is a single pavement image. The objective here, is to
propose a solution which is cost effective and time saving.
Therefore, everyday users of the roads become the sensors or
data collection part of the system. The cameras used here,
have large fields of view, around 120 to 170 degrees.
According to Bennett & et.al., data collection which is the
most important module of the system needs most attention.
According to the literature a few considerations in data col-
lection include how much data needs to be collected, only
relevant data needs to be collected, the devices used for data
collection should just suite the requirement because expensive
high end devices may not always be required for conducting a
road survey. The road management information according to
Bennett and Peterson (2000) has five Information Quality
Levels(IQL), which are IQL 1-5 explained in [3]. To make the
data helpful in making managing decisions, it must be referred
with location specifications, which can be spatial, one which
contains set of coordinates and linear, consisting of distance
and direction from a known point. The data collection
methods are expensive, hence only the required data for
management process must be collected. Dynamic measuring
devices for surface distress evaluation, roughness evaluation
and, in some instances, texture measurement may be required
depending upon the type of data the surveyor intends to
collect. In current practices, portable equipment are installed
in local vehicles and used to collect a range of data through a
single pass of a multifunctional vehicles. Data should be
properly referenced by using a good referencing system,
which ideally combines linear and spatial measurements
where, practical video logging is desirable.
Much research has been performed for overcoming the
limitations of current practices and automating pavement
defect detection. Extensive research has been done with
respect to the defect of cracks. Methods that perform crack
detection, real-time crack analysis, crack classification, crack
depth estimation and even automated crack sealing have been
proposed. Detection of other surface defects such as patching,
potholes and raveling in 2D images has also been investigated.
One such method for evaluating and rating the road surface
distress is based on Support Vector Machine(SVM)
 2016 International Conference on Emerging Technological Trends [ICETT]
classification [4]. In this method the image captured is into
square blocks and SVM trained and tested using feature
vector. However this method proposed by M Greorgios &
et.al., is limited to patches only. R Kapela & et.al., [5]
considered the fact that cracks require automated detection
and classification, hence require efficient computer
algorithms. Here the data is collected using cameras facing
downwards positioned orthogonal to the pavement axis. The
system requires a large number of data for initial system
Fig.1. Pothole Detection
training. The method based on block based analyses was
proposed by H Oliveira & et.al., [6]. The method used a
Portuguese Distress Manual for classifying the detected crack.
This method like the previous one requires large data set for
system training, which is a time consuming process. Objective
of the research proposed by Brilakis et.al., in [7], is to test
whether a ubiquitous and comprehensive framework can
detect, recognize, spatially locate and evaluate the magnitude
of defects on pavements from a network of passenger vehicles.
The motivation lies in the deficiencies of current practice and
the potential of gradually and inexpensively converting
passenger vehicles into ubiquitous sensors and reporters of the
road’s condition. The general flowchart for pothole detection
is shown in fig 1 is derived from [7]. Here, the parking camera
is replaced by high speed fish eye camera that can tilt
downwards during driving forward. The processing by an in-
car computer has two stages. First stage contains application
of high speed algorithms for detecting frames that might
contain evidence of defects. Then apply defect detection
algorithm on those frames selected in first stage. According to
the geometric properties of a defect region the potential
pothole shape is then approximated. This methodology can be
extended to crack detection as well.
S. Battiato & et.al., [8] considers nonlinear image analysis
where the data collection is semi-automatic. The image is
rectified for the geometric distortions and then preprocessed.
The system analyses both patch and crack but fails to classify
it. Moreover it does not address the issues due to presence of
shadow in the image. Another method aiming at detecting
potholes, tracking them in videos and measuring their proper-
ties has also been proposed .
Proper choice of thresholds play an important role in the
detection of any distress on the road surface. There are 3
thresholding methods [9] which can be used and they are
strongly related to the standard deviation. These methods are
neighboring difference histogram methods which use the
advantage that the distressed pavement images are darker than
the surroundings. The pixels are basically divided into two
classes which are the object and background. The various
methods suggested here are Otsu method which is a between
class variance method, Kapur method which obtains maxi-
mum information between the object and background and
Neighboring Difference Histogram Method (NDHM) which
uses the weighted difference statistical value of each cracking
pixel with their surrounding pixels. These methods are time
consuming and becomes complex as the features in the image
increases. Also these methods are useful if the images have a
bimodal histogram.
Shadow removal is all about recovery of the intrinsic
image by using the illumination and reflectance components
of the image. From the methods proposed by Finlayson et. al.
[11], [12], processes illumination invariant image for shadow
removal & removes the edge as well which are assumed to be
lying in the boundary of the shadow region. Zeroing the
corresponding gradient and reintegration yields a shadow free
region. These methods assume that within the shadow region
the illumination is constant and the gradients are entirely due
to changes in reflectance which does not account for the self-
shadowing caused by the small scale geometry on the surface
and for the changes in the amount of ambient illumination that
might occur across the shadowed region. Hence, small error in
any of the process will be visible in the results. In the shadow
editing tool proposed by Mohan et al. [13], edges of the
shadow region are ignored. Automatic shadow removal is not
possible in this system because it involves the user to select
the shadow regions.
In short, the above mentioned methods are practically time
consuming, expensive in terms of equipment used and biased
to an extent. Therefore, we propose an algorithm for images
which can be extended to video, where the distress detection is
made fully automated. This is accomplished by using the
parking camera of a car for data collection, which reduces the
data collection cost.
 2016 International Conference on Emerging Technological Trends [ICETT]
II. PROPOSED SYSTEM
We introduce a system that aims to develop a fully
automatic system which can analyze a road before and after
maintenance, designed to be cost effective owing to the less
expensive devices used and speed up the process by reducing
the image size taken for processing. The system proposed here
in fig 2 is first of its kind where road maintenance is made
completely automatic and it includes shadow handling as well.
Fig.2. Flow Chart
In the detection method proposed, to make sure that it works
and have value, the viewport needs to depict intact pavement,
patch and other defects as well. To accomplish this, a few
considerations may be made. For example, if an image
contains only a patch, the system may not be able to detect it,
because there is nothing to differentiate it from. The situation
is similar when looking with naked eye, where one can’t
identify a patch to be a newly laid pavement or a patch in real.
Also, if the patch is wide enough to reach the lane markings,
then, it is no more a patch but a pavement repair.
A. Overview
The common road distresses include cracks and potholes.
The main causes of these distresses owe to the traffic on the
roads and the climatic conditions. Heavy rains damage the
roads to a great extent and cause almost 50% of the damages.
The proposed system analyses the road images captured and
looks for the presence of distresses in it. According to the
severity of the distress the surveyor can decide the type of
maintenance work to be carried out. Any image input to the
system is converted to gray scale because as far as road
images are concerned, all relevant information are more
towards the lower pixel values. Hence colour information has
no much importance here. To reduce the processing
complexity, and to speed up the process, gray scale images are
used. The gray scale images are median filtered. Median filters
are used because the edges of the images need to be preserved.
The noises in these images are mostly salt and pepper.
The filter used here is a Gaussian filter of size 3X3. The
image is converted to binary using an adaptive thresholding
method based on weighted means method. Threshold is
calculated using local weighted mean on rows or pair of rows.
Gaussian smoothing is used to calculate the local mean. Next
step is morphological image closing. Morphological closing
include dilation and erosion. The resultant images are used for
making mask for defining the boundaries of the distress
region. The above mentioned mask is used for detection and
definition of the distress. The area of each distress is
calculated. The distress regions with area less than 177cm2 are
considered to be irrelevant owing to that fact that oil, water or
even tar spills can be detected as distress regions requiring
attention. These are filtered out and only the relevant pixels
carrying the distress information are retained and put in
bounding box after classifying them as crack, pothole or patch.
The classification is made by defining certain thresholds for
the parameters like circumference or width of each distress
detected [1].
In any case there is a shadow in the image captured, the
results vary. The shadow regions may be wrongly detected as
a patch or sometimes even a pothole. To overcome this, the
system uses a shadow removal mechanism. Shadows are the
regions with minimum pixel values in an image. This property
is utilized for removing them.
Shadow removal as suggested in [14] , discusses various
light models, each of which are rectified in different ways.
The light model used here is basiclightmodel. In the shadow
removal algorithm used here the ground image is analysed.
Image is in RGB space, with colour values between 0 and 1.
The image contains one homogeneous texture in part of the
shadow. The algorithm used here, finds the lower intensity
areas in the picture, and if these are textures or objects that are
not related to shadows it leads to false detection. In such
situations, the decision is made by considering whether border
is smooth or not. Because, the border is usually rough for
objects. The result of the shadow detection is a binary shadow
mask, which will be the input to the shadow removal
algorithm. The shadow removal is accomplished by using
equation (1) where, r is the ratio between the average lit area
and average shadow area. The shadow removal for basic light
model is shown in fig 3.
result = true image (1 - mask) + mask × r × true image (1)
 2016 International Conference on Emerging Technological Trends [ICETT]

Fig.3. Shadow removal for basiclightmodel

TABLE I
DISTRESS PARAMETERS
Distress Standard Deviation Circumference
Pothole >22 <10
Patch >10 •10
Crack >10 <10

Shadow, in case present is removed after detecting the

surface distress. The shadow is removed from the ground
image and the mask created for distress detection is used on
the shadow removed image to give the final result.

III. EXPERIMENTS AND RESULTS

The proposed method was implemented on MATLAB
R2014a. The image processing was done on a laptop Intel
Corei3 CPU 1.90Ghz, and 4GB RAM. The database collected
for the implementation of the proposed algorithm which is
derived from [1], is from the local roads of Anakkal, Kanjira-
pally town, Kerala. The data collection was performed under
different conditions, like, sunny day, cloudy day, etc. The
processing results of images collected on rainy days show that
rainy days are not suitable for conducting a survey using the
proposed system. An iPhone5 with 5MP camera placed at an
approximate height of 1m from the ground was used for
capturing the images for the experiments. The test samples
contain features such as normal roads, distress like potholes,
cracks, patches, discoloration, dark spots and shadows of
trees, and objects.
The output for various stages of the system is shown in fig.
4. Once the distresses are detected they need to be classified to
their respective categories. This is based upon the following
table I, assuming that the image size on paper is 680 480.
These values are derived from the experiments done on 150
images captured from the same road on different days and
different times.
A. Effect of Shadow
To analyse the effect of shadow the images were taken on
different days at different times, i.e., from 0900hrs to 1700hrs.
On processing these images one can see that, the results of
each process vary depending on the light. The angle of the
camera and its height from the ground are important factors
Fig. 4. Crack Detection System
that affect the system output. Keeping these factors constant
i.e., setting the camera at a fixed height of 1m from the ground
and the angle of inclination at 45 degrees with the ground,
only the variation in the results with the light is considered
while conducting the experiment. Analysis of these images
group them into three, 1) Group A: images taken before
1100hrs, 2) Group B: images taken between 1100hrs and
1400hrs, and 3) Group C: images taken after 1400hrs.
The analysis of these images can be done using the
precision recall metric can be seen in table II, where precision
and recall are given as, in equation (2) and equation (3), where
TP is the correctly detected distresses, FP is those distresses
that were not detected and FN is the wrong detection results.
Precision = TP/(TP+FN) (2)
Recall = TP/(TP+FP) (3)
From the above analysis we can see that, Group B images are
the images give the most accurate results. Thus, the best time
suited for conducting the survey is between 1100hrs and
1400hrs. One can conclude that the requirements for
conducting a survey that gives proper results for further
analysis is a camera set up at a height of 1m from the ground
inclined at an angle of 45degrees from the ground. The ideal
conditions can be concluded as sunny day between 1100hrs
and 1400hrs. The result of such conducted experiment is
shown in fig 5.
 2016 International Conference on Emerging Technological Trends [ICETT]
Fig. 5. Ideal result
IV. IMPLEMENTATION CONSTRAINTS
Shadow removal from single image has always been a
concern. This is because inclusion of shadow removal into the
crack detection system resulted in decrease of the system
accuracy. This is due to the errors in the pixel replacement.
The shadow regions get replaced by the surrounding healthy
pixel values. These values are not always reliable because
these pixels may contain information like plants and other
objects in the surrounding. So the shadow removal part of the
system needs to be modified first and generalized making it
suitable for all images. Even then the mask creation may have
errors. The challenge here was to include the modification in
such a way that the system accuracy does not get affected
much. Hence shadow removal was done on the ground image
and a mask with shadow removed is created. This mask is
combined with the mask created for crack detection and thus
we have the shadow removed output.
Another challenge especially on Indian roads is that the
pot-holes may be filled with water. Such potholes were
detected as patches. This issue needs to be addressed as a
future scope of the system.
V. CONCLUSION
The proposed automated system, is a novel approach that
takes the effect if shadow into consideration for a road survey.
The system is faster compared to other existing system and
moreover is fully automated, thus reducing the job on the
surveyors’ part. Since the equipments used are easily
accessible and not very expensive, the cost of conducting the
survey reduces. The data collection is so simple than anybody
who has a mobile phone with a camera with it can conduct the
TABLE II
RESULT ANALYSIS
Group No. of images TP FP FN Precision Recall
A 30 3 15 12 0.2 0.16
B 50 27 10 13 0.675 0.73
C 20 10 4 6 0.625 0.714
survey. Thus reducing the cost of a dedicated vehicle and
cameras. The cost can be further reduced if the system is
modified to work on parking cameras of vehicles. By
determining the ideal time for conducting the survey, one can
avoid multiple efforts. The system finds application in
conducting road surveys for maintenance management and
deciding the usability of a road. The system attains an
accuracy of 83% without shadow removal module. However
inclusion of shadow removal into the system affects the
accuracy and performance. The execution time of the same
image with shadow removal increases by 10 seconds. Also the
system accuracy falls to 78%. Therefore further enhancements
to the system can include shadow removal methods which are
faster in terms of processing time and accurate as well. The
system can be modified to work for raw video taken from the
images. By determining a severity level, the analysis can be
used to determine, whether the road requires immediate
maintenance or whether the defects are not that severe.
References
[1] Lokeshwor.H, Sud S.K, “Method for automated assessment of potholes,
cracks and patches from road surface video clips”, Procedia - Social and
Behavioral Sciences 104 (2013), pp. 312 321
[2] FAO road corporate repository, “Road Maintenance”, Chapter 7
[3] Bennett, C.R., Chamorro, A., Chen, C., Sominihac, H.D., & Flintsch,
G.W. “Data collection for road management” (version 2.0). East Asia
Pacific Transport Unit, Washington: The World Bank (2007)
[4] M Georgios, E Symeon, Patricio A. Vela, “Automated detection of
pavement patches utilizing support vector machine classification”, Elec-
trotechnical Conference(MELECON) 2016, Apr 2016
[5] R Kapela, P Sniatala, A Turkot, “Asphalt surface pavement crack
detection based on histograms of oriented gradients”, Mixed Design of
Integrated Circuits & Systems (MIXDES), 2015 22nd International
Conference, Jun 2015
[6] H. Oliveira, PL Correia, “Automatic Road Crack Detection and
Characterization, IEEE Transactions on Intelligent Transportation
Systems, Volume: 14, Issue: 1, March 2013
[7] Koch, C., Brilakis, L. “Pothole detection in asphalt pavement images”.
Advanced Engineering Informatics, Volume 25, Issue 3, August 2011,
pp. 507-515
[8] S. Battiato, S. Cafiso, A. Di Graziano, L. Rizzo, F. Stanco, “Pavement
surface distress by using non-linear image analysis techniques”,
Minisymposium: Image Analysis Methods for Industrial Applications,
2006
[9] Q. Li, X. Liu, “Novel approach to pavement image segmentation based
on neighboring difference histogram method”, Image Signal Process.
2008 CISP08 Congr. On, 2008, pp. 792796
[10] Yael Shor, Dani Lischinski, “The Shadow Meets the Mask: Pyramid-
Based Shadow Removal, EUROGRAPHICS 2008 / G. Drettakis and R.
Scopigno, Volume 27 (2008), Number 2
[11] Finlayson G. D., Drew M. S., LU C. “Intrinsic images by entropy
minimization”, Proc. ECCV (2004), vol. III, pp. 582595
 2016 International Conference on Emerging Technological Trends [ICETT]

[12] Finlayson G. D., Hordley S. D., LU C., Drew M. S. “On the removal of
shadows from images”, IEEE Trans. Pattern Anal. Mach. Intell. 28, 1
(Jan. 2006), 5968
[13] Mohan A., Tumblin J., Choudhury P., “Editing soft shadows in a digital
photograph”, IEEE Comput. Graph. Appl. 27, 2 (2007), 2331
[14] B. Corina, JK Peter, B. Zoltan, V Laszlo, “Shadow detection and re-
moval from a single image”, 19th Summer School in Image Processing,
July 2011