Review

Chronological development history of XY table based pavement crack sealers and

research ndings for practical use in the eld
Young S. Kim
a
, Hyun S. Yoo
a
, Jeong H. Lee
b,
, Seung W. Han
a
a
Department of Architectural Engineering, Inha University, South Korea
b
Center for Cost Engineering Research, Inha University, South Korea
a b s t r a c t a r t i c l e i n f o
Article history:
Accepted 19 February 2009
Keywords:
Automation
Pavement
Crack sealing
Machine vision
Robotics
Teleoperation
During the last two decades, several teleoperated and machine-vision assisted systems have been developed
to automate the overall process of routing and sealing pavement cracks. Productivity improvement, improved
safety and quality, and reduced road user costs have motivated these developments. This paper presents the
chronological development history of xy table based pavement crack sealers, which have been developed
and demonstrated since the early 1990s, and compares their technical advances. This paper also discusses
primary research ndings in machine vision software and hardware designs of an automated pavement crack
sealer to be newly developed for practical use in the eld. Finally, conclusions and recommendations are
made concerning the value of implementing and practically using the automated pavement crack sealer.
2009 Elsevier B.V. All rights reserved.
Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
2. Automation needs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
3. Automated pavement crack sealing systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
3.1. Chronological development history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
3.2. Evolution of the control paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
3.3. Evolution of machine vision software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
3.4. Evolution of hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516
4. Primary research ndings for practical application in the eld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
4.1. Software design requirement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
4.1.1. Path planning (full automation) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
4.1.2. Graphical user interface design for crack sealer control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
4.2. Hardware design requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
4.2.1. Types of crack to be sealed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
4.2.2. Crack image collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
4.2.3. Entire crack sealing system architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
4.2.4. Manipulator and end-effector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
5. Conclusions and recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
1. Introduction
Crack sealing, a routine and necessary part of pavement main-
tenance, is a dangerous, costly, andlabor-intensive operation. During the
last two decades, several systems based on xy table for automatically
Automation in Construction 18 (2009) 513524
Corresponding author. Center for Cost Engineering Research, Inha University, 253
Yonghyun-dong, Nam-gu, Incheon, 402-751, SouthKorea. Tel.: +8232 872 9757; fax: +82 32
866 4624.
E-mail address: inhacmr@hotmail.com (J.H. Lee).
0926-5805/$ see front matter 2009 Elsevier B.V. All rights reserved.
doi:10.1016/j.autcon.2009.02.007
Contents lists available at ScienceDirect
Automation in Construction
j our nal homepage: www. el sevi er. com/ l ocat e/ aut con
routing and sealing pavement cracks have been developed. Examples
include: 1) CMU laboratory prototype (1990) [1,2], 2) CMU-UT eld
prototype (1992) [2,3], 3) UT Automated Road Maintenance Machine
(ARMM) (1997) [36], and4) AutomatedPavement Crack Sealer (APCS)
(2004) [7,8]. Since automating pavement crack sealing can improve
safety, productivity, and quality, and also reduce road user costs, there
has been extremely large demand for practical use of automated
pavement crack sealers in the areas of road construction and
maintenance. While early works sought to completely automate crack
mapping and sealing activities, experience and the resulting deeper
understanding of the enabling technologies have highlighted the
importance of nding a desirable balance between human and machine
functions in the control of automated pavement crack sealers.
Through trial and error and about 20 years of perseverance, the
APCS, the most recent deliverable research, has achieved a desirable
balance between manual and automated functions for automated
pavement crack sealing. Recent eld trials of the full scale APCS have
also indicated that automated pavement crack sealing is now
technically, economically, and nancially feasible. Despite such
numerous efforts to automate conventional crack sealing operations,
lessons learned from previous system developments and eld trials
have indicated that several improvements in both software and
hardware designs are still required for their practical application in
the eld. The primary objective of this paper is to present overall
software and hardware design requirements for practical use of an
automated pavement crack sealer in an effort to fulll the aforemen-
tioned demand in road construction and maintenance. This paper
presents the chronological development history of xy table based
pavement crack sealers, which have been developed and demon-
strated since the early 1990s, and compares their technological
advances. This paper then proposes primary research ndings in
machine vision software and hardware designs of an automated
pavement crack sealer to be newly developed for practical use in the
eld. A conceptual hardware design for a new model is proposed in
this paper as well. Finally, conclusions and recommendations are
made concerning the value of implementing and practically using the
automated pavement crack sealer.
2. Automation needs
Crack sealing is a maintenance technique commonly used to
prevent water and debris penetration and reduce future pavement
degradation. The conventional crack sealing operations are, however,
dangerous, costly, and labor-intensive. With respect to crack sealing
crews, labor turnover and training are also increasing problems.
Automation of the crack sealing process would improve productivity
and quality, and offers safety benets by getting workers off the road.
The reduction in crew size and the increase in productivity of the
automated sealing process are expected to be translated directly into
signicant cost savings.
For example, APCS eld test results indicated that the daily
productivity would be 1.59 km/day. Compared with the productivity
of a conventional crack sealing method (1.21 km/day), that of the
APCS was as much as 0.39 km/day higher. On-site tests and a
performance analysis of the APCS demonstrated that its use would
allow a 50% reduction of the labor force and 32.5% enhanced
productivity [8]. Furthermore, when considering nighttime operations
and possible hardware improvements of the developed APCS, the
productivity of the APCS would be even higher.
The results of an economic feasibility analysis of the APCS also
revealed that automating the conventional crack sealing operation is
highly feasible and would potentially bring enormous cost savings.
Information for the analysis was gathered to estimate costs and
benets, analysis perspectives were chosen, and the market was
studied. Rate of return, benetcost ratio, break-even point, and
sensitivity analyses were used to verify the economic feasibility of
implementing the automated method in place of the conventional
method. Under assumptions such as an APCS purchase cost of US
$72,000, 100 working days per year, use of 1 APCS, 10% MARR, a
10 year planning horizon, a 50% reduction in labor force, etc., it was
anticipated that a contractor would be able to cut conventional
maintenance costs by 43.6%. With the above assumptions, the
economic analysis results of the APCS also showed a value of 122.5%,
5.5, a 15 month in rate of return period, benetcost ratio, break-even
point, respectively, thus making the use of APCS highly attractive. The
results of a sensitivity analysis and predictions pertaining to reduction
of road user costs obtained using Paramicssimulation software have
been presented elsewhere [8].
3. Automated pavement crack sealing systems
In this chapter, xy table based systems developed since the early
1990s for automatically routing and sealing pavement cracks are
briey described. Technological details of the systems are presented,
and related research accomplishments, concerns, and technical
advances are identied. Visual appearances of each prototype system
are illustrated as well.
3.1. Chronological development history
Table 1 briey describes the accomplishments and major limita-
tions of previous research works. The hardware of early xy table
based pavement crack sealing systems was incomplete in the early
stage. At the same time, the software for mapping and modeling the
crack network to be sealed and path planning were not efcient in
terms of productivity, quality, and accuracy for practical use in the
eld. In addition, the hardware and software were not integrated
properly, causing inaccurate and inefcient movement of the auto-
mated pavement crack sealing systems. The experience acquired from
such early attempts and recent advances in relevant robotic
technologies motivated authors to develop more advanced pavement
crack sealer (APCS). Although the APCS employs a unique man-
machine interfaced control process and provides innovative technical
advances compared to previous research works [16,10], they have
not been practically used on sites due to limitations in their hardware
and software. Detailed comparisons and several limitations regarding
the control paradigm, machine vision software, and hardware in the
xy table based pavement crack sealers developed in previous
research works are presented in Sections 3.23.4.
3.2. Evolution of the control paradigm
Complete autonomy [1,3] could be achieved for the whole process
(image acquisition, crack detection and mapping, path planning,
blowing, sealing, and squeegeeing) of automated pavement crack
sealing, but usually at a cost, speed, and accuracy that is impractical
and unacceptable. Complex evolution of the control paradigm has
resulted in a functional production prototype system [46,9,10] that
achieves a good balance between manual functions and automated
functions by taking advantage of the respective strengths of man and
machine in the whole process. Teleoperation based on remote video,
man-machine interfaces, machine vision, and graphical programming
alone can achieve benets of automation in the unstructured
pavement crack sealing work environment. Lessons learned from
system developments and eld trials have also indicated that
computer assistance in the form of man-machine interfaced graphical
programming and machine vision is essential and can cost-effectively
help to achieve improvements in the productivity and quality of
automated methods. Considering recent successful developments in
teleoperated construction eld robots, it is thought that evolution
toward teleoperation as a control paradigm of the automated
pavement crack sealer is highly desirable. The ARMM and the APCS
514 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
employ some forms of man-machine interfaced control process for
automatically sealing pavement cracks.
3.3. Evolution of machine vision software
In general, machine vision software in automated pavement crack
sealers can be classied into the following four categories:
Image acquisition
Inpreviously developedpavement crack sealers, a computer imaging
system is typically used to view cracks to be sealed on the surface. The
use of remote video cameras is widely employed to provide visual
feedback in the machine. To capture and digitize video images of the
cracks in the machine's work space, a commercial image processing
board is installed in the PC that controls the automated crack sealers.
One or two security cameras mounted on a super-structure of the XY-
table manipulator ((1)(4) in Table 1) acquire live pavement surface
images displayed on the PC's monitor.
Initial crack detection, mapping, and representation
To automatically seal cracks, exact crack locations are the most
important information. Indeveloping a newimage processingalgorithm
for automaticallysensing, mapping, andrepresenting cracks to be sealed
in pavements, the quality of the segmentation must take the highest
priority. The overall efciency of the algorithm is also important for
project success, if the cracks are automatically detectedandmappedbya
computer.
In CMU laboratory and CMU-UT eld prototypes, a video camera is
used to acquire images. These images are then digitized and subse-
quently combined with laser range data of surface contours using a
specially designed multi-layer quadtree model and image analysis
algorithms, via the process of sensor fusion [13]. However, it was
concluded that detection and mapping pavement cracks to be sealed by
computer functions alone may not be desirable. This is due to the fact
that most pavement cracks are highly noisy and unstructured in their
digitized format the pixel level. The lowcontrast between the distresses
and the background also further complicate the pavement analysis. As a
result, the lessons learned revealed that a human operator should
interact directly withthe computer inthe crack detection, mapping, and
representation processes, as in the ARMM and the APCS.
As an example, such a man-machine interface was rst tested to
detect, map, and represent pavement cracks to be sealed in the ARMM
[46,9,10]. Under proper conditions of illumination, humans can see
color, brightness, and form, thus easily distinguishing real cracks from
pavement background noise such as sealed cracks or oil marks or skid
marks. As such, allowing the operator to point out the existence and
locationof pavement cracks usinga stylus ona touchsensitivemonitor is
an effective approach. A graphical program can be used to generate a
computer-based model of the surface cracks to be sealed and to provide
visual feedback to the system operator. A good model can be achieved
since the work environment of the automated pavement crack sealer is
fairly static. Despite such advantages, this approach has a drawback
that can signicantly degrade the quality of the resultant seal. The
imperfection of human hand-eye coordination causes errors even in a
good work environment. Also, operator arm fatigue can increase such
errors whentracingcracks onthe monitor. Inthe ARMM, machine vision
based line snapping or manual editing was thus required to compensate
human hand-eye coordination errors [5,6]. In a subsequent research
effort, a similar man-machine interface was applied to the ARMM for
detecting, mapping, and representing the pavement cracks to be sealed.
However, in the APCS, crack network detection, mapping, and
representation are operated in two modes: full automation and
semi-automation. A unique and innovative machine vision algorithm
has been developed for the automation of pavement crack sealing.
Detailed descriptions of both fully and semi-automated crack network
detection, mapping, and modeling algorithm have been presented
elsewhere [9].
Path planning
Compared to conventional crack sealing operations, the efciency of
movement (path plan) of an automated crack sealer is a key factor with
regard to performance. Efciency of movement is governed by the
length of the path that the end-effectors followto cover the whole crack
network. Other performance factors include manipulator speed and
accuracy, and the duration of crack detection, mapping, and representa-
tion. If cracks are fully monitored and mapped by the human naked eye,
a pathplanis not required. However, if cracks are mapped manually ona
computer monitor, animplicit path plan is generated by the sequence of
strokes made by the operator, including the beginning and end points.
Meanwhile, automated path planning is necessary when cracks are
automatically mapped by the computer. In the CMU-UT eld prototype
and the ARMM, a series of greedy path planning algorithms using a
double linked data structure and an array data structure were proposed
in an effort to automatically generate a path to effectively traverse the
crack network to be sealed[10]. The test results showedthat, onaverage,
the time taken to compute a greedy path was much less than the time
saved by executing the greedy path. Recently, optimal and more
advanced greedy path planning algorithms were successfully developed
and applied to the APCS [7]. In an effort to minimize the idle path in the
Table 1
Previous research works related to the xy table based pavement crack sealing systems.
Previous research works Accomplishments and major concerns
(1) CMU laboratory
prototype (1990)
Conceptual design of S/W and H/W for
full automation
Crack detection, mapping and
representation using digital image
processing
Incomplete and unstable fabrication of
H/W(XY manipulator)
Excessive processing time in the crack
detection, mapping and modeling
(2) CMU-UT eld
prototype (1992)
Development of a machine vision
algorithm using sensor (vision and
laser range scanner) fusion
Development of path planning algorithm
using linked data structure
Fabrication of more robust H/W
(XY-manipulator) and partial integration of
S/W and H/W
Partial verication of fully automated
crack sealing process
Excessive processing time due to slow
range scanning
(3) ARMM (1997) Suggestion of a man-machine balanced
control process for effectively sealing
pavement cracks
Development of a machine vision
algorithm (manual mapping, line snapping
and manual editing) using graphical
programming and user-friendly control
software
Development of greedy path planning
algorithm using array data structure
Need to improve the design of turret and
machine vision algorithm [2]
(4) APCS (2004) Suggestion of a man-machine balanced
control process for effectively sealing
pavement cracks
Development of an innovative machine
vision algorithmwhich can both
automatically and semi-automatically seal
pavement crack network using user-friendly
graphical control software
Development of greedy and optimal path
planning algorithms
Nighttime operation using lighting
system mounted on the super-structure of
the machine
515 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
crack network to be traversed, as well as to maximize the efciency
(productivity) of the APCS, the developed optimal and greedy path
planning algorithms are intelligently and selectively used, based on the
complexity of crack networks to be traversed in a given workspace.
Table 2 briey summarizes the technical advances related to the control
paradigm and machine vision software made in previous research
works.
3.4. Evolution of hardware
In previous research works, the system architectures for the xy
table based pavement crack sealing consisted of multi-units (tow
truck, sealant melter, and crack sealer). The CMU laboratory proto-
type, CMU-UT eld prototype, the ARMM, and the APCS use multi-
units as shown in Table 1. XY-table manipulators with a turret
structure as an end-effector and multi-DOF manipulator arms were
employed as the main bodies of the crack sealing hardware. Table 2
compares the technical aspects of the previous automated pavement
crack sealing hardware.
The APCS is an extended version of the ARMM with innovative
improvements to both hardware and software. The hardware of the
APCS is composed of the following three units: 1) a towtruck, inwhich
a systemoperator controls the APCS using a PC; 2) a sealant melter that
supplies 170180 C sealant to the APCS; and 3) a crack sealing unit for
blowing, sealing, and squeegeeing cracks in pavements. The major
components of the crack sealing unit of the APCS are as follows: 1) XY-
manipulator with cart, gantry, turret, and CCD camera, 2) control box,
3) industrial PC with a frame grabber and a touch sensitive monitor,
and 4) power supplies. In designing and fabricating the hardware of
the APCS, the following factors were considered:
(1) Designing an XY-table manipulator that does not twist against
the pavement level and various working situations;
(2) Adding rotating and telescoping functions to the end-effector
(turret structure) to effectively blow, seal, and squeegee the
crack network to be sealed in rutted pavement;
(3) Adding a heating function to the inside of the turret structure to
inject sealant with a certain degree of viscosity without rapid
solidication;
(4) Designing a turret structure to effectively squeegee the sealant
injected into the pavement cracks (turret with a squeezing
device having a V or U shape); and
(5) Mounting a lighting system for night work to minimize road-
user-costs and increase productivity.
Basedon the identiedmajor considerations, the APCS was designed
and fabricated as shown in Fig. 1. The APCS uses an XY-manipulator with
a rotating turret to blow, seal, and squeegee pavement cracks in one
sealingtravel. While themanipulator is movingwithinits work space, its
frame is stationary. Sealing cracks in one work area and then moving to
the next work area is considered one work cycle. To control the APCS
through a work cycle, the following six steps are required:
(1) Acquire a pavement image including the crack network using a
digital CCD camera installed on the superstructure of the APCS
and a frame grabber board installed in the PC;
(2) Stop the APCS if any crack network is detected on the operator's
touch sensitive monitor;
(3) Automatically eliminate noises for mapping and modeling the
crack network (This process is required only for the case of fully
automated crack network mapping and modeling, to be
explained in Section 3.3);
Table 2
Comparison result of control paradigm, machine vision software, and crack sealing hardware developed in previous research works.
Control paradigm (1) (2) (3) (4)
Full automation
Teleoperation
Machine vision software (1) (2) (3) (4)
Vision Image acquisition Fully automated (vision sensor)
Fully manual (naked eyes)
Initial crack identication on video image Fully automated
Fully manual (naked eyes)
Crack mapping and representation Fully automated
Semi-automated (MMI)
Manual editing
Fully manual (naked eyes)
Graphical user interface
Path planning Fully automated Optimal
Greedy
Not considered
Hardware (1) (2) (3) (4)
Entire system architecture Single unit
Multi-unit
Manipulator and end-effector XY-table manipulator with turret
Multi-DOF manipulator arm
Independent sealing apparatus
Functions equipped Blow
Seal
Squeegee
Telescope
Heat
Crack image collection Remote video
Laser range scanner
Human driver(sys. operator)
Lighting system for nighttime operation
Types of cracks sealed Transverse cracking
Longitudinal cracking
Block cracking
(1) CMU laboratory prototype, (2) CMU-UT eld prototype, (3) ARMM, (4) APCS.
516 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
(4) Extract the exact spine of the crack network using the
developed crack network mapping and modeling algorithm
(In the APCS, the systemoperator can map and model the crack
network to be sealed in a given work space by both automated
and semi-automated methods selectively);
(5) Automatically perform optimal path planning, based on the
results of mapping and modeling for the crack network; and
(6) Automatically blow air, inject sealant into the crack network,
and squeeze the sealant based on the calculated path plan.
The eld test results showed that the APCS was able to more
intelligently detect cracks through video images, and subsequently
automatically blowdust, inject sealant materials into cracks, and more
efciently squeegee the sealed cracks than the previous version (the
ARMM). Longitudinal, transverse, and block (random) cracks could be
effectively repaired by this machine. Although the current prototype
showed signicant improvements in hardware and software system
development, too much time was required to set up the whole system
including the tow truck, sealant melter, and crack sealing unit. Also, it
was difcult for the driver to precisely control the directional change
of three different units connected sequentially. For practical use under
a more controlled work environment, a newdesign is thus required to
reduce the preparation time and improve the mobility. Table 3 briey
summarizes the advantages and disadvantages of hardware design
used for the APCS. This state-of-the art automated pavement crack
sealing research prototype is the most advanced research deliverable
in the form of xy table manipulator.
4. Primary research ndings for practical application in the eld
It has taken approximately 20 years to achieve a balance between
human and machine functions for automated pavement crack sealing,
instead of pursuing an all-in-one full automation system. Over the
course of developing four physical crack sealing prototype systems
using xy tables (Table 1), a series of unique man-machine balanced
control loops for successful automation of crack sealing have been
developed for computer assisted teleoperation (Fig. 1). A complex
evolution in both software and hardware design has resulted in
functional production prototype system (APCS) that veries the
Fig. 1. Automated pavement crack sealer (APCS).
Table 3
Advantages and disadvantages in hardware design of the APCS.
APCS
Advantages
_
More effective design of end-effector (turret)
_
High accuracy for crack sealing
_
Cracks in pavement with rutting can be effectively sealed.
_
Potential productivity improvement and cost savings due to night
time operation
Disadvantages
_
Poor mobility: three units a tow truck, a sealant melter, and the
XY-table manipulator with turret
_
Only cover the area within the XY table. It may require multi-paths
to cover one full lane, if the workspace of XY-table is small.
_
Only used for technology feasibility demonstration
517 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
prevalence of the automated crack sealing method. The following are
lessons learned and primary research ndings in software and
hardware designs obtained from previous works over the last two
decades.
4.1. Software design requirement
(1) Image acquisition (full automation): As aforementioned, tele-
operation was recommended as a preferred control option for
crack sealing over complete automation. Quite often, the
teleoperation user is in a location that provides either poor or
no visual feedback. A remote camera is therefore generally used
to provide visual feedback. A security or infra-red camera can
be used for automatically acquiring live pavement images. The
images acquired by means of the remote camera are then
digitized by a frame grabber, and the results can be displayed on
the monitor.
(2) Initial crack identication on video images (manual): Since
humans can instantly distinguish real cracks from pavement
background noise such as sealed cracks or oil or skid marks
under proper conditions of illumination, a human operator
should interact in the process of initial crack identication. As
shown in the test results of the most recent xy table based
pavement crack sealer (APCS), lessons learned also show that
initially identication of a crack is better performed by a human
than by a computer in a very time-effective manner.
(3) Crack detection, mapping, and modeling (full or semi-automa-
tion): Automatically detecting, mapping, and modeling cracks in
pavement are desirable and technically feasible. However, there
is considerable concern about the accuracy and time component
of the processed results, because most pavement cracks are
highly noisy and unstructured in their digitized format the pixel
level. Allowing the operator to point out the existence and
location of a pavement crack using a mouse or a stylus on a touch
sensitive monitor is economical and benecial. A graphical
program can be used to generate a computer-based model of
the surface cracks to be sealed and to provide visual feedback to
the operator. A good model can be achieved since the work
environment of the automated pavement crack sealer is fairly
static. Despite such advantages, this approach has a drawback
that can signicantly degrade the quality in the resultant seal.
That is, the imperfection of human hand-eye coordination and
armfatigue when tracing cracks on the monitor can cause errors,
even in a good work environment.
A good model for crack network detection, mapping, and represen-
tationwas proposed for the APCS. In the APCS, crack network detection,
mapping, and representation are operated in two modes: full automa-
tion and semi-automation. A unique and innovative machine vision
algorithm has been developed for automation of pavement crack
sealing. Fully automated crack network detection, mapping, and
representation process successively includes: binarizing, noise elimina-
tion, dilation, edge thinning and linking, and complete crack network
modeling, as shown in Fig. 2. Sixty pavement crack images with shades
and various intensity of pavement surface were experimented to
measure the accuracy (86.6%) and efciency (0.46sec./image) of the
developed algorithm. Fig. 2 illustrates the semi-automated crack
network detection, mapping, and representation process of the APCS
in a very simplied form. Detailed descriptions of both fully and semi-
automated crack network mapping and modeling algorithms have been
presented elsewhere [7,8].
The test results showed that the semi-automated crack network
mappingandmodelingalgorithmwas superior interms of accuracyover
the fully automated crack network mapping and modeling algorithm,
but inferior in terms of productivity. However, both methods could
eventually guarantee 100% accuracy, because a manual editing function
witha rubber banding capability (Fig. 2b) could be used inboth the fully
automated and semi-automated methods. Complex and innovative
technical advances have been made in machine vision-based crack
network mapping and modeling algorithms compared to previous
researchefforts. During eldtrials, it was foundthat the systemoperator
usually uses the automated crack network and modeling method, as its
accuracy was sufcient for practical use. Furthermore, if there were any
errors in the extracted model, the resultant crack network model could
be easily adjusted using the manual editing function. Finally, the authors
estimate that the automated and semi-automated crack mapping and
modeling algorithms and processes developed for the APCS can be
directly applied to the automated pavement crack sealer to be newly
developed.
4.1.1. Path planning (full automation)
If there are several lines in the crack network extracted from a
pavement image, the number of paths is 2
n
n!. In the greedy path
planning algorithm, the nearest node point fromthe home point among
the end node points of each crack network is the start node point for
sealing. The end-effector of the automatedpavement crack sealer injects
sealant alongthe crack network until it reaches the other endnode point
in the greedy path planning algorithm (Fig. 3a). Previous studies
proposed some greedy path planning algorithms for automated
pavement crack sealing. However, the paths planned by such greedy
algorithms often were not the optimal path.
An optimal path planning algorithm that can always provide the
shortest path in a given crack network was developed for the APCS. In
the optimal path planning algorithm, a computer calculates the lengths
of all paths so that the shortest path can be selected (Fig. 3b). Ex-
perimental test results showedthat theoptimal pathplanningalgorithm
is effective if the number of crack lines in a network is 6 or less, and the
greedy path planning algorithm is useful if the number of crack lines in
a network is more than 6, from a computational time perspective. Once
the algorithm automatically identies the number of crack lines in a
given crack network, the path planning method (optimal or greedy
path plan) is then selectively chosen and their x and y coordinates in
2D workspace, which are ultimately required for control of the auto-
mated pavement crack sealer, are generated. Therefore, the path
planning algorithm proposed for the APCS can be directly applied to
the automated pavement crack sealer to be newly developed as well.
4.1.2. Graphical user interface design for crack sealer control
In any man-machine system, an effective and user-friendly design of
the graphical user interfacefor controllinghardware is very important in
terms of nal acceptance in the market. Poor user interface design or
difcult system operation signicantly degrades the system perfor-
mance andproductivity. Most workers inthe eldare seldomexposedto
complicated computer control systems for road construction and
maintenance tasks and labor turnover rate is relatively high. As such,
ease of operation based on a user-friendly graphical interface design
wouldbe a critical factor inattracting prospective end-users as well as in
practically using the man-machine system in the eld. It is also
estimated that overall efciency of the automated pavement crack
sealer will depend as much on the user-friendliness of software and
hardware controls as on the capabilities of the machine vision algorithm
and crack sealing speed.
Findings in previous research efforts have indicated that the method
of clicking graphical menu buttons on a touch sensitive monitor is the
most appropriate design option for effective control of an automated
pavement crack sealer. The graphical user interfaceof the APCSshownin
Fig. 4 is a good example of this. Under such a user-friendly graphical
interface, the system operator should be able to easily understand the
man-machine interfaced pavement crack sealing process and effectively
control the machine. The system operator can also observe the current
working status via the monitor in real time and instantly command
actions to be performed by clicking the menu buttons on the touch
518 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
F
i
g
.
2
.
C
r
a
c
k
n
e
t
w
o
r
k
m
a
p
p
i
n
g
a
n
d
m
o
d
e
l
i
n
g
i
n
t
h
e
A
P
C
S
.
519 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
sensitive monitor. This user-friendly graphical interface design would
alsosignicantly reduce possible problems relatedto labor turnover and
training of crack sealing crews. Therefore, it is anticipated that the
graphical user interface of the APCSshowninFig. 4canbe adoptedinthe
present form, or used as a reference in developing a new model of the
automated pavement crack sealer.
4.2. Hardware design requirements
4.2.1. Types of crack to be sealed
In general, crack patterns existing on pavements are classied into
the following four categories: 1) longitudinal, 2) transverse, 3) block,
and 4) alligator cracks. Longitudinal, transverse, and block cracks are
mainly repaired by sealing, while overlay or patching is used for
repairing alligator cracks. In order to appeal to private contractors and
government departments, as well as for practical use in the eld, it
must be guaranteed that the automated pavement crack sealer can
seal any type of longitudinal, transverse, and block cracks on normal
and/or rutted pavement surfaces.
4.2.2. Crack image collection
As aforementioned, the systemoperator of the automated pavement
crack sealer is in a location (e.g., the tow vehicle's cab) where visual
feedback of the workspace is not available. In this situation a typical
method for providing visual feedback is through the use of remote video
cameras. The video cameras provide several different views of the
equipment in its workspace, as well as the crack sealing status in real
time. This operating environment provides the systemoperator with the
necessary visual feedback to control the automated pavement crack
sealer. The remote video signal is also processed by a computer using a
machine visionalgorithmtoaidinthecompletionof theimagecollection
task for crack identication. Single or multi-CCD (or infrared) cameras
can be used based on the hardware architecture and the size of the
workspace designed. Aseries of lightingsystems must alsobe considered
for nighttime operation of the automated pavement crack sealer.
4.2.3. Entire crack sealing system architecture
Since a multi-unit crack sealer (e.g., APCS), which consists of a tow
truck, sealant melter, and crack sealer, would require excessive
mobilization and demobilization time, a single self-contained vehicle
[11,12] would be a better alternative for the entire system architecture
(appearance) of an automated pavement crack sealer. The reduction in
time to connect/disconnect the three independent units and to hook/
unhook all the cables for mobilization and demobilization would be
directly translated into signicant potential improvements in daily
productivity of the crack sealer.
Fig. 3. Comparison of greedy and optimal path plan results in the APCS.
520 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
4.2.4. Manipulator and end-effector
An automated pavement crack sealer to be newly developed needs
a minimum of 4 degrees of freedomto effectively move and seal along
the spine of crack networks. X-axis and Y-axis movements are required
to move in any direction on a 2D surface. Z-axis movement
(telescoping function) is also required to guarantee that cracks in
rutted pavement are sealed properly. In addition, the turret needs to
rotate to effectively blow, seal, and squeegee along the spine of the
crack network. Therefore, a motion control systemis required to direct
the crack sealer along the X-, Y-, and Z-axes and to rotate the end-
effector of the automated pavement crack sealer. From previous
studies, ow ability of the melted sealant has been identied as an
important factor. To maintain the sealant in an appropriately viscous
state without any internal congestion in the hose, the length of the
Fig. 5. Software and hardware design requirements for enhancing the eld applicability.
Fig. 4. User-friendly graphical interface design of the APCS [7].
521 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
Fig. 6. Conceptual hardware design of automated proposed pavement crack sealer.
5
2
2
Y
.
S
.
K
i
m
e
t
a
l
.
/
A
u
t
o
m
a
t
i
o
n
i
n
C
o
n
s
t
r
u
c
t
i
o
n
1
8
(
2
0
0
9
)
5
1
3

5
2
4
hose between the melter and the end-effector (turret) should be
minimized. Also, incorporation of a heating function for the turret
during the entire sealing process is strongly recommended. In general,
it is well known that a multi-degree arm robot [11,12] is superior to a
Cartesian robot ((1)(4) in Table 1) in terms of workspace coverage
and mobility. While a Cartesian robot cannot cover more than its xy
frame size, a multi-degree armrobot can cover multi-lanes including a
shoulder, since the arm can be easily extended over its base vehicle
frame. Furthermore, compared to a Cartesian robot frame, the robot
arm can be easily folded for transportation, which can signicantly
reduce the overall crack sealing unit size, thereby providing better
mobility from one site to another job site. Fig. 5 briey describes the
essential software and hardware design requirements of the auto-
mated pavement crack sealer discussed in this section.
In this paper, authors suggest to use a 4 DOF arm as a manipulator.
The robot armshouldbe extended long enoughto cover a minimumof a
full lane, or the robot can be mounted on a motorized rack and pinion
systemwhich enables the robot to extend its arm greater than the base
vehicle width. In summary, Figs. 6 and 7 illustrate a conceptual design of
the hardware and its overall system architecture recommended for
practical use of the automated pavement crack sealer in the eld.
5. Conclusions and recommendations
Numerous efforts have been made to automate conventional crack
sealing activities. A complex evolution over the last two decades has
resulted in functional production prototypes (APCS) that achieves a
desirable balance between manual and automated functions for
pavement crack sealing. Many tools and algorithms have been
developed and experimentally evaluated to implement the prototypes.
This paper focused on illustrating the primary research ndings in
machine vision software and hardware designs required for enhancing
the applicability of an automated pavement crack sealer in the eld.
Inconclusion, it is anticipatedthat the unique man-machine interfaced
control loop including automated/semi-automated crack detection,
mapping, representation, line editing and optimal path planning
algorithms presented in this paper will be applicable to the model to be
newly developed after minor modications. Experimental results and
lessons learned from recent eld trials of the full scale crack sealers also
lead us to conclude that a single self-contained vehicle with a 4 to 6
degrees of freedom manipulator and a turret structure as an end-effector
would be the most appropriate system architecture for the newmodel of
the automated pavement crack sealer. Based on these research ndings, a
recommended conceptual hardware design and the entire system
architecture of the new model were presented in this paper. A detailed
design analysis based on the conceptual design proposed in this paper is
currently underway in an effort to develop a newmodel of the automated
pavement crack sealer. Finally, it is estimated that partial modication of
the algorithms and tools used in the automated pavement crack sealer
would eventually lead to broader applications in a wide variety of
automation of infrastructure condition assessment and maintenance.
Acknowledgements
This work was supported by the Korea Research Foundation Grant
funded by the Korean Government (MOEHRD). (KRF- 2006-214-
D00183).
References
[1] C. Haas, C. Hendrickson, Computer based model of pavement surfaces, Transporta-
tion Research Record 1206 (1990) 9198.
[2] C. Haas, C. Hendrickson, S. McNeil, D. Bullock, A eld prototype of a robotic
pavement crack sealing system, Proc. of the 9th ISARC, Tokyo, Japan, September
1992, pp. 313322.
[3] C. Haas, Evolution of an automated crack sealer: study in construction technology
development, Automation in Construction 4 (4) (1996) 293305.
[4] Y. Kim, J. Husbands, C. Haas, R. Greer, A. Reagan, Productivity model for performance
evaluation of the UT automated road maintenance machine, Proc. of the 14th ISARC,
Pittsburgh, PA., U.S.A., September 1997, pp. 443450.
[5] Y. Kim, C. Haas, A model for automation of infrastructure maintenance using
representational forms, Automation in Construction 10 (1) (2000) 5768.
[6] Y.S. Kim, C.T. Haas, A man-machine balanced rapid object model for automation of
pavement crack sealing and maintenance, Canadian Journal of Civil Engineering 29
(3) (2002) 459474.
Fig. 7. Overall system architecture of the proposed automated pavement crack sealer.
523 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524
[7] J.H. Lee, H.S. Yoo, Y.S. Kim, M.Y. Cho, J.B. Lee, The development of a machine vision
assisted, teleoperated pavement crack sealer, Proc. of the 21st ISARC, Jeju, Korea,
September 2004, pp. 149158.
[8] J.H. Lee, Y.S. Kim, J.B. Lee, M.H. Jeong, Development of an automated pavement crack
sealing machine and its economic feasibility analysis, Korea Institute of Construction
Engineering and Management(KICEM), Seoul, Korea 7 (6) (2006) 151164.
[9] Y. Kim, H. Yoo, J. Seo, Machine vision algorithm for automation of pavement crack
sealing, Submitted to Journal of Computer-Aided Civil and Infrastructure
Engineering on February (2007).
[10] Y. Kim, C. Haas, R. Greer, Path planning for a machine vision assisted, tele-operated
pavement crack sealer, ASCE Journal of Transportation Engineering 124 (2) (1998)
137143.
[11] S.A. Velinsky, Fabrication and testing of an automated crack sealing machine,
SHRP-H-659, National Research Council, Washington, D.C., 1993.
[12] S. Velinsky, B. Rabani, Longitudinal crack sealing and random crack sealing in
introduction to advanced highway maintenance and construction technology on
http://www.ahmct.ucdavis.edu (2008).
524 Y.S. Kim et al. / Automation in Construction 18 (2009) 513524