1

A Stairway Detection Algorithm based on

Vision for UGV Stair Climbing
Yang Cong Xiaomao Li Ji Liu and Yandong Tang 1

'Robotic Laboratory, Shenyang Institute ofAutomation, Chinese Academy of Science 110016, China

2Graduate School of the Chinese Academy of Science, Beijing 10038, China

images and reconstructing the 3D environment through line

Abstract-In the paper, we present a vision based algorithm matching. In order to make the UGV climb the stairways
used to guide the Unmanned Ground Vehicles (UGV) for automatically, two parameters are very important and play key
autonomous stairways climbing and implement it on UGV roles: the angle between the vehicle's heading direction and
successfully. The reliability of guiding UGV to climb stairs the stair's horizontal direction, and the position of the vehicle
requires evaluating two offset parameters: the position of vehicle
on stairs and the orientation angle to stairs. The intention of our on the stairs. The angle is used to guide the vehicle to the right
algorithm is to estimate these two parameters through extracting direction, and the position to avoid the vehicle colliding to the
the stair edges robustly. To achieve this goal, we apply the Gabor walls. For evaluating these two offset parameters accurately a
filter to eliminate the influence of the illumination and keep edges, stairway detection algorithm based on vision for UGV stair
and propose a fast method to remove small lines. Finally we link climbing is presented in this paper.
stair edges, and estimate the offset parameters used to steer the
vehicle by RANSAC algorithm. For validating our algorithm, a visual servo system is
Experiments on various stairways including indoor and introduced to control the UGV stair climbing by these two
outdoor are given in various light conditions. The results validate parameters and to guide the UGV to go forward along the
our algorithm. lines bisecting the stairs ( to see Fig. 1). A vision algorithm for
the visual servo system is proposed to extract a set of long,
Index Terms- computer vision, UGV, stair climbing, Gabor horizontal lines robustly in any conditions and to estimate the
filter, RANSAC parameters by the set of lines. In practice, the UGV will be
confronted with many different conditions, such as the stairs
of different styles and materials, different application
I. INTRODUCTION
environments including indoors and outdoors, illumination
changes, shadows, etc. In order to deal with these situations
A s the mobile robot technology making remarkable
progress, it has become the preferred solution for many tasks.
In order to navigate autonomously in unknown environments,
mobile robots, such as UGV, must be equipped with the
followings: sensors for measuring the terrain, software for
interpreting the data, path planning to decide a safe route of
action, and control to oversee the plan.
The ability to detect and climb the stairways automatically
is of a great challenge to UGV. For detecting the stairways
based on vision automatically, Stephen Se[1] used monocular
to detection curbs and stairways for the partially sighted
human; in [2] JS Gutmann introduced stereo vision to
guided the human robot to climb up and down the stairways;
xiaoyu[3] and R.Turchetto[4] proposed methods to detect and
locate curbs and stairways using stereo vision by segmentation
Manuscript received October 1, 2007. This work was supported in part by
Shenyang institute of Automation (SIA) of Chinese Academic of Science
(CAS). i 1;.i 11in uveraii Ulisign U1 11n U V, L11nC LWU L1raCK aSIuw Ln1 lUUUL 1 USCU
All authors are with Robotics Laboratory, Shenyang Institute of for driving and turning and the other two tracks in front of the robot is used
Automation, Chinese Academy of Sciences Shenyang 110016, China for climbing.
(congyang8 1 0gmail.com)

1806

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on March 20,2010 at 06:18:51 EDT from IEEE Xplore. Restrictions apply.

robustly, Gabor filter is applied to eliminate the influence of
the illuminate. Furthermore, a fast method is proposed to filter
out small and vertical lines after the edges are detected. This
step improves the efficiency and the accuracy of the algorithm
in the stair edge detecting and linking processing. Then, a
robust edge linking algorithm is used to extract the set of
parallel lines. Finally, we estimate the two offset parameters
based on the theory of RANSAC algorithm [10][11][12]. The
experimental results validate our algorithm.
The paper is organized as followed: in section 2, we
describe the whole flow of our algorithm. In section 3, we
estimate the two parameters by the RANSAC and describe the
strategy of steering the UGV. In section 4, the results of our
experiments are shown. We end this paper with conclusions
and discussions in section 5.
II. THE WHOLE FLOW OF OUR ALGORITHM
Stairways are of different styles, different materials, and
under different lighting conditions. The only feature of
stairways is that their profile includes a set of parallel lines in
3D space. When the UGV is creeping on the stairways, the
projection of stair edges on the 2D image plane are nearly a
group of parallel lines.
The intention of our vision algorithm is to detect long,
horizontal lines in an image, and to extract the most similar
ones among these lines which should be the stair edges. These
extracted lines can be used to estimate the current offset angle
.L
N
y
v
Fig.2 The whole flow of our algorithms
1807
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on March 20,2010 at 06:18:51 EDT from IEEE Xplore. Restrictions apply.
2
and the current offset position of the UGV on the stairways,
which make it possible to control the UGV to creep the stairs
automatically. Fig.2 illustrates the whole flow of our
algorithm.
Our algorithm includes the following steps:
Firstly, the Gabor function is used to filter the image in
order to eliminate the influence of the illuminate and keep the
stair edges.
Secondly, the generic edge detector is applied to the filtered
image. With our proposed fast algorithm the most of the small,
vertical edges are removed. It can improve the efficiency and
accuracy of the linking algorithm in the next step.
Thirdly, the remainder adjacent edges are linked into a long,
horizontal edge (which should be the stair edges) according to
some basic constraints.
Finally, the two offset parameters are estimated by the
RANSAC method. And the UGV can be steered by the two
parameters to climb the stairs. When the vehicle arrives at the
linking between two staircases, it restarts the Stair Detection
Program to detect the stairs automatically. If the vehicle
detects the stair, it will go on climb the stair based on vision.
A. The Gaborfilter
The Gabor filter can eliminate the influence of the
illumination to an image efficiently and reserve the multi-scale
and multi-direction information. The Gabor filter has also
been shown to be benefit to the receptive fields of simple cells
in the striate cortex and hence their relationship to models for
the early vision of manuals [7,8,9]. The Gabor filter is applied
to eliminate the influence of variety illumination and shadows
effects on an image. The appropriate selection of the scale and
the frequency of the Gabor filter to maintain the level of the
staircase edges will be a precondition for the following
processing.
Equation 1 shows the Gabor function:
1 I1X2 2
ga ,, (x, Y) exp[- (- + y 2 )] exp[jo (x + y)] (1)
where (x, y) is the point position in the horizontal and vertical

direction, ax, a is the standard deviation of the x direction
and y direction respectively, and o0 is the space frequency of
the complex plane-wave.
By selecting the appropriate parameters of ax, (y, and o
to maintain the level of the stair edges, it will be conducive to
further processing. For our current work, we use the
multi-channel filtering approach with a set of predetermined
filter parameters. In our experiments, we assume that all the
stair edges are horizontal or nearly horizontal (the slope is less
than 0.5). So in order to get the appropriate parameters of
Gabor filters, we experimentally apply a series of Gabor filters
to stair images in order to choose the filter which has the
largest response in vertical direction and the lowest response
in horizontal direction as the most suitable one. Through
experiments, the parameters are determined, which direction is
vertical and the band width equal to 1. And these parameters
are used in all of our experiments. Fig.3 shows the original
image, and Fig.4 shows the result of the image convolved by
the Gabor filter.
Fig. 4 The result of Gabor filter
B. Eliminate small Lines
The Gabor filtered image is segmented by the generic edge
detector, and there are still a lot of small lines in the image.
Fig.5 shows their results respectively. It is inefficient if we
apply Hough transform, RANSAC or other edge link
algorithms directly on the output binary edge images for
detecting the stair lines. In order to remove most of small,
vertical lines which are not wanted, we propose a new fast
(a) The edge of no Gabor filtered (b) The edge of Gabor filtered
Fig.5 The results of the edge detector
1808
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on March 20,2010 at 06:18:51 EDT from IEEE Xplore. Restrictions apply.
3
The main idea of our template operation is that if a line gets
through or partial get through the template boundaries, it may
be a line preserved. If a line does not get through, it would be
eliminated. We define the radius of the template as r, the width
of the template as W=2*r+1 and the image point as img (i, j),
the boundary point of the template as Edge{i, j}
where i = 1 or i = W or j = 1 or j = W . The set of
neighbor points of the Edge{ i, j} are denoted as
adj{fi, j} . The method to compute the sum of the neighbor
points of the edge points in the template is as follows:
sum = E adj(i, j)xEdge(k,l)
edge point adj i_j
If sum > 1,
There would be a line pass through the template;
Else if sum < 1,
There would be no line, and so we elim inate all
of the edge pixels in the template.
This method can remove all of the isolate points and most of
the short discontinuous lines, and save much time in edge
linking. The result of the method on synthesize image is
illustrated in Fig.6. And Fig.7 shows the results of this method
on the stair binary image.
'The synthesize image 'The result of the eliminate small lines
Fig.6 The result of the eliminate small lines on synthesize image
Fig.7 The result of eliminating small lines on real image
 4

C. Edge Linking coordinates and y coordinates.

After eliminating the small lines, the binary image, some A series of rules are designed to link the edges, and the
basic constraints are set to find the stair edges in the image: details of the linking processing are listed below:
1) Stairway edges are always long and straight lines; 1. We sort the line segments by their area ai in descend
2) Stairways edges are parallel to each other;
3) The direction ofthe stairway is nearly horizontal: so we order, and denote the set of lines segments as L:
assume the slope the line less than a threshold. L.{1.Ill.2..N. .
where N is the total number of the lines.
Let us define some parameters about the ith line segment or 2. The merging process is:
region in the binary edge image: Fo-r i=l :X
ai: the area of the region i; For j=1A4, where j-i
x;: the x coordinate of its left endpoint; If I k' - kj II> thresholdl
x: the x coordinate of its right endpoint; Continue;
Else If xl _x > thresholdx
yI: the y coordinate of its left endpoint;
yx: the y coordinate of its right endpoint; or Xl IJ.
-X > thresholdx
Continue;
k: the slope of its fitted line;
b': the intercept of its fitted line.
Else If y' -Yr
I Jr
> thresholdy
or y -yY > thresholdy
The ith region is fitted as a line. The equation of the line Continue;
segment is y = k'x + b' . Then the parameters k' and b' Else
Merge the line ith and line jth into line ijth.
are computed by equations (2) and (3):
Recompute the kj, bWj and the other
kiarea iS' -_'S'ZS parameters about line ith.
(2) If k1' does not close to zero
area SXxx - Sx'
x x Reject the line ijth,
k =

S>S1y xI- s
area I l s I I Continue.
=area'S> - Si Si x
(3) Else
Insert line ijth into L,
S' yiscythe sum of all its x coordinates, S' is the N=N+ 1.
sum ofWhere
all its
sum of all its y coordinates, S'. is the sum of the square of For i=l: total number of the merged lines
If the length of the line i is less than width/4,
all its x coordinates, S'y is the sum of the square of all its y Delete.
coordinates, and Syis the sum of the product of all its x Else If x;> width/2 or xl <width/2
Delete.

The remaining linked lines are most likely the staircase

edges. And in the experiments, threshold] equals to 0.5,
thresholdx and thresholdy are both 5.
Fig.8 shows the set of extracted parallel lines from the 2D
image.
III. ESTIMATE THE PARAMETERS AND STEER THE
VEHICLE
A. Estimate the parameters
Not all the detected parallel line segments from the image
belong to the stairways, and some line segments which belong
to the stairways may be only a part of the stair edges. This is
caused by the influence of different illumination. The offset
parameters, the offset angle AO and offset position Ax, can
Fig.8 The result of extracting parallel stair edges

1809

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on March 20,2010 at 06:18:51 EDT from IEEE Xplore. Restrictions apply.

be calculated from these parallel lines using RANSAC
algorithm.
The RANSAC algorithm is an algorithm for robust fitting
of models. It was first introduced by Fischler and Bolles in
1981 [10]. It is robust in the sense of good tolerance to outliers
in the experimental data and is capable of interpreting and
smoothing data containing a significant percentage of gross
errors. The RANSAC algorithm has been applied to a wide
range of model parameters estimation problems in computer
vision, such as feature matching, registration or detection of
geometric primitives[ I1] [ 12]. In this paper, we use RANSAC
to estimate the offset parameters, and the detail of the
RANSAC algorithm is described below:
Set of data points:
* Pj =xj, yj} as the center point of each line
wherej=tl ...N},N as the total number of the
detectedparallel lines.
* 64=kj, bj} as the line parameters which is
fitted by the center points of Pj, where
yj =kj*xj+bj.
* h(®j) as the accumulate of the distance of
point i from line 64
* S as the largest number of circulation
Forj 1, ,S
Select a minimal subset {pj,l,;*,pj,M} from
the {pl,;-,pN} points. M equals to 2.
Generate an estimate ®j from the minimal
subset, using the equation (2) and (3) in the
last page. We fit the center points as a line.
Evaluate h(®j) on the N-M points not in the
minimal subset.
Retain Oj as the best if h(Oj) is the
minimum thus far. Denote the best as ®j*.
* Gather the inliers to the best estimate, Oj8, and refine
using (weighted) least-squares.
Fig.9 shows the process of estimating the offset parameters.
The red lines is the detected parallel lines, and the white points
means the center point of each line. The blue line shows the
heading direction of the vehicle, the orientation of the UGV is
in the vertical direction, and the position of the UGV is in
width/2 of the image. The white line computed by the
RANSAC algorithm shows the estimated center line of the
stairways.
The direction of the stairway is 0 = arctan(k*) and the
average of x coordinates is x xi / N So we can
calculate the offset angle AO and offset position Ax as
equation (4),
AO =T /2-O
Ax= w idth / 2 - x (4)
These two estimated parameters will be used to steer the
vehicle.
1810
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on March 20,2010 at 06:18:51 EDT from IEEE Xplore. Restrictions apply.
5
Fig.9 Estimate the offset parameter
B. Steer the vehicle and turn the corner
Our algorithm is used to guide the vehicle to climb the stair
automatically. The ideal objection of our algorithm is to make
the vehicle stay in the center of the stairways and steer it go
ahead along the stairways. In practice it is no use to make the
UGV always stay in the precision center of the stair. In fact, if
the vehicle is not too close to the sides of the stairways and the
heading orientation angle of the vehicle is almost along the
stairways, the UGV will climb the stairways safely. Through
experiments we find out that the offset angle AO and offset
position Ax are change slightly in the two continuous frames.
And this phonemena is mainly caused by changing of
viewpoints and the influence of illumination. So a set of
thresholds are preset. If both of AO and Ax are less than the
preset thresholds, the UGV will run along the orientation of
the last operation; but if either AO or Ax is great than the
threshold, we will change the direction of the vehicle to make
it along the right orientation. Through experiment we preset
the thresholds of AO and Ax as 0.25zT and 20 pixels.
When the UGV get to the landing plane, the stair detection
program will be restarted to recognize the stairs, planed the
path, and navigated the vehicle to the next stair
automatically. So the UGV can turn the corner and go upstairs
or downstairs autonomously.
IV. EXPERIMENTS
We test our system and algorithm under various lighting
conditions on various style and materials of stairs which are
indoors and outdoors.
Fig. 10 shows the results under various conditions, the
detected stair edges are labeled by red lines. In the outdoor
condition, all the stairways are recognized regardless of light
conditions. In the indoor condition, when facing to the sun
only a portion of stair edges are not labeled that is because the
image is saturated by the sunshine.
Fig. 11 illustrates a steering processing when the UGV is
mounted on the side of the stair or inclined on the stair, our
algorithm can still detect the stair edges (red lines), guides the
vehicle to the center of the stairways, and steer the vehicle up

to the stairway. The white lines indicate the estimated center
of the stairwells. Fig. 11(a) shows the vehicle is mounted on
the side of the stair. The offset parameters marked by the
white line are estimated successfully. Figi 1(b) shows that our
vision system is guiding the vehicle to the center of the
stairways. Fig.11(c) shows the vehicle inclines on the
stairwells and our algorithm can still detect the stair edges and
estimate the parameters robustly. At last, Fig. 11(d) shows the
vehicle is adjusted in the center of the stair by our algorithm.
The indoor condition fqce to the, -un
The outdoor condition
Fig. 10 The results under various conditions
Fig. 11 A steering processing when the UGV are mounted on the side of the
stair
1811
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on March 20,2010 at 06:18:51 EDT from IEEE Xplore. Restrictions apply.
6
Finally, implemented on a laptop computer (Pentuium4
2.4GHZ) using C language, the over time of the systems is
less than 60ms, including extracting the edges, estimating the
time, and steering the UGV.
V. CONCLUSION
We have demonstrated a vision system based on our own
algorithm which can guide the UGV to climb the stairs
automatically under various conditions. The results proved the
validity of our algorithm.
REFERENCES
[1] S. Se and M. Brady, "Vision-based Detection of Kerbs and Steps",
Proc. BMVC97, 1997
[2] JS Gutmann, M Fukuchi, M Fujita "Stair climbing for humanoid
robots using stereo vision" Intelligent Robots and Systems,
2004.(IROS 2004)
[3] Xiaoye Lu, Manduchi, R. " Detection and Localization of Curbs and
Stairways Using Stereo Vision" Proc.ICRA 2005 4648- 4654
[4] R. Turchetto and R. Manduchi, "Visual Curb Localization for
Autonomous Navigation", Proc. IEEE/RSJ IROS'03, Las Vegas,
October2003.
[5] Y.Xiong and L.Matthies, "Vision-Guided Autonomous Stair
Climbing",Proc. ICRA, 2000.
[6] Anastasios I. Mourikis, Nikolas Trawny, Stergios I. Roumeliotis,etc
"Autonomous Stair Climbing for Tracked Vehicles" The International
Journal of Robotics Research, Vol. 26, No. 7, 737-758 (2007)
[7] G.Daugman. "Uncertainty relation for resolution in space,
spatial-frequency, and orientation optimized by two-dimensional visual
cortical filters". J.OPT.Soc.Am., 2(7):1160-1169,1985
[8] Tai Sing Lee. "Image Representation Using 2D Gabor Wavelet" IEEE
Trans. on Pattern Analysis and Machine Intelligence, 1996,
18(10):959-971
[9] J. K. Kamarainen, V. Kyrki and H. Kalviainen. "Invariance Properties of
Gabor Filter-Based Features" Overview and Applications. IEEE Trams.
On Image Processing, 2006, 15(5):1088 1099
[10] M.A.Fischler and R.C.Bolls. Random sample consensus: A paradigm for
model fitting with application to image analysis and automated
cartography. Communication of the ACM, 24(6):381-395,1981
[11] D.R.Myatt, P.H.S.Torr, S.J.Nasuto, J.M.Bishop, and
R.Craddock.Napsac :High noise, high dimensional robust estimation-
it's in the bag. British Machine Visioin Conference, Cardiff, UK, pages
458-467,2002.
[12] 0. Chum and J. Matas. Randomized ransac with t(d,d) test. British
Machine Vision Conference, Cardi*), UK, pages 448 {457, 2002.