GPS - IMAGE PROCESSING IN MOBILE

ABSTRACT
Mobile Mapping Systems have been widely used to map transportation
infrastructure features for several years. First generation Mobile Mapping Systems,
however, could not provide positioning accuracy better than one meter, especially in
built-up urban areas. Thus, these data sets have been predominantly used to feed various
GIS systems, primarily concerned with infrastructure inventory and facility management.
With recent technological developments such as improving imaging sensors and,
more importantly, the introduction of ring-laser gyro inertial systems. The prototype-
positioning component of the system is based on a tightly integrated GPS/INS system,
and the imaging component comprises a single downlooking high-resolution, 1K by 1K,
digital camera.
The process of automatically identifying centerlines, extracting image features
and matching them is demonstrated on a variety of data sets, indicating clearly that the
algorithmic performance has reached a sufficient threshold such that human interaction is
no longer required, and consequently, the only limiting condition of the real-time
implementation is the available computer processing power.

1
ABSTRACT
INTRODUCTION
SYSTEM CONCEPT
PERFORMANCE OF THE AUTOMATED IMAGE SEQUENCE PROCESSING
HARDWARE IMPLEMENTATION
COLOR SPACE TRANSFORMATION
CENTERLINE EXTRACTION
FEATURE POINT EXTRACTION
STRIP FORMATION
POSITIONING PERFORMANCE
SUMMARY AND CONCLUSION

2
INTRODUCTION
Direct georeferencing of imaging sensors by means of integrated GPS/INS has
been in the spotlight in the surveying/mapping and remote sensing communities since the
mid-nineties (He et al., 1994; Bossler and Toth, 1995; El-Sheimy and Schwarz, 1999;
Schwarz, 1995).
One reason is that the primary driving force behind this process is a need to
accommodate the new spatial data sensors, such as LIDAR or SAR (airborne systems).
The second reason is that a substantial cost decrease, a possibility of data
reduction automation, and a short turn-around time are the most attractive features
offered by this technology
The main features of the system are the high image capture rate, the online use of
navigation estimates, and the on-the-fly image and stereo data processing. From a
navigation standpoint, the post processing of GPS/INS data provides more accurate
orientation as a benefit of forward and backward trajectory processing and precisely
synchronized timing information.
The high-accuracy GPS/INS/CCD system designed for monitoring linear highway
features is based on the concept of tight sensor integration, combining post-processing
with real-time image processing. The two primary components of the mobile mapping
system currently being implemented are precise navigation and digital imaging; both
allow for flexible and optimal system design, leading potentially to near-real time overall
data processing.

3
SYSTEM CONCEPT
Mobile Mapping Systems are built on the concept of combining high-performance
georeferencing with electronic imaging on a moving platform. MMS systems have been
using image sequences for a long time since it is an essential part of the concept.
However, progress toward the automation of the image sequence processing has been
slow for two reasons.
First, economy; the actual feature extraction represents approximately less than
20% of the overall cost, and therefore, the financial motivation is weak.
Second is the varying image scale, which makes the object recognition task quite
difficult in the feature-rich object space.
Traditional MMS systems work with forward- or side-looking cameras, while our
system uses a down-looking camera. This way the image scale changes very slightly and
there is an almost constant scale along the vehicle trajectory. The object contents of the
images are rather simple and predictable, such as the line marks, the primary interest to
us, surface texture variations, cracks, potholes, skid marks, etc.
Figure 1 shows the generic model of the dedicated centerline mapping system.

HARDWARE IMPLEMENTATION
The prototype of the integrated GPS/INS/CCD system designed for precision
monitoring of the highway edge- and centerlines comprises two dual-frequency Trimble
4000SSI GPS receivers and a medium-accuracy and high-reliability strapdown Litton

4
LN-100 inertial navigation system, based on Zero-lockTM Laser Gyro (ZLGTM) and A-
4 accelerometer triad (0.8 nmi/h CEP, gyro bias – 0.003°/h, accelerometer bias – 25 g).
Estimation of errors in position, velocity, and attitude, as well as errors in inertial
and GPS measurements, is accomplished by a 21-state centralized Kalman filter that
processes GPS L1/L2 phase observable in double-differenced mode together with the
INS strapdown navigation solution.
The estimated standard deviations are at the level of 2-3 cm for position
coordinates, and 5-7 arcsec and ~10 arcsec for attitude and heading components,
respectively. The imaging component is built around the Basler A201 camera, Kodak 1K
by 1K colour CCD with 9.07 mm by 9.16 mm imaging area (9-micron pixel size) and 15
images per second acquisition rate (15 Hz), which allows for 60% image overlap at
normal highway speed.

Figure 2 shows the system design, including the sensors, dataflow, processing steps.

PERFORMANCE OF THE AUTOMATED IMAGE SEQUENCE PROCESSING

5
 To assess the feasibility of automated line extraction with 3D positioning and
consequently its real-time realization, a rich set of the potential image processing
functions was developed in a standard C++ programming environment.
Figure 3 shows the overall dataflow and processing steps, which will be
illustrated in more detail later. In short, the real-time image processing is feasible due to a
simple sensor geometry and the limited complexity of the imagery collected.

COLOR SPACE TRANSFORMATION

Color images are preferred over monochromatic ones. Figure 4 illustrates various
cases, including an extreme situation where the yellow solid lines are hardly visible in the
B/W image. A simple histogram analysis in the RGB (Red, Green, Blue) color space
easily reveals two peaks in the red and green channels representing the yellow color of
the centerline.

6
 Although there are many image processing algorithms working with various color
data (multichannel gray-scale imagery), the great majority of the core functions work
only on simple monochrome image data. Therefore, if possible, a color space conversion
is desirable, in other words, moving from the 3D color space into one dimension, a color
direction, which shows the best possible separation for the objects we want to distinguish.

Since the quality of the extracted centerlines still show visible differences, a
filtering process has been implemented to remove this dissimilarity. The output of the
median filter is converted to a binary image. End results show no significant difference
between the centerline segments extracted from the very different images as illustrated in
Figure 6.

7
CENTERLINE EXTRACTION
After the RGB to S transformation and filtering, the geometry of the centerlines is
extracted from the binary images For a raster line, its centerline is of primary interest.
Skeleton is more like a one-pixel-width line, while centerline can be used to express a
one-pixel-width line in both raster and vector lines.
The skeleton is then extracted by shrinking the raster line from its boundary in all
directions until the one-pixel-width eight-connected line remains. In the medial axis
transformation method (Montanvert, 1986), the discrete medial axis pixels are the local
maximum of a transformation value.
A robust recursive filtering technique can eliminate noise such as gaps, although
most of the gaps and grey-scale irregularities already have been removed during the color
space transformation, as well as provide segmentation for multiple centerlines such as
double solid lines. Figure 7 depicts the results of this processing step.
Once boundary points are extracted, a line-following routine can generate the boundary
lines, which are subject of further cleaning such as removing irregularities by applying
geometrical constraints. In the final step, the midpoints are computed and the centerline is
extracted as shown in Figure 8.

8
FEATURE POINT EXTRACTION
To achieve the highest accuracy possible, the 3-dimensional centerline positions must be
obtained from stereo imagery. Knowing the camera orientation, both interior and exterior,
and the matching (identical) entities between the 2-dimensional centerlines, the 3-
dimensional centerline position can be easily computed.

I denotes the smoothing operation on the grey level image I(x, y). Ix and Iy indicate the x
and y directional derivatives respectively. Figure 9 depicts feature points extracted
around the centerline region from overlapping images.

9
MATCHING FEATURE POINTS
The matching of the feature points is accomplished through correlation. The
search space is constrained by the availability of epipolar geometry. For a given feature
point s, a correlation window of size n×m is centered at its location in the first image, at
point s1. Then a search window around the approximated location of the same object
point s in the second image is selected, point s2, and the correlation operation is
performed along the epipolar line. The search window size and location are determined
by navigation data. The correlation score is defined as

STRIP FORMATION
After determining the transformation parameters between consecutive images, the
centerline segments are connected and an approximate centerline can be incrementally
formed. However, the final coordinates of centerline can be computed only in post-
processing mode once the final navigation data have become available. An image strip
built by transforming five consecutive images into the same frame as shown in Figure 10.

10
POSITIONING PERFORMANCE
The multisensor system calibration is a key task to achieving the ultimate
accuracy of the given sensors. System calibration is defined here as the determination of
spatial and rotational offsets between the sensors as well as imaging sensor calibration.
Continuous calibration of the INS system is provided by GPS and thus is very dependent
on GPS anomalies such as satellite signal obstructions, multipath, interference, etc. The
effective ground pixel size was about 2-4 mm. Figure 11 shows the calibration range, and
the road area with control points.

A comprehensive analysis of the system calibration and positioning performance

are available in (Grejner-Brzezinska and Toth, 1999, and 2000). Using these boresight
parameters, the comparison of ground coordinates obtained by the photogrammetric
methods from the directly oriented imagery to the GPS-measured ground truth delivered
the ultimate accuracy performance of the overall system. The control points used in this
test were GPS-measured with an accuracy of ~1.5 cm per coordinate and were located
about 18 m from the perspective center of the camera.

SUMMARY AND CONCLUSION

This paper introduced a concept of an all-digital mapping system designed for
precise mapping of highway linear features. The test results presented here indicate that
an integrated, land-based system supported by a medium to high quality strapdown INS

11
and dual frequency differential GPS offers the capability for automatic and direct sensor
orientation of the imaging sensor with high accuracy.
In addition, the concept of real-time extraction of highwaylinear features such as
centerlines was demonstrated. The overall system performance was extensively tested by
a prototype positioning module (for more details see Grejner-Brzezinska and Toth, 1999,
and 2000), while the feasibility of automated feature extraction was evaluated only by
simulations.

12