See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/284285195

Remote Sensing Image Mosaic by Incorporating

Segmentation and the Shortest Path
CONFERENCE PAPER in COMMUNICATIONS IN COMPUTER AND INFORMATION SCIENCE JANUARY 2013
DOI: 10.1007/978-3-642-45025-9_67

READS

4 AUTHORS, INCLUDING:
Yindi Zhao
China University of Mining Technology
21 PUBLICATIONS 127 CITATIONS
SEE PROFILE

Available from: Yindi Zhao

Retrieved on: 11 March 2016

Remote Sensing Image Mosaic by Incorporating

Segmentation and the Shortest Path
Yindi Zhao, Tianqing Han, Shuna Feng, and Congcong Miao
School of Environment Science and Spatial Informatics,
China University of Mining and Technology, Xuzhou, Jiangsu 221116, China
{zhaoyd,hantq,fsn2012,miaocc}@cumt.edu.cn

Abstract. Combining segmentation and shortest path searching, a novel

seamline extraction method for remote sensing imagery is presented. It detects
an optimal seamline along the boundaries of land cover objects so as to reduce
the effect of the discontinuity on the seamline of the mosaic image. First, the
valid overlaps of the reference image and the registered image was determined,
as well as the starting point and the ending point. Second, watershed
segmentation was applied to sketch out the boundaries of salient objects in the
overlaps of the reference image. Third, after a difference image between the
reference image and the registered image was obtained, those segmentation
paths which pass through segments with an enormous difference were removed.
Finally, an undirected weighted graph was built according to the remaining
segmentation paths, and the seamline defined by the shortest path from the
starting point to the ending point was found via the Dijkstra algorithm.
Experimental results on Landsat-7 ETM+ images suggested that the presented
method was capable of generating a seamless mosaic image.
Keywords: image mosaic, seamline extraction, watershed segmentation,
shortest path, Dijkstra algorithm.

Introduction

In the field of remote sensing, image mosaic is often used to make a larger image
covering the entire region of interest [1]. As a result, there has been significant
attention paid to develop automatic techniques for generating seamless remote
sensing mosaic images. Generally, image mosaic consists of two main processes. The
first process is to register the neighboring images into a unified geographic coordinate
system, and the following one is to stitch the registered images into an entire image.
Simply superimposing registered images tends to cause visible seams [2]. Thus many
approaches have been proposed in order to generate a seamless mosaic image, which
can be categorized into two groups: boundary transition smoothing and optimal
seamline searching.
Boundary transition smoothing is a rather straightforward way to reduce visible
differences between neighboring images. It is commonly referred to as feathering or
blending. Multiresolution blending is an effective way to leverage bi-image
F. Bian et al. (Eds.): GRMSE 2013, Part I, CCIS 398, pp. 684691, 2013.
Springer-Verlag Berlin Heidelberg 2013

Remote Sensing Image Mosaic by Incorporating Segmentation and the Shortest Path

685

differences of geometric and radiometric properties, which has been successfully used
in 2-D panoramas [3] and 3-D models [4]. However, losing clarity of ground object
textures is inevitable due to the blurring and ghosting in the overlap. Therefore, the
seamline-based mosaic method is an alternative solution. Seamline optimization
found in literatures can be mainly classified into two categories: pixel-based methods
[5-8] and object-based methods [9-12].
Pixel-based seamline optimization methods do a pixel-by-pixel search by means of
path planning techniques. Uyttendaele et al. [5] observed that, for well-registered
images, changes in image objects produce the most noticeable artifacts. To overcome
this problem, the moving objects were recognized by smoothing the difference image
with morphological erode and dilate, and ghost effects caused by moving objects were
removed by the weighted vertex cover algorithm whilst positioning the seamline at
locations with consistent states. Davis [6] described an image blending method that
computed the relative photometric difference of bi-images and searched for the
dividing boundary along the low intensity of the difference image using the Dijkstra
algorithm, but the computation cost is considerably high for remote sensing image
mosaic involving megapixels. Pan and Wang [7] combined image gradients and
image difference to avoid seamline crossing edge features. Yang et al. [8] presented a
bisector seamline algorithm, in which overlaps between image pairs was
approximated by a quadrangle, and a bisecting polyline was detected as the seamline.
These pixel-based methods placed seamline without taking the image contents into
consideration. As a result, some salient objects might not be circumvented by the
seamline, which leads to significant artifacts.
To solve this problem, object-based methods were presented. Zuo et al. [9]
distinguished ground area and non-ground area according to the high precision digital
surface model (DSM), and used the greedy snake algorithm to detect the optimal
seamline circumventing buildings. But the effectiveness of this method is limited to
images covering building groups. Wan et al. [10] presented a vector-based seamline
determination approach, in which a weighted graph was built by overlapping the
existing vector roads with the extracted skeleton, and the lowest cost path referring to
the optimal seamline was found by the FloydWarshall algorithm. Soille [11]
proposed a image compositing procedure based on mathematical morphology and its
marker-controlled segmentation paradigm to diminish the visibility of seams in the
output mosaic. Gracias et al. [12] divided the overlaps into sets of disjoint segments
by watershed segmentation and then searched the seamline using graph cut. Refs. [11,
12] suggest that object boundaries as natural transitions are desirable for image
mosaic and thus the seamline along these boundaries is less perceptible in the mosaic
image.
In this paper, an automatic image mosaic method by the combination of watershed
segmentation and the Dijkstra shortest path search algorithm was developed to make
seams invisible while still preserving image texture and details. The paper is
organized as follows. Section 2 detailly describes the proposed mosaic scheme by
incorporating segmentation and shortest path. The experimental results are given in
Section 3. The paper concludes in Section 4.

686

Y. Zhao et al.

Method

Fig. 1 is the illustration of the proposed image mosaic method, in which image
segmentation and the shortest path search are incorporated to extract an optimal
seamline. Seamline extraction consists of four steps. Firstly, the valid overlaps
between the reference image and the registered one is identified to narrow the
subsequent search after registering the neighboring images using the method
presented in [13]. The rough seamline as well as its starting point P and ending point
P are then obtained. Secondly, the overlap region of the reference image is
segmented by the watershed segmentation algorithm, and then the boundaries of land
cover objects are determined. Thirdly, the reversed difference image within the
overlaps and the corresponding binary difference image are calculated. The
segmentation result is overlaid with the binary image to build an undirected weighted
graph, and the shortest path from P to P is found by the Dijkstra algorithm. Finally,
taking the shortest path as the optimal seamline, seamline-based image mosaic is
done. In this paper, we focused on seamline search, without considering the issue of
color balance.

Image registration
Determine the valid overlap region
Segmentation of valid overlap region

Reversed difference image I

Binary difference image I

Mask the segmentation with I

Generate the undericted weighted graph
Determine the seamline using the Dijkstra algorithm
Image mosaic based on the seamline
Fig. 1. Flow chart of the proposed method

2.1

Determination of the Valid Overlaps

Valid regions of registered images are quadrangles rather than rectangles and their
invalid regions are filled with black pixels. That the seamlines may fall in invalid
regions would lead to holes in the output mosaic, unless valid overlaps are
determined.

Remote Sensing Image Mosaic by Incorporating Segmentation and the Shortest Path

687

In order to determine the valid overlaps, two simulation images need to be

generated in advance. As shown in Fig. 2, A is the same size simulation of the
reference image, and B is the same size simulation of the warp image. All of the pixel
values in A were initialized to 2. The geometric transformation of B was done by the
same way with that of the warp image. The valid region in B was initialized to 1, and
the remaining invalid regions were initialized to 0. Fig. 2(c) is the sum of A and B, in
which the brighter region with the value 3 corresponds to the valid overlaps. The
brighter region is commonly a quadrangle with four vertices. The two vertices that do
not match the corners of the input images are defined as the two ends of the seamline,
thus the rough seamline can be determined. Note that the initial pixel values of A and
B are used to distinguish the valid overlaps which is defined as the region with the
maximum pixel value in the sum of A and B, and thus can be set to arbitrary values
except 0.
Rough seamline
Valid overlap region
A

(a)

Ps

A
B
Pe

B
(c)
Invalid region
(b)
Fig. 2. An illumination of the valid overlaps and rough seamline. (a) is the simulation of
reference image. (b) is the simulation of registered warp image. (c) is the determined valid
overlaps (denoted by the white box) and the rough seamline (denoted by the solid black line
within the white box).

2.2

Watershed Segmentation of the Overlaps

In this part, watershed segmentation is applied to sketch out the boundaries of land
cover objects in the valid overlaps of the reference image. These object boundaries
will be treated as candidate positions to locate the seamline in the following shortest
path search.
Watershed transformation is a popular segmentation approach inspired by
topographic relief flooded by water. Existing literatures provide a large number of
application examples of watershed segmentation. See Ref. [15] for a review of several
existing definitions of the watershed transform and associated algorithms.

688

Y. Zhao et al.

In order to avoid over-segmentation caused by noises and surface extrema, the

reference image is smoothed using morphological techniques called opening-byreconstruction (erosion followed by morphological reconstruction) and closing-byreconstruction (dilation followed by morphological reconstruction) before watershed
segmentation. For the images used in this paper, good smoothing result was achieved
using a disk structuring element with a size of 5.
Moreover, segmentation results of reference image are generally not completely
consistent with object boundaries in registered image due to the presence of changing
objects. As a result, the seamline may pass through those objects. This problem can be
improved by determining regions of difference (RODs) between images. First, the
difference image of overlaps was calculated and then smoothed using a Gaussian low
pass filter. Then, gradation reversal and binarization were applied to the smoothed
difference image to determine the RODs. After that, the segmentation paths passing
through the RODs were removed to avoid seamline being located in these regions.
2.3

Extraction of Optimal Seamline Using Dijkstra Algorithm

After segmentation, the seamline extraction problem can be formulated as a shortest

path searching problem. Here we used the Dijkstra algorithm [14] to solve this
problem. An undirected weighted graph
,
needs to be generated before
shortest path search.
Suppose each pixel of the segmentation paths as a vertex, and edges exist only
between adjacent pixels. Let
, , ,,
denote the set of vertices and
, , ,,
, , ,,
edges in the graph and
weights
corresponding to the edges. The weight of the edge linking the two adjacent vertices
and
is caluclated by
2

1,

1,

(1)

is the perpendicular distance from the vertex

to the rough seamline, and
where
is the perpendicular distance from the vertex
to the rough seamline. That is to
say, the weight of each edge is the average distance from the two vertices it joints to
the rough seamline. Note that
,
is normally stored as an adjacency matrix in
which the number of vertices and edges is often very large due to the large number of
pixels in the overlaps. Fortunately, edges exist only between adjacent pixels on the
segmentation paths, a vast majority elements in the adjacency matrix are zero.
Therefore, it is very beneficial for reducing memory consumption to store the
adjacency matrix in a sparse form.
After building the undirected weighted graph, to find the optimal path from the
starting point P to the ending point P , the Dijkstra algorithm was applied to
minimize the cost function defined by formula (2):

(2)

where
is the amount of edges in the path, and
is the weight of the -th edge.
The path corresponding to the minimum
is defined as the optimal seamline.

Remote Sensing Image Mosaaic by Incorporating Segmentation and the Shortest Path

689

Experimental Results
R

To evaluate the performan

nce of the proposed method, a pair of Landsat-7 ET
TM+
scenes with the same row and
a the neighboring path were used. The data was proviided
by the Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciennces
(http://ids.ceode.ac.cn/index
x.aspx).The image of path/row 121/036 captured on
September 29, 2002 was treated as the reference, and the image of path/row 122/0036
on October 6, 2002 the warp.
w
For this series of experimental data, overlappping
acquisitions covering Xuzh
houCity, Jiangsu Province, China occurred across the saame
row. Two sub-images with the size of 600 600 were cropped from the overlappping
regions of the pair of sceness respectively [see Fig. 3(a) and (b)].

(a)

(b)

(c)
1

0.9

0.8

0.7

0.6

(f)

0.5

0.4

0.3

0.2

0.1

(d)

(e)

(h)

(g)

(i)

Fig. 3. Seamline extraction of the experimental images. (a) and (b) are the used images, withh (a)
being the reference image. (c)) is the seamline extraction result, in which the valid overlapps is
indicated with the yellow box
x and the seamline is indicated with the red line. (d) is the RO
ODs
(black regions) of the overlap
ps. (e) is the watershed segmentation result superimposed onn the
overlaps. (h) is the result of masking
m
(e) with (d). (f) and (g) are enlarged views of the w
white
dotted boxes in (e) and (h), (i) illumination of weight of edges in the overlaps.

The watershed segmenttation was done after smoothing the reference imagee as
presented as Section 2.2 usiing a disk structuring element with a size of 5. A threshhold
of 0.7 was used to determin
ne the RODs after smoothing the difference image usinng a
Gaussian low pass filter with a standard deviation of 1.4 pixels and a 9 9 Gausssian
kernel. A computer with an
a AMD Turion (tm) Dual-Core Mobile M520 2.3G
GHz

690

Y. Zhao et al.

processor, 2GB internal memory was used for data processing, and the all the
experiment was simulated with MATLAB.
Fig. 3 shows the mosaic results of the experiment images. As described in Section
2.1, the warp image in Fig. 3(b) was registered with the reference in Fig. 3(a)
beforehand. Fig. 3(e) is the valid overlaps overlaid by object boundaries sketched by
watershed segmentation. However, the problem of over-segmentation was not
completely avoided. Fig. 3(f) is the enlarged view of the white dotted box in Fig. 3(e),
in which several segmentation paths passing through the Yunlong Mountain are
apparent. The located seamline would not circumvent the mountain if these paths
were all involved in the process of seamline optimization. As presented in Section
2.2, the RODs of overlaps shown as black regions in Fig. 3(d) were used to solve this
problem. Comparing Fig. 3(d) and the overlaps represented by the yellow box in Fig.
3(c), we can find that most of the black regions in Fig. 3(d) are mountains and waters.
This phenomenon is caused by the great relief displacement of mountains in the biimages and the illumination sensitivity of waters. After gradation reversal and
binarization, the mountains and waters were marked as RODs. As shown in Fig.3(f)
and Fig.3(g), the segmentation paths passing through the mountains were cuted off so
as to make the seamline bypass the mountains. Fig. 3(i) is the weight map of edges.
As discussed in Section 2.1, in order to ensure the extracted seamline not be too far
from the rough seamline, the edges closer to the rough seamline were given smaller
weights. As a result, those edges would be more likely to be a part of the final
seamline. Fig. 3(c) is the result of seamline optimization. The optimal seamline was
positioned along salient structures, successfully bypassing the mountains and nearing
to the rough seamline.

Conclusions

This paper presented an automatic image mosaic method via the combination of
watershed segmentation and the shortest path search. Different from pixel-based
seamline optimization algorithms, the proposed method positions seamlines along
boundaries of image objects. RODs are pre-defined after the reversed difference
image is binarized. The land cover object boundaries are sketched out using the
watershed segmentation algorithm. And then they are refined by removing the
segmentation paths passing through the RODs. In the next step, the shortest path as
the optimal seamline are extracted by the Dijkstra algorithm when the refined object
boundaries are candidate paths. Finally, mosaic is performed based on the extracted
seamline. The experimental results on Landsat-7 satellite images suggested that the
proposed method was capable of determining an optimal seamline following salient
structures and achieving a desirable composited image.
Nevertheless, the study of this paper emphatically discussed the issue of seamline
optimization without considering exposure compensation. Color balance should be
involved in our further work.

Remote Sensing Image Mosaic by Incorporating Segmentation and the Shortest Path

691

Acknowledgments. The authors would like to acknowledge financial support from

the Fundamental Research Funds for the Central Universities (2013QNB11). We
would also like to thank the Institute of Remote Sensing and Digital Earth, Chinese
Academy of Sciences for providing Landset-7 ETM+ images.

References
1. Goshtasby, A.A.: 2-D and 3-D Image Registration: for Medical, Remote Sensing, and
Industrial Applications. John Wiley & Sons Inc., United States (2005)
2. Szeliski, R.: Computer Vision: Algorithms and Applications. Springer (2010)
3. Brown, M., Lowe, D.G.: Recognising Panoramas. In: ICCV 2003: 9th International
Conference on Computer Vision, pp. 12181225. IEEE Comput. Soc., Los Alamitos
(2003)
4. Baumberg, A.: Blending Images for Texturing 3D Models. In: BMVC 2002: British
Machine Vision Conference 2002, pp. 613622. British Machine Vision Assoc.,
Manchester (2002)
5. Uyttendaele, M., Eden, A., Skeliski, R.: Eliminating Ghosting and Exposure Artifacts in
Image Mosaics. In: Proceedings of the 2001 IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, CVPR 2001, pp. 509516. IEEE Comput. Soc.,
Los Alamitos (2001)
6. Davis, J.: Mosaics of Scenes with Moving Objects. In: Proceedings of the 1998 IEEE
Computer Society Conference on Computer Vision and Pattern Recognition, pp. 354360.
IEEE Comput. Soc., Los Alamitos (1998)
7. Pan, J., Wang, M.: A Seam-line Optimized Method Based on Difference Image and
Gradient Image. In: 2011 19th International Conference on Geoinformatics. IEEE,
Piscataway (2011)
8. Yang, Y., Gao, Y., Li, H., Han, Y.: An Algorithm for Remote Sensing Image Mosaic
Based on Valid Area. In: 2011 International Symposium on Image and Data Fusion (ISIDF
2011). IEEE, Piscataway (2011)
9. Zou, Z., Zhang, Z., Zhang, J., Cao, H.: Seamlines Intelligent Detection in Large Scale
Urban Orthoimage Mosaicking. Acta Geodaetica et Cartographica Sinica 40, 8489 (2011)
10. Wan, Y., Wang, D., Xiao, J., Lai, X., Xu, J.: Automatic Determination of Seamlines for
Aerial Image Mosaicking Based on Vector Roads Alone. ISPRS Journal of
Photogrammetry and Remote Sensing 76, 110 (2013)
11. Soille, P.: Morphological Image Compositing. IEEE Transactions on Pattern Analysis and
Machine Intelligence 28, 673683 (2006)
12. Gracias, N., Mahoor, M., Negahdaripour, S., Gleason, A.: Fast Image Blending Using
Watersheds and Graph Cuts. Image and Vision Computing 27, 597607 (2009)
13. Han, T., Zhao, Y., Liu, S., Bai, Y.: Spatially Constrained SURF Feature Point Matching
for UAV Images. Journal of Image and Graphics 18, 669676 (2013)
14. Dijkstra, E.W.: A Note on Two Problems in Connexion with Graphs. Numerische
Mathematik 1, 269271 (1959)
15. Roerdink, J.B.T.M., Meijster, A.: The Watershed Transform: Definitions, Algorithms and
Parallelization Strategies. Fundamenta Informaticae 41, 187228 (2000)