Multinational License Plate Recognition System: Segmentation and Classification

Vladimir Shapiro Georgi Gluhchev

Orbograph Ltd., Institute of Information Techonolgies
P.O.Box 215, Yavne 81102, Israel 1113 Sofia, Akad. G. Bonchev St., bl.29A
vladimir.shapiro@orbograph.com gluhchev@iinf.bas.bg
Abstract
Image-based Car License Plate Recognition (CLPR)
systems provide an inexpensive automatic solution for
remote vehicle identification. Localization stage of the
CLPR yields a gray-scale plate clip with printed
characters. This paper describes the method of plate
clip segmentation into isolated characters, feature
extraction and classification. The method is
independent on character size, thickness, illumination
and is capable of handling plates from various
countries. The method uses extensively the gray-scale
information and is robust to breaks in character
connectivity. It is tolerant to character deformations,
such as shear and skew. Promising results have been
obtained on Israeli and Bulgarian plates.
1. Introduction
While the first industrial automatic systems of CLPR
began emerging in 80-ies, an outburst of commercial
systems occurred in the past 10 years [1,5,6,11].
Although hundreds of CLPR systems are available in
the market worldwide, the research and development
still continues and new sophisticated solutions appear.
This is due to the growing demand for the automatic
vehicle identification required for traffic and border
control, calculation of parking time and payment,
search for stolen cars or unpaid fees, the requirement
for reliable identification at different lighting
conditions, presence of random or structured noise in
the plate, and nationality specific features, concerning
plates size and font. A CLPR system can be
conceptually considered as containing two major
components:
- License Plate Localization (LPL)
- License Plate Optical Character Recognition
(LPOCR)
The LPOCR has a potential to serve as a verifier,
indicating whether the image fragment clipped by the
LPL, referred to as a plate candidate, comprises the
actual plate. Otherwise LPL attempts to find better
candidate [10].
It would be tempting to adopt a commercial off-
the-shelf OCR as recognition engine for LPOCR
system. Available today at very affordable price, such
commercial OCRs are designed to primarily meet
needs of a printed page processing and recognition,
where a bright background of uniform intensity is
implicitly assumed. Achieving read-rates of 98-99% is
not an unusual practice then. Unfortunately, such OCR
packages fail to segment the plate strip properly in very
many cases. License plate imagery is equivalent to a
text, scanned with very low resolution, additionally
hindered by a non-homogeneous background of multi-
modal intensity distribution and by significant, often
unpredictable, noise factors. Various statistics useful
for text segmentation such as e.g. the text pitch [8]
cannot be reliably estimated on a short plate string. All
that causes the OCR performance to drop abruptly.
Plate segmentation, i.e. separation into characters
seems to be the most challenging LPOCR task.
Segmentation, based on connected component analysis
[6], would fail if characters are broken or binarization
is not perfect. It is assumed in [11] that character
connected components are typically well separated as
no more than two characters are connected. This
assumption seems to be too optimistic for the real life.
The paper describes the LPOCR component of the
CLPR system. Its scope is defined in Section 2;
Section 3 considers the feature extraction; the plate clip
segmentation is explored in Section 4. Classification
and post-processing procedures are presented in
Section 5 and 6, respectively. Discussion and
Conclusions follow.
2. Scope of the Method
Figure 1. Examples of a typical source image and
localized plates.
A typical output of the LPL stage [10] after
deskewing is shown in Figure 1. The LPOCR has much
broader responsibility in the described system than just
conventional recognition of isolated characters at the
Proceedings of the 17th International Conference on Pattern Recognition (ICPR04)
1051-4651/04 $ 20.00 IEEE
final stage of the work. The paradigm implies multi-
purpose use of the recognition: at the plate localization
stage the recognition serves as a verifier of the plate
candidates provided by the segmenter, that generates
a number of such candidates. The recognition is
activated in an accuracy-optimized mode for achieving
optimal recognition results after having the best plate
candidate approved.
3. Training and Feature Extraction
The training process is partially supervised in the
sense that character candidates segmented
automatically are visually verified to contain exactly
one character. Then, an isolated character is manually
associated with the corresponding class. At the stage of
recognition, when the system is already trained, the
automatic classifier plays the role of the automatic
verifier.
Training is performed with character templates of
mixed countries of origin. Each template keeps,
however, its origin country label, which will allow
later determination of the plate nationality.
Size and intensity normalization is carried out. Each
character is warped to the matrix of predefined size
that is equal for the training and recognition, here 32 x
60. Gray-scale matrix is normalized through histogram
stretching.
3.1. Thinned Object Representation
Line thickness independent feature ) ( X S called
Object Thinned Representation (OTR) is chosen as a
shape descriptor. Let us assume that the character
rectangle consists of an object X of cardinality
X
K ,
which is X s area, and a background X , which is the
objects complement set of cardinality
X
K . S(X) is
iteratively evaluated according to the formula (1),
where B is a structuring element, is a thinning
operation (see [2]), A is rectangles area and
X
w is a
coefficient:
A w K B X X S
X X i
< = while ; ) (
4 mod
(1)
,...] 1 , 0 [ i and

=
0 0 0
0 1 1
0 0 0
,
0 1 0
0 1 0
0 0 0
,
0 0 0
1 1 0
0 0 0
,
0 0 0
0 1 0
0 1 0
3 2 1 0
B B B B
.
X
w was empirically set to 0.25. Eq.(1) says that the
object is iteratively shrinking as i increments until
X area contracts to A w
X
portion of the rectangle size
A . This technique does not preserve object
connectivity, which is not critical for the classification
method.
The OTR somewhat reminds on a skeleton. It is
sensitive to a certain degree to boundary profile and
tends to produce non-desirably spurious shapes. An
algorithm [3] with proper setting of parameters to
inhibit small but spiky boundary behavior could be
tried as an alternative.
3.2. Characteristic Background Spots (CBS)
Sometimes, a clip submitted for the recognition
represents homogeneous region mostly or fully
populated by approximately uniform gray levels. The
latter clip obviously needs to be rejected. A single
OTR vector would not be sufficient to do so. This is
the reason we introduce another feature vector in
addition to the OTR. The other potentially problematic
situation, that the OTR intends to resolve, is a case
when a character in question represents a subset of the
template, e.g. 3 in question is matched to the
template of 8.
(a) (b) (c)
Figure 2. (a) An original character; (b) its OTR
template; (c) CBS template.
The CBS describes a shape of character image
rectangle occupied by the background. The CBS is
obtained via shrinking the character background by a
number of iterations (a natural way of implementing
that is the morphologic erosion). The shrinking
continues until a certain percentage of the background
area remains unpopulated by the object pixels. A
numbers of iterations required to get CBS depends on
the initial
X
K . Formally, CBS, denoted as
) ( X S
, is
expressed as:
A w K B X X S
X X
i
< = while ; ) (
4 mod
(2)
where ,...] 1 , 0 [ i .
X
w was set to 0.2. Eq.(2) says that
the background iteratively contracts as i increments
until the background area
X
K contracts to A w
X
portion of the rectangle A .
Both OTR and CBS representations are invariant to
a large degree to the original character thickness.
4. The Segmenter
The segmenter aims at separating the plate clip into
isolated character bounding boxes. Zero overlapping is
assumed. The algorithm is based on the adaptive
iterative thresholding and connected component
analysis.
Due to the non-homogeneous illumination and/or
presence of stamps, bolts, etc noise on the plate,
characters might occupy more than one connected
component. Most frequently, the components are
situated one above the other. Thus, we prepare an
extended bounding box of the component by
Proceedings of the 17th International Conference on Pattern Recognition (ICPR04)
1051-4651/04 $ 20.00 IEEE
spanning up to the plate bottom. Separated component
resided below might belong to noise see Figure 4 f.
To solve the problem of disconnectedness, the
character classification algorithm, see Section 5, works
on both the original and the extended connected
components, see e.g. Figure 4c and Figure 4d, and
accepts the one with higher confidence level.
Plate Character Isolation Algorithm Outline (Figure 3)
assess global threshold
g
T [9] and binarize the image
do while (right plate boundary not reached)
begin
find connected components // on the rest of the plate
if (one character) // based on width and proportion
begin
perform classification
jump to the next component
reassess global threshold
g
T //on the rest
end
else // two or more characters
reduce the
g
T by T // reduces connectivity
end
Note: Extra boundary peeling might be required to
eliminate some margins around the plate at the
preliminary stage.
Gray-scale clip 90 =
g
T
90 =
g
T
90 =
g
T
90 =
g
T
87 =
g
T
90 =
g
T
81 =
g
T
90 =
g
T
87 =
g
T 90 =
g
T
Figure 3. Sequential stages of plate segmentation as a
function of adaptive local thresholding ( 3 = T ) (plate
colors are inverted for better visibility)
(a)
(b)
(c)
(d)
(e)
(f)
Figure 4. Non-connected components belonging to same
character: (a),(b) source plates; (c),(e) connected
components found; (d) extension led to proper character
aggregation; (f) extension led to introducing extra noise
As a result of segmentation each successfully
isolated and approved characters bounding box is cut
out from the gray-scale plate clip and presented to the
classification algorithm described below.
5. The Classifier
An unknown character segmented by the algorithm,
described in Section 4, is normalized to a predefined
size. Thus, the unknown character and the templates
(OTR and CBS) are of same size and can be analyzed
element-wise.
Each of the two templates serves as a mask for
extracting the gray-scale pixels from the zones in
which the template hits the unknown character
matrix; see Figure 5. In the case of matching the
unknown character against its intra-class feature
vector, it is reasonable to anticipate that the OTR
would mask primarily the pixels belonging to the
character, and CBS mostly the background ones.
Oppositely, when the unknown character matrix is
matched to other class vectors, the OTR would hit
many of background pixels and CBS many of the
object pixels.
To estimate the similarity quantitatively we
introduce the following distance measure
k
D between
an unknown character gray level matrix G and a
template
k
C
T , where
k
C is a class label and
N k ,..., 0 = number of classes i.e. allowed characters in
license plate alphabet.
), ( ) (
CBS
CBS
OTR
OTR k
k
C
k
C
T G Var w T G Var w D + = (3)
where Var is the variance;
OTR
w and
CBS
w are weight
coefficients set to 1.0 and 0.9, respectively.
k
D is
expected to be minimal for the unknown characters
native class
k
C , i.e.
) ( min arg
k k k
D C = (4)
Plates, having high
k
C , do not resemble text-like
information [7] and are thus rejected.
(a)
(b) (c) (d)
(e) (f)
Figure 5. (a) License plate gray-scale matrix; (b)
character after segmentation; (c) characters bounding
box clipped from (a); (d) same, after size normalizati on;
(e) its masking by one of OTR templates of class 9; (f)
its masking by a corresponding CBS template
Compared to the OCR carried out on purely black-and-
white image matrix of normalized size, the gray-scale
recognition gives a boost of 10-12%.
The above-described classification procedure is
Proceedings of the 17th International Conference on Pattern Recognition (ICPR04)
1051-4651/04 $ 20.00 IEEE
fully generic, not limited to the license plate context. It
could serve recognition of any printed texts with noisy
environment.
6. The Postprocessor
The postprocessor is responsible for refining
recognition results by making use of the specific
license plate context.
A regular mode of classification extensively used at
the segmentation phase is the so-called international,
i.e. when templates of all countries of origin are mixed
within the same pool. Although the algorithm is, to a
large degree, independent on the template font used,
more precise classification results are obtained when
native country templates are applied in matching.
Thus, the postprocessor automatically determines the
country locale of the current plate. The system has
been tested on plates from Israel and Bulgaria, and its
architecture is open for further geographic extension.
The benefit of applying a proper country locale is
also due to a difference in country alphabets. Both
contain digits; the Bulgarian alphabet incorporates
several capital letters. Implementation-wise the
majority of characters recognized in the international
mode plus some grammar-inspired clues, determine the
country of origin. The additional semantic-based
refinement pass of classification picks up only the
native country templates, which allows improving the
read-rate furthermore.
(a)
(b)
(c)
(d)
Figure 6. (a),(b) Bulgarian plate characters; (c),(b)
same characters from Israeli plates
The semantic rules vary from country to country. In
Bulgaria, the plates start with a letter or two, followed
by a space, several digits and a couple of letters, see
e.g. Figure 1. The grammar model could be
schematically designated as L[L] DDDDLL, where
L and D stand for a letter and a digit, respectively.
Israeli plates strictly follow the rule: DD-DDD-DD.
On the test sample of 500 plates, 81.2% of plate
candidates have been approved at the localization
stage. The percentage of wrongly approved plates
(false positives) was 0%, as the rest to 100% plates
have been rejected. Character level read-rate versus the
misread was 85.2% / 3.4%.
7. Discussion and Conclusions
The above-described system achieves several
important goals:
- Allows a very reliable verification of a plate
candidate generated at the phase of localization
- Adaptively segments the plate image coping with
tough illumination conditions and image
distortions
- Classifies gray-scale characters of variable size
and resolution with a very reasonable accuracy.
As mentioned in Section 1, OCR packages are
incapable of delivering a magic-wand solution,
however, they could be useful when the segmentation
task is completed, and characters are properly
separated and clipped. Then, a relatively high read rate
could be anticipated. At least on most of the readable
by a human eye plates, we could expect read rate levels
of 90% and higher. The experiments are on going.
Further directions of the research lay in applying
approaches known in a context of conventional
OCR/ICR systems as multi-expert combination, or
voting [4]. Using RGB cameras with known in
advance plate background/foreground colors would
allow higher precision in character isolation. Applying
the algorithm to multiple frames of the plate video
stream would let additional gain in accuracy [1].
8. References
[1] Y.T. Cui and Q. Huang: Extracting Characters of License
Plates from Video Sequences, Machine Vision and
Applications vol 10, 1998, pp. 308-320
[2] A.K.Jain, Fundamentals of Digital Image Processing,
Prentice Hall, Englewood Cliffs, NJ, 1989
[3] W-B. Goh and K-Y. Chan, Part-Based Shape
Recognition Using Gradient Vector Field Histograms,
CAIP03, LNCS 2756, 2003, pp. 402-409,
[4] Y.S. Huang and C.Y. Suen, Combination of multiple
experts for the recognition of unconstrained handwritten
numerals, IEEE PAMI-17, 1995, pp. 90-94
[5] L. Jilin, M. Hongqing, L. Peihong, A High Performance
License Plate Recognition System Based On The Web
Technique, IEEE Conf. On Intelligent Transportation
Systems, Oakland, CA, 2001, pp.14-18
[6] S-H. Lee, Y-S. Seok, E-J. Lee Multi-National Integrated
Car-License Plate Recognition System Using Geometrical
Feature and Hybrid Pattern Vector, Conf. Circuits/Systems,
Computers and Comm., Phuket, Thailand, 2002, pp. 1256-
1259
[7] R. Lienhart and W. Effelsberg, Automatic Text
Segmentation and Text Recognition for Video Indexing,
ACM/Springer Multimedia Systems, vol. 8, 2000 pp.69-81
[8] Y. Lu, Machine Printed Character Segmentation - An
Overview, Pattern Recognition, vol.28, 1995, pp.67-80
[9] N. Otsu, A Thresholding Selection Method from Gray
Level Histograms, IEEE Trans. Syst., Man, Cybern., vol.9,
1979, pp.62-66
[10] V. Shapiro, G. Gluhchev, S. Bonchev, V. Velichkov,
License Plate Localization, Automatics and Informatics03,
Sofia, 2003, pp. 41-43
[11] M. Shridhar, J.W.V. Miller, G.Houle, L. Bijnagte,
Recognition of License Plate Images: Issues and
Perspectives, ICDAR, Bangalore, India, 1999, pp. 17-20
Proceedings of the 17th International Conference on Pattern Recognition (ICPR04)
1051-4651/04 $ 20.00 IEEE