ASeminarReporton

BARCODERECOGNITIONINCOMPLEXSCENESBY CAMERAPHONES
Submittedinpartialfulfillmentoftherequirements fortheawardofB.TechDegreein ElectronicsAndCommunicationDept. by JineshRS

AUGUST2010

ADISHANKARAINSTITUTEOFENGINEERING &TECHNOLOGY
(AnISO9001CertifiedInstitution) VIDYABHARATHINAGAR,KALADY,KERALA

ADISHANKARAINSTITUTEOFENGINEERING &TECHNOLOGY
(AnISO9001CertifiedInstitution) VIDYABHARATHINAGAR,KALADY,KERALA

CertifiedthatthisistheBonafideRecordoftheSEMINARentitled BARCODERECOGNITIONINCOMPLEXSCENESBYCAMERAPHONES submittedduringtheyear2010inpartialfulfillmentfortheawardofBachelorof TechnologyinElectronics&Communicationengineeringbythecandidatewith Name:JineshRS RegisterNo:56028 Branch:ElectronicsandCommunication StaffIncharge

2

Certificate

H.O.D

Date....

ACKNOWLEDGEMENT

ItisagreatpleasuretoexpressmysinceregratitudetoDr.S.G.IYER, Principal,ASIET,forhisguidanceadviceandencouragementthroughoutthis endeavor.I'malsogratefulforgrantingmeallthefacilitiesfortheseminar presentation. I'mgreatlyindebtedtoHOD,Electronics&CommunicationDept.,forthe helpandguidanceatstagesofmywork. Itakethisopportunitytoextentmysinceregratitudeandthanksto Ms.Anju,seminarguideforherguidance&helpinconnectionwiththisseminar. Iwishtoexpressmygratitudeandsincerethankstoallthosewhohelped mebygivingtheirvaluableadviceandguidance.
3

ABSTRACT

Mobilephonewithcamerafunctionsarecapableofcapturingimageandprocessing task. Mobile phone canuseforcapturingbarcodewith their cameras anddecodingthem with software running on the phone. Barcode is a fast, easy and accurate automatic data collection method.Barcodeenablesproducttobetrackedefficientlyandacuratelyatspeedsnotpossible usingmanualdataentrysystem.Quickresponsecodeisa2Dbarcodewhichhasbeenwidelyused inindustrialinformationtaggingapplicationswherehighdatacapacityandrobusterrorcorrections arerequired.Indealwiththeinfuencebythedifferentlightconditionsandnoisearecognition algorithmisused.First,wecombinefilter,roughlocationandbinarizationtoerasenoisesand redusecomputation.Then,weproposeanaccuratelocationandorientation,skewcorrectionwith thehelpofalignmentpatternalgorithmtobuildabarcodegrid.Finally,weuseerrorcorrectionand decodingprocesstogeneratetheresult.

TABLE OF CONTENTS
ACKNOWLEDGMENT...................................................................................I ABSTRACT.....................................................................................................II LIST OF FIGURES.....IV CHAPTER. 1 INTRODUCTION......................................................................1 CHAPTER. 2 WHY USE MOBILE PHONES.3 CHAPTER. 3 BAR CODE................................................................................4 3.1 DATA MATRIX.4 3.2 QR CODE....6 3.2.1 FEATURES OF QR CODE...7 CHAPTER. 4 VARIOUS ALGORITHMS...........8 CHAPTER. 5 RECOGNITION ALGORITHM...10 5.1 GRAY CONVERSION......10 5.2 FILTER AND LOCATION........................................................11 5.3 BINARIZATION........................................................................12 5.4 ACCURATE LOCATION AND ORIENTATION.....................13 5.5 CORRECTION THE DEFORMED SHAPE...............................14 5.6 ERROR CORRECTION AND DECODE...................................16 CHAPTER.6 EXPERIMENTAL RESULTS....................................................17 CHAPTER.7 RECOGNITION TEST...............................................................19 CHAPTER.8 CONCLUSION AND FUTURE WORK...................................20 REFERENCES............................................................................21

LISTOFFIGURES
FIGURE1..4 FIGURE2..6 FIGURE3.14 FIGURE4.15

CHAPTER1

INTRODUCTION

Modernmobilephonesarehighresolutioncolourdisplays,theysupportdifferent standards of wireless networking, and they have reasonable processing power. Although still primarilyusedforvoicecommunication,withtheinclusionofdigitalcamerasthesedeviceshave becomeapotentialplatformformachinevisionapplicationsuchasbarcoderecognition.

In recent years, advanced technology has succeeded in continuously producing

smalleryetsmarterdevices.Nowmobilephonescanimplementmanynewkindsofapplications suchastakingphotos,andmovieshootingbyusingembeddedcameradevices.Soaninteresting approachiscapturingbarcodewiththeircamerasanddecodingthemwithsoftwarerunningonthe phoneIn August 2006, 82.4 percent of the respondents whohad camera phones with QR Code (QuickResponseCode)readersusedcameraphoneswithQRCode.Butpreviousresearchworkhas shownthatrecognitionof2Dbarcodeinmobilephoneisverydifficultbecauseofthehighnoise, nonuniformillumination,skewdistortion,lowresolutionandopticalblur.Itisverydifficultto robustlyextractaccuratefeaturessuchasedgesandpeaksofthebarsandspacesfromthebarcode

imagestakenbyacameraphone.

Many new algorithms are presented for dealing with 1D bar code in complex situation.Ohbuchietal.presentedanalgorithmcapableoftherealtimerecognitionofbarcodesona mobilephone.Sunetal.introduceanalgorithmtoanalyzeandcorrectthedistortedimageofQR Code.Thealgorithmincludesgrayscaleimagetransformation,binaryimage,cannyedgedetection, external contours finding, inverse perspective transformation and cell grids generating. In this method,binaryimageandedgedetectionisveryimportantforfollowingdecode.Butitisvery difficulttobinariesimageandextractaccurateedgeintheblurredanddamagedbarcodeimages. Herewedescribeanewimagerecognitionalgorithmwhichappliedtocaptureimageinvarious lightcondition.

10

CHAPTER2

WHYUSEMOBILEPHONE

BARCODEREADERS

TooExpensiveScanners. HaveNoOtherUse. TooBulky. WorkersHaveToBeTrained. 11

MOBILEPHONES

AffordablePricesAndCommon. HaveVariousUses. TooHandy. NoTraining.

CHAPTER3

BARCODE

TherearetwotypesofbarcodesarepresentOnedimensionalandtwodimensional. 12

1Dbarcodeshavelowdatacapacity,sowecommonlyuse2Dbarcodes.Thereartwostandardised 2DbarcodesDatamatrixandquickresponse(QR)code.

3.1DATAMATRIX

Datamatrixisoneofthemostwellknown2Dbarcodestandards.Itiswidelyused in the automotive, aerospace and computer manufacturing industries, for large data capacity labelling,suchasdirectpartmarkingandpackagemarking.

Figure1
Itconsistofasolidlinelocator(thetwosolidline),apatternedlinelocator(thetwo alternatingdarkandlightpatternedlines),theinsideareawithencodeddatainbinary,andaquiet zone(ablankarea)surroundingthewholetag. 13

Its capacity is 2334 alphanumeric characters or 1556 8bit ASCII characters, encodinganydata.ItemploystheReadSolomonerrorcorrectiontoenableaccuratereadseven whensubstentialpatsofthecodearedistorted.

14

3.2QRCODE

It comprised of the following patterns: finder pattern, timing pattern format information,alignmentpattern,anddatacell.Thefinderpatternslocatedatthethreecornersofthe symbolintendedtoassistineasylocationofitsposition,sizeandinclination.Threedarklightdark squersareoverlappedineveryfinderpattern,andthedarklightratiois1:1:3:1:1.Thetiming patternsprovidethesecondaryinformationwhichcanhelpustolocatethesymbol,decidethe rotationdirectionofsymbolandthewidthofmodule.Thelocationofalignmentpatterncenteris importantforcorrectingthedeformedshape.

15

Figure2

3.2.1FeaturesofQRcode

1) Highcapacityencodingofdata QRcodehashighcapacityencodingofdata,itsmaximumsymbolcanencode7089numericdataor 4296alphanumericdata.

2) Highspeadreading Adaptedwithccdreading,itcanrecognizemoreqrcodesymbolpersecondthanpdf417symbolfor encodingsamedatacapacity.

3) Chineseencodingcapability Chinese and Japanese characters are represented by a two byte compination in other two 16

dimensionalbarcode.ButinQRcodethereisaspecificchinesemode,itcanuse13bitsencodinga chinesecharacter. 4) Readablefromanydirectionfrom360degree

CHAPTER4

VARIOUSALGORITHMS

Many new algorithms are presented for dealing with 1D bar code in complex situation.

17

InImage Pre-processing for Bar Code Detection in Mobile Devices , a de-noising and vignetting elimination is performed first, followed by the start and the ending points detection, finally psf estimation and line de-blurring is adopted.

A statistical method is proposed in which, wavelet-base and knowledge-base approach is used to locate the bar code area and segment the bar code character

Because of the feature of 2D bar code is different from 1D bar code, the above algorithm used in 2D bar code is not well.

Ohbuchi et al. presented an algorithm capable of the real-time recognition of barcodes on a mobile phone. The most important drawback of the method is the fact that it has been hand tailored for one certain hardware device. It relies on access to a powerful but also very specific hardware element, which is not accessible by normal application developers. Besides that, the algorithm relies on two specific conditions: First is prior to the codes decoding, the barcodes position is detected using a spiral scanning algorithm that runs on the devices signal processor. Second is this algorithm makes the assumption that the point in the middle of the screen is located in the code. Thisassumption is not realistic in usually environment. So the algorithm is difficult to be used in 2D bar code decoding.

An algorithm capable of recognizing the PDF 417 barcode is proposed in real time with a mobile phone. This method includes three steps. The first step detects the code region using the Otsu algorithm and the Least Square Method. The second method searches for the cutoff rules with a scanning approach. In third step symbol characters are segmented from the original image. So the successful binarization of the code areas during the first step is the most key step in this algorithm. This is mean that the recognition result will rely on the ef18

fect of the Otsu method. But the Otsu method could be failed in complex lighting conditions, including highlight spots, low contradistinction, nonhomogeneous lighting, and various mixed conditions . Additionally, many constraints are added in this method, so the application scope is decreased.

Sun et al. introduce an algorithm to analyze and correct the distorted image of QR Code. The algorithm includes gray-scale image transformation, binary image, canny edge detection, external contours finding, inverse perspective transformation and cell grids generating. In this method, binary image and edge detection is very important for following decode. But it is very difficult to binaries image and extract accurate edge in the blurred and damaged bar code images.

An efficient scheme for 2D bar codes decoding is proposed in [9], which specific for low quality cameras. The algorithm includes Hough transformation, BCRL(location with horizontal and vertical waveforms), grid vibration (Bayesian decision), and deformable model (psf estimation) and decode. In this method, the recognition time will be cost more than above because there is no binarization operation.

CHAPTER5

RECOGNITIONALGORITHM
19

It consists of several steps, gray scale image conversion, filter and location, binarization,andnormalizationtheskewshape,errorcorrectionanddecoding.Theinputisacolor imagewhichiscapturedbymobilephoneandtheoutputisadecodingresult.

5.1GRAYCONVERSION

QRCodesymboliscapturedbymobilephonewithcamera,andimagesarecaptured inRGB24bitformatbymostphones,butQRCodesymbolisasetofdarkandlightpixels.Itis needlesstodealwithcolorinformationandthegrayimagecalculatedquicklywithlittlespace,so grayconversionisneededtodofirstly.

20

5.2FILTERANDLOCATION

Performmorphologicalclosingonthegrayscaleimagefromabovestep.Thiswill smooththebarcodeandreducetheinfluenceofnoise.Beforebinarizingtheimage,abarcodearea inimageisextractedthatourapproachcanreducecomputationalpowerandincreasetheaccurate rate of binarizing bar code. There are several methods to detect code area, such as pattern searching, texturesegmentation, waveletanalysis etc. Compared with the background, the gray changefrequencyofthecodeareaisquitehigh,sowecaneasilygettheroughbarcodeareain frequencydomain.

21

5.3BINARISATION
Binarization is a necessary step in recognition bar code, and it is used in most recognitionalgorithm.Selectionofaproperbinarizationmethodiscriticaltotheperformanceof barcoderecognitionsystem.Inbinarizinganimage,asimpleandpopularmethodisthreshold. Ohbuchi proposeanimprovingglobalthresholdmethod.Firsttheimagewasdividedinto nine blocks.Thenthegrayhistogramiscalculatedforeachblock,andthegrayvaluewassorted.The middlevaluewaschoosingasthethresholdofeachblock.Finallythesmallestvalueofthesemiddle valuesistheglobalthresholdofthiswholeimage.Thismethodisexistedtwodrawbacks:(1)the barcodesymbolmustbeinthecenterofthecapturedimage,otherwisethemethodwillfailed.(2) This method results in excessive segmentation. Using a global threshold method, the resulting binaryimagewillbeverybad,ifanimagewasinvariablelightingconditions.Inthiscase,alocal thresholdmethodperformsbetter.Themainproblemswithalocalthresholdmethodarehardtoset 22

arightwindowsize,eliminatetheblockeffect,andreducetheexecutiontime.

Thebarcodeareaisgettinginpreviousstep,sothenonuniformilluminationhas littleeffectonbarcodeareacomparingwiththewholeimage.Amongmorethan20threshold methods,itisconcludedthatOtsusmethodisthebest,whichchoosesthethresholdthatminimizes withingroupvariance.SoweusedtheOtsustechniqueasthethresholdalgorithm.

5.4ACCURATELOCATIONANDORIENTATION

Afterbinarizingthecodearea,theaccuratelocationandorientationisimplemented. TherearethreeidenticalpositiondetectionpatternslocatedatthreeofthefourcornersofQRCode. Three darklightdarksquaresareoverlappedineveryfinderpattern,andthedarklightratio is 1:1:3:1:1.Thereistheoffchancethatsimilargraphicexistedinbarcodesymbol.Sotheratioarea shouldbequicklyfind.Butwhenoneofthefinderpatternsispartiallydirtyordamaged,wecanuse thetimingpatterns.Thetimingpatternsprovidethesecondaryinformationwhichcanhelpusto 23

locatethesymbol,decidetherotationdirectionofsymbolandthewidthofmodule.Rotatedimages are handled after orientation in common. But the interpolation operation is used in rotating operation. The amount of calculation is great and is not accurately. So we can skip this step, modular distance offset algorithm was used without rotating symbol. After located the finder patterns, the module widthandheightand theangleofrotationwereknown.Wegetthe grid movingalongthelinewhichconnectsthecentralofthreefinderpatterns.Thismethodavoidsthe rotationandinterpolation,increasesthecomputingspeed.

5.5CORRECTIONTHEDEFORMEDSHAPE

Theinputimagehasadeformedshapebecauseofbeingcapturedfromtheembedded
24

cameradevice.,Aninverseprojectiontransformationisusedtonormalizethecodesharp.Butthe specificfeatureofQRCodesymbolisnotusedinthismethod,sothecomputationcomplexdegree algorithmishigh.Differentfromtheother2Dbarcode,inordertocorrectthecontortedQRCode symbol,therearemanyalignmentpatternsinQRCodesymbol.Withtheversionincreased,the numberofalignmentpatternswasadded,asshowninFig.Whentheversionis3,thenumberof alignmentpatternsis1,butwhentheversionis7,thenumberofalignmentpatternsis6.Linkthe centralpointofthealignmentpatternsandthreepositiondetectedpatterns,thesmallsamplinggrid isformed.Inasmallsamplinggrid,distortionislittle,sowecombinetheaffinetransformationand thecentralcoordinatesofalignmentpatterntocorrectthedeformedshape.

Figure3

Thelocationofalignmentpatternscenterisimportantforcorrectingthedeformedshape.
Ininternationalstandard,thelocationalgorithmofalignmentpatterncenterwillbefailedwhenthe barcodesymbolisdeformed,becausethe70percentof estimatelocationofalignmentpatterncenterisinsidethealignmentpattern,theother30percentis

25

outside of alignment pattern. So we proposed an eightdirection estimation algorithm, it can improvetheaccuraterate,and96percentofalignmentpatterncenterwassuccessfullocation. AssumethepointAistheestimationlocation,asshowninFig,wecansearchtheeight directionofpointAinordertofindtheareaofapproximatelydarklightratiois1:1:1:1:1.An additionalconditionisthattheeightneighborpointsofalignmentpatterncenterareallblack,sothe pointCisomitted.Finally,wecangetthepointEisthealignmentpatterncenter.

Figure4

26

5.6ERRORCORRECTIONANDDECODE
When the finder patterns and alignment patterns were located successfully, the segment iseasytodo,andthegridiseasytogenerate.Thenthepixelsweregettingtoready decoding.

Theerrorcorrectionanddecodingprocessisthelaststepofrecognitionbarcode.It employstheReedSolomonerrorcorrectiontoenableaccuratereadsevenwhensubstantialpartsof thecodearedistorted.Decodingisjustthereverseoftheencodingprocedureandthedecoding stepscanreferencetointernationalstandardofQRCode.

27

CHAPTER6

EXPERIMENTAL RESULTS
This application is implemented in our test hardware architecture. An Intel XScale processor of Motorola, Motorola A1200r is run at up to 312MHz. In addition to the processor core, its associated caches and memory management units, including additional 512MB Micro-SD, and camera interface (twenty thousand pixel CMOS sensors) and LCD controller etc.. The barcode image processing algorithm is developed with windows mobile 5.0 and embedded in this mobile.

28

A sample set of 100 unconstrained QR Code barcode image were tested to verify the proposed recognition methods. All original images are acquired by camera phone. The images captured have a resolution of 640*480 pixels. The experiments were done in real conditions. In order to evaluate the validity of our method, these images were divided into four groups: the first group barcode image was captured in normal lighting conditions; the second group image was captured with highlighted spots and background noise; the third group image was captured in low contrast or non-homogeneous lighting conditions; the fourth group image was captured with deforming barcode.

In order to confirm the effectiveness of the proposed recognition method, which including filter location, binarization, accurate location and orientation of finder pattern, alignment patterns location from eight directions, and decoding. We evaluated it with whole recognition performance. For the evaluation, the barcode image was captured and decoded in mobile phone. The recognition rate and executing time were compared in Table . The experimental results have shown that the barcode recognition rate is higher (the first group image), average 97%, and the recognition time is litter in the normal lighting conditions, about average 400ms. With the lighting conditions became weak or uneven, the captured barcode image can be recognized slower and the recognition rate decreased. The four group is the deformed barcode image, so the recognition rate is the slowest, decode an image will cost average more 500ms. In a whole, the experiment verified the proposed method is efficient, and higher recognition rate was obtained.

Compare of different group Group Thefirstgroup Thesecondgrop Thethirdgroup Thefourgroup Theaverageoftotalrecognition time(ms) 380 430 470 540 Recognitionrate 97.5% 95% 91% 87%

29

CHAPTER7

RECOGNITIONTEST

Inordertoconformtheeffectivenessoftheproposedrecognitionmethod,which includingmultilevelthresholdingalgorithm,determinationofthelocationandorientationofthe finderpattern,locatedalignmentpatternsfromeightdirections,anddecoing,weevaluateitwiththe 30

wholerecognitionperformance.Fortheevaluation,wecapturedmorethan70barcodeimageand runrecognitionalgorithmtakenbyourtesthardware.Therecognitionresultwerecomparewithtwo other techniques, the reference algorithm of QRcode standard, and Ohbuchis method. The experimentalresultaresummarisedintable.Experimenthaveshownthatover98%ofthecaptured imagecanbesuccessfullyrecognizedwiththeautomaticrecognitionmethod.Higherrecognition ratewasobtainedwithouralgorithmascomparedwithothermethods.

Binarizationtime Referencemethodof standard Ohbuchsmethod Ourmethod 40 20 30

Totalrecognition time(ms) 90 60 70

Recognitionrate 75% 83% 98%

CHAPTER8

CONCLUSIONSANDFUTUREWORK
31

,
Thebarcoderecognitionsystemdiscussedasimplebutuniquemobileapplicationsystem thathasahugepotentialtobewidelyuseddue.Thisapplicationfacilitatesconsumerswithquick andeasywaytoverifyproductsstatusbyenablingthemtocapturethebarcodeimagesratherthan typinginthebarcodenumber.Inthefuturewewillenhanceoursystemtoincludemoreapplications thatarerelatedtoconsumerindustry,notnecessarilyrelatedwithHalalproducts.Usabilityand usertestingistobecarriedoutinthelaterstage.Wewillenhanceourworktofindthebestwayto solvetheproblemofhighlydistortedandlowresolutionbarcodeimagesbyimprovingthework.

32

REFERENCE

H.Kato.T.Tan.2DBarcodesformobilephones,2005 2nd InternationalConferenceon MobileTechnology,ApplicationsandSystems,1517Nov.8pages.2005.

Wang,Kongqiao,etal.Barcodereadingfromimagescapturebycameraphone,20052nd InternationalConferenceonMobileTechnology,ApplicationsandSystems.1517Nov.2005.

YueLiu,MingjunLiu.AutomaticRecognitionAlgorithmofQuickResponseCodeBased onEmbeddedSystem, ProceedingsoftheSixthInternationalConferenceonIntelligent SystemsDesignandApplications(ISDA06),Volume02,Nov.2006,pp.783788.

DecodingwithCameraEquippedMobilePhones,Proceedings of the Fifth Annual IEEE International Conference on Pervasive Computing and Communications Workshops (PerComW'07), 19-23, March,2007,pp.597600

Hao Wang, Yanming Zou. Camera Readable 2D Bar Codes Design and Decoding for Mobile Phones, IEEE International Conference on Image Processing, ICIP06,8-11 Oct., 2006,pp.469-472.

P.K. Sahoo, S. Soltani, and A.K.C. Wong. A survey of thresholding techniques, Computer Vision, Graphics, and Image Processing, vol. 41, issue 2, Feb., 1998,pp.233-260.

33

34