You are on page 1of 6

PresentedatIASTED98,Waikiki,HI,August1214,1998,

FASTFRAMEHYBRIDOPTOELECTRONICNEURAL
OBJECTRECOGNITIONSYSTEM
TAHERDAUD,TIENHSINCHAO,ANDANILKUMARTHAKOOR
CenterforSpaceMicroelectronicsTechnology
JetPropulsionLaboratory,CaliforniaInstituteofTechnology
4800OakGroveDrive,Pasadena,CA911098099

ABSTRACT
The Hybrid Optoelectronic Neural Object
RecognitionSystem(HONORS)developmentatthe
JetPropulsionLaboratory(JPL)promiseshighspeed
(>1000 frames/s) processing of large frame size
(1000x1000pixel)images.Itconsistsoftwomajor
building blocks: (1) an advanced grayscale optical
correlator(OC);and(2)amassivelyparallel,VLSI
basedneural3Dprocessor(N3DP). TheOC,with
itsinherentparallelprocessingandshiftinvariance,
is used for target of interest detection and
segmentation. The N3DP, with its high speed
(1012 /s)convolutionoperationsandneurallearning
capability is used for target classification and
identification. This paper presents the system
architectureandprocessingalgorithms. Inaddition,
the results from simulations and experiments,
including the detection, classification, and tracking
oftanksandaircraftaresummarized.

KEYWORDS:
Object recognition, optoelectronic, MACH filter,
neuralnetworks,3Dprocessor.

INTRODUCTION
Automatic target recognition (ATR) applications
requirehighspeedprocessing,recognitionofobjects
from clutteredbackground,andintelligent decision
making. Acompact, highspeed ATR system isa
movingtarget.Enhancementinprocessorspeedsgo
handinhandwithbetterandlargerimagersrequiring
higher data processing rates and autonomy in
processing and decision making. Various civil,
military,andspaceapplicationswouldseeordersof
magnitudeadvancement intheircapabilitieswitha
systemperformingATRat1000framespersecond
especially with large format (1000x1000pixel)
imagers.
JPLisdevelopingahybridoptoelectronic
neural object recognition system (HONORS) for

highspeed detection, segmentation, classification


and identification of objects from noisy/cluttered
backgrounds. Advantages of HONORS include
highspeed (<1 ms per frame), large input frame
(1000X1000pixel), high discrimination accuracy
(>90%),andeaseoftraining.HONORSconsistsof
an optical correlator (OC) and a neural 3
dimensional processor (N3DP). The OC performs
wide area surveillance for target of interest (TOI)
detection and segmentation. The N3DP
subsequently classifies the input TOI as a feature
vectoramongstmultipleclassesofobjects.

SYSTEMARCHITECTURE
ThesystemblockdiagramofHONORSisshownin
Figure1. TheentireATRfunctionisimplemented
infiveconsecutivesteps[1]:
(1) Sensor data formats are fed through a frame
buffer(FB)device.
(2) After format conversion, FB feeds OC for
preprocessing for target of interest (TOI)
detection and segmentation based on target
shape,size,texture,andgrayscaledistribution.
Training target images are computed with a
distortion invariant correlation filter algorithm
anddownloadedintotheopticalcorrelator.
Use W ord 6 .0c or later to

vi ew Ma ci ntosh pi cture.

FIGURE 1. SYSTEM BLOCK DIAGRAM OF


HONORSFORATR.
(3) With a cue from OC, a column loading input
chip(CLIC)(apartoftheN3DP)acquiresthe
segmentedTOIimagessavedinFB.

(4) TOIimagesarefedtothecubeconsistingof:an
eigenvectorbased feature extraction 3D
stacked electronic processor and an analog
neuralnetworkclassificationchipforfinaltarget
classificationandidentification.
(5) TheoutputofN3DPisaviableinputfortarget
tracking, navigation & guidance, sensor
retasking,andmissionreplanning.
A highspeed camcordersized grayscale
optical correlator [24] consisting of a grayscale
inputSLMreplacesthebinarySLMusedinprevious
stateoftheart optical correlator systems. A
ferroelectric liquid crystal (FLC) SLM capable of
encoding realvalued data is also used for the
correlationfilterimplementation.
A system architecture for this innovative
OC and a photograph of the palmcorder size
grayscaleOCisshowninFigure2.A25mwdiode
laser,emittingat690nm,iscollimatedandusedas
the light source. A 640x480pixel liquid crystal
display(LCD) with a2.4mm pitch isused asthe
inputSLM. TheFLCSLMisusedasafilterand
operatesinatransmissionmodewhiletheLCDSLM
operates inareflection mode. Theinput imageis
Fouriertransformedanddirectedtoaddressthefilter
SLMviaapolarizingcubicbeam splitter. Ahalf
waveplateisinsertedbetweenthebeamsplitterand
thefilterSLMtosteerthepolarizationorientationof
the throughput light beam to ensure bipolar
amplitudeoutputfromtheFLCSLM. Thereadout
beam,reflectedbackfromthefilterSLMisinverse
Fouriertransformedatthe

Use Word 6.0c or later to

view Macintosh picture.

FIGURE2. ASCHEMATIC DIAGRAMANDA


PHOTOGRAPH OF THE PALMCORDERSIZE
GRAYSCALEOPTICALCORRELATOR.
outputcorrelationplane.AhighspeedCCDisused
tograbtheoutputforpeakdetection.
TheinputSLMprovidesa15.4mmx11.2
mmaperture,verysuitable forcompact packaging.
The contrast ratio is about 100:1 with an 8bit
grayscaleresolutionwhenusedforimagingdisplay.
TheSLMisaddressedinaVGAgraphicmode.The
key device leading to the realvalued filter
modulation isthe 128x128 analog FLC SLM built
using liquid crystal on silicon technology [1]. It
utilizeshightiltFLCmaterialresultingintheuseof
allpositivereal amplitudes,binaryphaseonly,and
bipolaramplitude modulations easily obtained by
varyingtheorientationofthehalfwaveplateplaced
infrontoftheSLM.TheFLCSLMhasaswitching
timeof50to100msthatprovidesaframerateupto
10kHz.Thecontrastratiois76:1.Anestimated46
bitoutputresolutioncanbeachievedbyusingan8
bitgrayscaleOCresolution.

OCFILTERALGORITHM

Use Word 6.0c or later to

view Macintosh picture.

The unique realvalued filter modulation capability


ofthegrayscaleOChasenabledustoselectamore
robust correlator filter algorithm for optical
implementation. A distortion invariant correlator
filter algorithm, MACH (maximum average
correlation height) [1,57], has been selected and
implemented. An overview of the MACH filter
algorithmhasbeengivenelsewhere[1,7].
Basically, the filter is synthesized as a
composite of several training images (samples
pickedfromthetargetdatabase).AMACHfilteris
desired to generate an output correlation plan that
has a highpeak response with respect to all input
imagesfromthesameclass.TocomputetheMACH
filter transfer function, h , a mean square error
criterion referred to as the average similarity
measure (ASM) is used as a metric for distortion.
Thefilterresponseismoreinvariantwithrespecttoa

smaller ASM. Therefore, in filter design, it is


requiredthat h haveahighcorrelationpeakwiththe
meanimagewhilemakingtheASMsmall.Itisalso
required to possess noise tolerance to reduce the
outputnoisevariance(ONV).

Use Word 6.0c or later to

view Macintosh picture.

The filter h maximizes the height of the


mean correlation peak relative to the expected
distortion. The superior performance of MACH
filter is attributed to the inclusion of the ASM
criterion which reduces the filters sensitivity to
distortionsandtotheremovalofhardconstraintson
the peak. This permits the correlation planes to
adjust to whatever value best permits the
optimizationofperformancecriterion.

OPTICALMACHFILTERDEMONS
TRATIONOFDISTORTION
INVARIANTTARGETDETECTION
In a laboratory experiment for distortion invariant
targetdetectionusingopticallyimplementedMACH
filter,asequenceof15IRimagesofatankmoving
downwardadesertterrainisusedastheinput. In
preparing the MACH filter, 5 of the inputs were
selectedfortrainingandtherestusedfortesting.A
MACHfilteriscomputedandthendownloadedinto
thefilter SLM. InFigure3,resultsoftheoptical
correlatoroutput areprovided. All15inputswere
successfully detected with a single MACH filter
trainedforrecognizingthetankimages.Fourofthe
images are shown in Fig. 3 (a), and their
corresponding correlation peaks and 3D plots are
shownin(b) and(c)respectively. Notice that the
correlation peaks remain sharp and uniform across
alltheinputimagesincludingthosewithscaleand
perspectivevariations. Thescaleratiobetweenthe
topandbottomimagesisabout2:1.Thisexperiment
validates that an optical correlator is an effective
targetdetectionpreprocessor.

(a)

(b)

(c)

FIGURE3.TARGETDETECTIONRESULTSOF
ANOPTICALMACHCORRELATORFILTER.

NEURAL3DPROCESSOR(N3DP)
N3DPisaground/airborneimageprocessing/target
recognition processor being developed as an
enhancedversionofa3dimensionalartificialneural
network (3DANN) processor architecture. Its
successiscoupledwiththeinnovativeeigenvector
based algorithm development anddemonstrationin
simulation.
ARCHITECTURE: A block diagram of the front
partoftheN3DPconsistingofaCLICandanNPM
cube (without the subsequent multilayer neural
network) is shown in Figure 4. It also shows a
photographofthehardware.ThefunctionofN3DP
is to have a 64x64pixel image window as an
electronicinputtobeconvolvedsimultaneouslywith
64selectedandstoredeigenvectortemplates(each
of 64x64pixel size) and provide the 64 electronic
outputs(oneinnerproductpertemplate)ataspeedof
about4MHz,reducingtheinputbandwidthof4096
tojust64outputs.
CLICconsistsofasetof8bitshiftregisters
and a 64x64 array of 8bit static random access
memory (SRAM) cells attached to respective
multiplyingdigitaltoanalogconverters(MDAC)[8].
The inputsare thusconverted byMDACsto4096
analog voltage signals, and fed inparallel intothe
NPMcubeevery250nanoseconds.

Within each of the 3D stacked neural


processing module (NPM) chip circuits, the chief
componentofeachcellofthe64x64arrayisahigh
speedmultiplyaccumulate(synapse)circuit[9].The
circuitry is designed to operate at a low power
consumptionofonly3to5watts,or<1wattper
cube. The templates are provided by the digitally
stored 8bit weights in the synapse SRAM circuit.
Eachtemplateisstoredcolumnwiseonall64chips,
firstcolumnofthe64chipsprovidingthefirststored
template,etc.[10].
NEURAL ALGORITHMS, SIMULATION, AND
ANALYSIS: TheNPMnetworkproduces64inner
products (one 4096element input with 64, 4096

element template vectors). A large image is fed,


withtheconceptofrastering,byinputtinganew64
bytecolumnorrowofa64x64subwindowtoCLIC
every250ns,therebyaccomplishing64convolutions
ofa256x256imagewith64x64masksinjust16ms.
The 64 analog values generated by NPM
every 250 ns are given to a neural network. The
neural network performs the desired target
recognition/trackingfunctionsontheNPMoutputas
showninFigure5.Commandandcontrolofvarious
operations (e.g., detection/classification/tracking
mode command, loading of templates, data
recording, etc.) are controlled through a host
machine.

Use Word 6.0c or later to

view Macintosh picture.

Use Word 6.0c or later to

view Macintosh picture.

FIGURE4.THE3DCONVOLVERNETWORKCONSISTSOFANEURALPROCESSINGMODULE(NPM
CUBE) WITH 3D STACKED 64 CHIPS, EACH WITH A 64X64 SYNAPSE ARRAY BASED ON 8BIT
MULTIPLYING DIGITALTOANALOGTECHNOLOGY ANDINCORPORATINGASPECIALPURPOSE
IMAGEWRITEDEVICETERMEDCOLUMNLOADINGINPUTCHIP(CLIC). CLICISBUMPBONDED
TOTHENPMCUBE.THECUBEHASBEENREALIZED(PHOTOGRAPH)ASA10gm,3cm 3PACKAGE,
WITHPOWERCONSUMPTIONOF~2.5W.

Neural Network Classifier


T OIs

Helicopter Plane
Missile

OC
OUTPUTS
CLIC

3DANN-M Templates
(Eigenvectors of
target objects)

Projection of CLIC and


3DANN-M templates

FIGURE5.GENERALTARGETRECOGNITIONISACHIEVEDUSINGEIGENVECTORPROJECTIONSIN
CONJUNCTIONWITHANEURALNETWORKCLASSIFIERTRAINEDONSELECTEDDATASETS.

To achieve efficient target recognition for


objects of arbitrary size and orientation, a
hierarchical approach based on eigenvectors is
employed. Using NPM as the dedicated synapse
weightmultiplierhardware,64eigenvectortemplates
representing the principle axes of a collection of
multidimensionaldatapoints(i.e.,objectimagesof
variousconfigurations)havebeenemployed[1113].
Since each data point is a 4096element vector,
finding a set of 4096 orthonormal eigenvectors is
possible(top64eigenvectorsderivedfromprincipal
component analysis of target imagery reduce the
image dimensionality while retaining much of the
relevantclassificationinformation).
Unless some restrictions are placed on
variationsinthetargetimagery,thetopcomponents
becomequitegeneralandperhapsunsuitableforfine
distinction of a target with all possible scale,
rotation,andperspectivevariations.Ourstrategyis
toparameterize (e.g.,lighting,pose,class,identity,
scale)andpartitiontheobjectspaceinahierarchical
fashion.Thetoplayeristhedetectionlayer,trained
torepresentthepresenceortheabsenceofatarget.
Then, each partition is first used to detect and
segmentallpossibletargets(withsomefalsealarms).
Toclassifyeachpartition,aneuralnetworkistrained
ondataimagerydrawnfromthesetofvariablesthat

definethepartitionandprojectedontoeigenvectors
suitableforthatparticulardistributionofdata.
Informationabouttheobject(class,identity,or
orientation)isprocessedinacoarsetofinemanner.
Forinstance,afterdetectinganobjectinaframe,a
roughestimateofimageorientation/scaleismade,a
resultthatcanthenbeusedtolimitthevariationthat
needs to be considered during subsequent object
classificationstep.Insimulation,objectrecognition
ratesof96%inclutteredbackgroundareobtained

CONCLUSIONS
HONORS,apowerfulATRsystemdescribedherein
combinesanadvancedopticalcorrelatoranda3D
integrated neural network based processor in a
compactpackagetoperformobjectrecognitionwith
unprecedented speed. Algorithms have been
identified and simulated for both the optical
correlator and N3DP. Demonstrations for realtime
detection,classification,andprecisiontracking,with
groundbased and airborne experiments on live
targetsareplanned. Itcanbeprojectedthatsucha
highperformancesystemwillfindvariedusesboth
intheNASAarenaandforcommercialandmilitary
applications.

ACKNOWLEDGMENTS

Theresearchdescribedinthispaperwascarriedout
bytheJetPropulsionLaboratory,CaliforniaInstitute
of Technology, and was jointly sponsored by the
Ballistic Missile Defense Organization and the
National Aeronautics and Space Administration.
TheauthorswishtoexpresstheirsincerethankstoS.
Suddarth,S.Udomkesmalee,T.Duong,C.Padgett,
andG.Reyesforhelpfultechnicaldiscussions.

REFERENCES
1. TH.Chao,T.Daud,andA.Thakoor,Honors:
Advanced Miniature Processing Hardware for
ATR Applications, Proceedings of the 4th
InternationalNEURAPConf.,Marseille,France,
March1113,1998,pp.451456.
2. TH. Chao, E. Hegblom, B. Lau, and W. J.
Miceli,"OptoelectronicallyImplementedNeural
NetworkwithaWaveletPreprocessor,"Proc.of
SPIEvol.2026,1993,SanDiego,CA.,pp.472
482.
3. TH. Chao and A. Yacoubian, Realtime
updatable optical wavelet processor, Proc. of
SPIE,vol.2490,1995,pp.118124,.
4. TH.Chao,A.Yacoubian,andB.Lau,Portable
opticalwavelet processor, Proc.ofSPIE,vol.
2752,1996,pp.254259,.
5. B.V.K. Vijaya Kumar, Tutorial survey of
compositefilterdesignsforopticalcorrelators,
Appl. Opt., vol. 31, no. 23, pp. 47734801,
1992.
6. A.Mahalanobis,B.V.K.VijayaKumar,S.Song,
S.R.F.SimsandJ.F.Epperson,Unconstrained
correlationfilters,Appl.Opt,vol.33,pp.3751
3759,1994.
7. A. Mahalanobis, Correlation filters for object
tracking, target reacquisition, and smart
aimpointselection, Proc.ofSPIE,vol.3073,
1997,pp.2532.
8. T.Duong,T.Thomas,T.Daud,A.Thakoor,and
B. Lee, 64x64 Analog input array for 3
dimensional neural network processor,
Proceedingsofthe3rdInternationalConference
on Neural Networks and Their Applications,
Marseilles,France,March1997.
9. W.S. McCulloch and W. Pitts, A logical
calculus of the ideas immanent in nervous
activity, Bulletin of Mathematical Biophysics,
vol.5,pp.115133,1943
10. T.Duong,S.Kemeny,M.Tran,T.Daud,andA.
Thakoor, Low power analog neurosynapse
chips for a 3D sugarcube neuroprocessor,
Proceedings of the IEEE International

ConferenceonNeuralNetworks,vol.III,1994,
Orlando,FL,pp.19071911.
11. M. Turk and A. Pentland, Eigenfaces for
recognition, J.ofCognitiveNeuroscience,vol.
3,pp.7186,1991.
12. C. Padgett, G. Cottrell, and R. Adolphs,
Categorical perception in facial emotion
classification,Proceedingsofthe18thAnnual
Conference of the Cognitive Science Society,
Hilldale,1996,pp.201207,.
13. C.Padgett,M.Zhu,andS.Suddarth,Detection
and object identification using VIGILANTE
processing system, Proc. SPIE, vol. 3077,
1997.

You might also like