14/4/2016

LevenbergMarquardtbackpropagationMATLABtrainlm

Definitions
LikethequasiNewtonmethods,theLevenbergMarquardtalgorithmwasdesignedtoapproachsecondordertraining
speedwithouthavingtocomputetheHessianmatrix.Whentheperformancefunctionhastheformofasumof
squares(asistypicalintrainingfeedforwardnetworks),thentheHessianmatrixcanbeapproximatedas
H=JTJ
andthegradientcanbecomputedas
g=JTe
whereJistheJacobianmatrixthatcontainsfirstderivativesofthenetworkerrorswithrespecttotheweightsand
biases,andeisavectorofnetworkerrors.TheJacobianmatrixcanbecomputedthroughastandard
backpropagationtechnique(see[HaMe94])thatismuchlesscomplexthancomputingtheHessianmatrix.
TheLevenbergMarquardtalgorithmusesthisapproximationtotheHessianmatrixinthefollowingNewtonlike
update:
T

1 T

xk +1 = xk [ J J + I] J e
Whenthescalariszero,thisisjustNewton'smethod,usingtheapproximateHessianmatrix.Whenislarge,this
becomesgradientdescentwithasmallstepsize.Newton'smethodisfasterandmoreaccuratenearanerror
minimum,sotheaimistoshifttowardNewton'smethodasquicklyaspossible.Thus,isdecreasedaftereach
successfulstep(reductioninperformancefunction)andisincreasedonlywhenatentativestepwouldincreasethe
performancefunction.Inthisway,theperformancefunctionisalwaysreducedateachiterationofthealgorithm.
TheoriginaldescriptionoftheLevenbergMarquardtalgorithmisgivenin[Marq63].TheapplicationofLevenberg
Marquardttoneuralnetworktrainingisdescribedin[HaMe94]andstartingonpage1219of[HDB96].Thisalgorithm
appearstobethefastestmethodfortrainingmoderatesizedfeedforwardneuralnetworks(uptoseveralhundred
weights).ItalsohasanefficientimplementationinMATLABsoftware,becausethesolutionofthematrixequation
isabuiltinfunction,soitsattributesbecomeevenmorepronouncedinaMATLABenvironment.
TrytheNeuralNetworkDesigndemonstrationnnd12m[HDB96]foranillustrationoftheperformanceofthebatch
LevenbergMarquardtalgorithm.

Limitations
ThisfunctionusestheJacobianforcalculations,whichassumesthatperformanceisameanorsumofsquared
errors.Therefore,networkstrainedwiththisfunctionmustuseeitherthemseorsseperformancefunction.

MoreAbout

collapseall

Algorithms
trainlmsupportstrainingwithvalidationandtestvectorsifthenetwork'sNET.divideFcnpropertyissettoa
datadivisionfunction.Validationvectorsareusedtostoptrainingearlyifthenetworkperformanceonthe
validationvectorsfailstoimproveorremainsthesameformax_failepochsinarow.Testvectorsareusedasa
furthercheckthatthenetworkisgeneralizingwell,butdonothaveanyeffectontraining.
trainlmcantrainanynetworkaslongasitsweight,netinput,andtransferfunctionshavederivativefunctions.
BackpropagationisusedtocalculatetheJacobianjXofperformanceperfwithrespecttotheweightandbias
variablesX.EachvariableisadjustedaccordingtoLevenbergMarquardt,
jj=jX*jX
http://www.mathworks.com/help/nnet/ref/trainlm.html

1/2

14/4/2016

LevenbergMarquardtbackpropagationMATLABtrainlm

je=jX*E
dX=(jj+I*mu)\je
whereEisallerrorsandIistheidentitymatrix.
Theadaptivevaluemuisincreasedbymu_incuntilthechangeaboveresultsinareducedperformancevalue.
Thechangeisthenmadetothenetworkandmuisdecreasedbymu_dec.
Theparametermem_reducindicateshowtousememoryandspeedtocalculatetheJacobianjX.Ifmem_reducis
1,thentrainlmrunsthefastest,butcanrequirealotofmemory.Increasingmem_reducto2cutssomeofthe
memoryrequiredbyafactoroftwo,butslowstrainlmsomewhat.Higherstatescontinuetodecreasethe
amountofmemoryneededandincreasetrainingtimes.
Trainingstopswhenanyoftheseconditionsoccurs:
Themaximumnumberofepochs(repetitions)isreached.
Themaximumamountoftimeisexceeded.
Performanceisminimizedtothegoal.
Theperformancegradientfallsbelowmin_grad.
muexceedsmu_max.
Validationperformancehasincreasedmorethanmax_failtimessincethelasttimeitdecreased(whenusing
validation).

http://www.mathworks.com/help/nnet/ref/trainlm.html

2/2