System Identification with BP Neural Network.docx

System Identification with BP Neural Network.docx

《System Identification with BP Neural Network.docx》由会员分享，可在线阅读，更多相关《System Identification with BP Neural Network.docx（16页珍藏版）》请在冰豆网上搜索。

SystemIdentificationwithBPNeuralNetwork

SystemIdentificationwithBPNeuralNetwork

Abstract：

ThispaperfirstlyintroducedMLP（themultilayerperceptron）andBackPropagationalgorithm,andpresentamethodofusingaBP（Back-Propagation）NeuralNetworktorealizesystemidentification.Thenthepapershowedhowitwasappliedtothreesystemsofdifferentsystemfunctions,andanalyzedtheaffectsofdifferentparametersoftheBPneuralnetwork.

Keywords:

MLP,Neurons,HiddenLayer,BPNeuralNetwork.

1IntroductionofMLP

ArtificialNeuralNetworks（ANNs）,orsimplyneuralnetworks,areabranchofartificialintelligence.ANNsareprogrammingparadigmthatseektoemulatethemicrostructureofthebrain,andareusedextensivelyinartificialintelligenceproblemsfromsimplepattern-recognitiontasks,toadvancedsymbolicmanipulation.

TheMultilayerPerceptron（MLP）isanexampleofartificialneuralnetworks,whichisusedextensivelyforthesolutionofanumberofdifferentproblems,includingpatternrecognitionandinterpolation.ItisadevelopmentofthePerceptronneuralnetworkmodel,thatwasoriginallydevelopedintheearly1960sbutfoundtohaveseriouslimitations.

ArtificialNeuralNetworksattempttomodelthefunctioningofthehumanbrain.Thehumanbrainforexampleconsistsofbillionsofindividualcellscalledneurons.Itisbelievedthatallknowledgeandexperienceisencodedbytheconnectionsthatexistbetweenneurons.Giventhatthehumanbrainconsistsofsuchalargenumberofneurons（toomanytocountthemwithanycertainty）,thequantityandnatureoftheconnectionsbetweenneuronsis,atpresentlevelsofunderstanding,almostimpossibletoassess.

Multilayerperceptronsformonetypeofneuralnetworkasillustratedinthetaxonomyin Fig.0.1. 

Fig.0.1Ataxonomyofneuralnetworkarchitectures

Themultilayerperceptronconsistsofasystemofsimpleinterconnectedneurons,ornodes,asillustratedin Fig.0.2,whichisamodelrepresentingnonlinearmappingbetweenaninputvectorandanoutputvector.

Fig.0.2Amultilayerperceptronwithtwohiddenlayers

Inthefigure0.2,

=[

]=outputvector.Thenodesareconnectedbyweightsandoutputsignalswhichareafunctionofthesumoftheinputstothenodemodifiedbyasimplenonlineartransfer,oractivationfunction.Itisthesuperpositionofmanysimplenonlineartransferfunctionsthatenablesthemultilayerperceptrontoapproximateextremelynon-linearfunctions.Ifthetransferfunctionwaslinearthenthemultilayerperceptronwouldonlybeabletomodellinearfunctions.Duetoitseasilycomputedderivativeacommonlyusedtransferfunctionisthelogisticfunctionasfollows,

asshownin Fig.0.3.Theoutputofanodeisscaledbytheconnectingweightandfedforwardtobeaninputtothenodesinthenextlayerofthenetwork.Thisimpliesadirectionofinformationprocessing,sothemultilayerperceptronisknownasafeed-forwardneuralnetwork.Thearchitectureofamultilayerperceptronisvariablebutingeneralwillconsistofseverallayersofneurons.Theinputlayerplaysnocomputationalrolebutmerelyservestopasstheinputvectortothenetwork.Thetermsinputandoutputvectorsrefertotheinputsandoutputsofthemultilayerperceptronandcanberepresentedassinglevectors,asshownin Fig.0.2.Amultilayerperceptronmayhaveoneormorehiddenlayersandfinallyanoutputlayer.Multilayerperceptronsaredescribedasbeingfullyconnected,witheachnodeconnectedtoeverynodeinthenextandpreviouslayer.

Fig.0.3Thelogisticfunctiony

Byselectingasuitablesetofconnectingweightsandtransferfunctions,ithasbeenshownthatamultilayerperceptroncanapproximateanysmooth,measurablefunctionbetweentheinputandoutputvectors.Multilayerperceptronshavetheabilitytolearnthroughtraining.Trainingrequiresasetoftrainingdata,whichconsistsofaseriesofinputandassociatedoutputvectors.Duringtrainingthemultilayerperceptronisrepeatedlypresentedwiththetrainingdataandtheweightsinthenetworkareadjusteduntilthedesiredinput–outputmappingoccurs.Multilayerperceptronslearninasupervisedmanner.Duringtrainingtheoutputfromthemultilayerperceptron,foragiveninputvector,maynotequalthedesiredoutput.Anerrorsignalisdefinedasthedifferencebetweenthedesiredandactualoutput.Trainingusesthemagnitudeofthiserrorsignaltodeterminetowhatdegreetheweightsinthenetworkshouldbeadjustedsothattheoverallerrorofthemultilayerperceptronisreduced.Therearemanyalgorithmsthatcanbeusedtotrainamultilayerperceptron.Oncetrainedwithsuitablyrepresentativetrainingdatathemultilayerperceptroncangeneralizetonew,unseeninputdata.

Themultilayerperceptronhasbeenappliedtoawidevarietyoftasks,allofwhichcanbecategorizedasprediction,functionapproximation,orpatternclassification.Predictioninvolvestheforecastingoffuturetrendsinatimeseriesofdatagivencurrentandpreviousconditions.Functionapproximationisconcernedwithmodelingtherelationshipbetweenvariables.Patternclassificationinvolvesclassifyingdataintodiscreteclasses.

2BackPropagationalgorithm

Trainingamultilayerperceptronistheprocedurebywhichthevaluesfortheindividualweightsaredeterminedsuchthattherelationshipthenetworkismodelingisaccuratelyresolved.Atthispointwewillconsiderasimplemultilayerperceptronthatcontainsonlytwoweights.Foranycombinationofweightsthenetworkerrorforagivenpatterncanbedefined.Byvaryingtheweightsthroughallpossiblevalues,andbyplottingerrorsinthree-dimensionalspace,weendupwithaplotliketheoneshownin Fig.1.1.Suchasurfaceisknownasanerrorsurface.Theobjectiveoftrainingistofindthecombinationofweightswhichresultinthesmallesterror.Inpractice,itisnotpossibletoplotsuchasurfaceduetothemultitudeofweights.Whatisrequiredisamethodtofindtheminimumpointoftheerrorsurface.

Fig.1.1Anerrorsurfaceforasimplemultilayerperceptroncontainingonlytwoweights.

Onepossibletechniqueistouseaprocedureknownasgradientdescent.Thebackpropagationtrainingalgorithmusesthisproceduretoattempttolocatetheabsolute（orglobal）minimumoftheerrorsurface.Thebackpropagationalgorithmisthemostcomputationallystraightforwardalgorithmfortrainingthemultilayerperceptron.Backpropagationhasbeenshowntoperformadequatelyinmanyapplications;themajorityoftheapplicationsdiscussedinthispaperusedbackpropagationtotrainthemultilayerperceptrons.Backpropagationonlyreferstothetrainingalgorithmandisnotanothertermforthemultilayerperceptronorfeed-forwardneuralnetworks,asiscommonlyreported.

Theweightsinthenetworkareinitiallysettosmallrandomvalues.Thisissynonymouswithselectingarandompointontheerrorsurface.Thebackpropagationalgorithmthencalculatesthelocalgradientoftheerrorsurfaceandchangestheweightsinthedirectionofsteepestlocalgradient.Givenareasonablysmootherrorsurface,itishopedthattheweightswillconvergetotheglobalminimumoftheerrorsurface.

Thebackpropagationalgorithmissummarizedbelow:

Theerrorsurfacein Fig.1.1 containsmorethanoneminimum.Itisdesirablethatthetrainingalgorithmdoesnotbecometrappedinalocalminimum.Thebackpropagationalgorithmcontainstwoadjustableparameters,alearningrateandamomentumterm,whichcanassistthetrainingprocessinavoidingthis.Thelearningratedeterminesthestepsizetakenduringtheiterativegradientdescentlearningprocess.Ifthisistoolargethenthenetworkerrorwillchangeerraticallyduetolargeweightchanges,withthepossibilityofjumpingovertheglobalminima.Conversely,ifthelearningrateistoosmallthentrainingwilltakealongtime.Themomentumtermisusedtoassistthegradientdescentprocessifitbecomesstuckinalocalminimum.Byaddingaproportionofthepreviousweightchangetothecurrentweightchange（whichwillbeverysmallinalocalminimum）itispossiblethattheweightscanescapethelocalminimum.

TheBPalgorithmisdeducedfromthesteepestgradientdescentmethod.Forthe

sample,wedefinethepowerfunctionas

isthedesiredoutputoftheqthsample;

istherealoutputofthenetwork.

Accordingtothesteepestgradientdescentmethod,wecangettheadjustmentoftheweightofeachconnectionasfollows:

[k+1]=

[k]+

Fortheoutputlayer,

Forthehiddenandinputlayer,

Intheformulasabove,f（s）istheactivationfunction,and

isthederivativeoff（s）,andsisequaltothedifferencebetweentheweightedsumoftheinputsandthethresholdofeachneuron.

isthelearningrate.

Turntothethresholdofeachneuron,wecanconcludethesimilarformulaasfollows:

Whenthenetworkhasbeentrainedwithallthesamplesforonetime,thealgorithmwouldfinishoneepoch.Thencalculatetheperformanceindex

.Iftheindexfittheaccuracyrequirements,thenendthetraining,elsestartanothertrainingepoch.

3Exercise

（1）

9trainingsamplesuniformlydistributedintheregionof

and361testsamples.

IchooseaMLPmodelwith1hiddenlayer,andweapplieddifferentnumberofneuronsofthehiddenlayertostudytheeffectsofthenumberofneurons.

Ichose9setsofuniformdatatobeusedtotrainthenetwork,andthentestedthenetworkwith361setsofuniformdata.ChooseMatlabasthesimulatingtool.PerformanceindexissetasE<0.01.

Duetotheexistenceofzerosinthedesiredoutput,therelativeerrorwillbehugeintheareanearbythezeros,andthatwillmaketherelativeerroru