A geometric interpretation of the covariance matrixWord下载.docx

A geometric interpretation of the covariance matrixWord下载.docx

《A geometric interpretation of the covariance matrixWord下载.docx》由会员分享，可在线阅读，更多相关《A geometric interpretation of the covariance matrixWord下载.docx（14页珍藏版）》请在冰豆网上搜索。

Gaussiandensityfunction.Fornormallydistributeddata,68%ofthesamplesfallwithintheintervaldefinedbythemeanplusandminusthestandarddeviation.

Weshowedthatanunbiasedestimatorofthesamplevariancecanbeobtainedby:

（1） 

However,variancecanonlybeusedtoexplainthespreadofthedatainthedirectionsparalleltotheaxesofthefeaturespace.Considerthe2Dfeaturespaceshownbyfigure2:

Figure2. 

Thediagnoalspreadofthedataiscapturedbythecovariance.

Forthisdata,wecouldcalculatethevariance 

inthex-directionandthevariance 

inthey-direction.However,thehorizontalspreadandtheverticalspreadofthedatadoesnotexplainthecleardiagonalcorrelation.Figure2clearlyshowsthatonaverage,ifthex-valueofadatapointincreases,thenalsothey-valueincreases,resultinginapositivecorrelation.Thiscorrelationcanbecapturedbyextendingthenotionofvariancetowhatiscalledthe‘covariance’ofthedata:

（2） 

For2Ddata,wethusobtain 

and 

.Thesefourvaluescanbesummarizedinamatrix,calledthecovariancematrix:

（3） 

Ifxispositivelycorrelatedwithy,yisalsopositivelycorrelatedwithx.Inotherwords,wecanstatethat 

.Therefore,thecovariancematrixisalwaysasymmetricmatrixwiththevariancesonitsdiagonalandthecovariancesoff-diagonal.Two-dimensionalnormallydistributeddataisexplainedcompletelybyitsmeanandits 

covariancematrix.Similarly,a 

covariancematrixisusedtocapturethespreadofthree-dimensionaldata,anda 

covariancematrixcapturesthespreadofN-dimensionaldata.

Figure3illustrateshowtheoverallshapeofthedatadefinesthecovariancematrix:

Figure3. 

Thecovariancematrixdefinestheshapeofthedata.Diagonalspreadiscapturedbythecovariance,whileaxis-alignedspreadiscapturedbythevariance.

Inthenextsection,wewilldiscusshowthecovariancematrixcanbeinterpretedasalinearoperatorthattransformswhitedataintothedataweobserved.However,beforedivingintothetechnicaldetails,itisimportanttogainanintuitiveunderstandingofhoweigenvectorsandeigenvaluesuniquelydefinethecovariancematrix,andthereforetheshapeofourdata.

Aswesawinfigure3,thecovariancematrixdefinesboththespread（variance）,andtheorientation（covariance）ofourdata.So,ifwewouldliketorepresentthecovariancematrixwithavectoranditsmagnitude,weshouldsimplytrytofindthevectorthatpointsintothedirectionofthelargestspreadofthedata,andwhosemagnitudeequalsthespread（variance）inthisdirection.

Ifwedefinethisvectoras 

thentheprojectionofourdata 

ontothisvectorisobtainedas 

andthevarianceoftheprojecteddatais 

.Sincewearelookingforthevector 

thatpointsintothedirectionofthelargestvariance,weshouldchooseitscomponentssuchthatthecovariancematrix 

oftheprojecteddataisaslargeaspossible.Maximizinganyfunctionoftheform 

withrespectto 

where 

isanormalizedunitvector,canbeformulatedasasocalled 

RayleighQuotient.ThemaximumofsuchaRayleighQuotientisobtainedbysetting 

equaltothelargesteigenvectorofmatrix 

.

Inotherwords,thelargesteigenvectorofthecovariancematrixalwayspointsintothedirectionofthelargestvarianceofthedata,andthemagnitudeofthisvectorequalsthecorrespondingeigenvalue.Thesecondlargesteigenvectorisalwaysorthogonaltothelargesteigenvector,andpointsintothedirectionofthesecondlargestspreadofthedata.

Nowlet’shavealookatsomeexamples.Inanearlierarticlewesawthatalineartransformationmatrix 

iscompletelydefinedbyits 

eigenvectorsandeigenvalues.Appliedtothecovariancematrix,thismeansthat:

（4） 

where 

isaneigenvectorof 

and 

isthecorrespondingeigenvalue.

Ifthecovariancematrixofourdataisadiagonalmatrix,suchthatthecovariancesarezero,thenthismeansthatthevariancesmustbeequaltotheeigenvalues 

.Thisisillustratedbyfigure4,wheretheeigenvectorsareshowningreenandmagenta,andwheretheeigenvaluesclearlyequalthevariancecomponentsofthecovariancematrix.

Figure4. 

Eigenvectorsofacovariancematrix

However,ifthecovariancematrixisnotdiagonal,suchthatthecovariancesarenotzero,thenthesituationisalittlemorecomplicated.Theeigenvaluesstillrepresentthevariancemagnitudeinthedirectionofthelargestspreadofthedata,andthevariancecomponentsofthecovariancematrixstillrepresentthevariancemagnitudeinthedirectionofthex-axisandy-axis.Butsincethedataisnotaxisaligned,thesevaluesarenotthesameanymoreasshownbyfigure5.

Figure5. 

Eigenvaluesversusvariance

Bycomparingfigure5withfigure4,itbecomesclearthattheeigenvaluesrepresentthevarianceofthedataalongtheeigenvectordirections,whereasthevariancecomponentsofthecovariancematrixrepresentthespreadalongtheaxes.Iftherearenocovariances,thenbothvaluesareequal.

Nowlet’sforgetaboutcovariancematricesforamoment.Eachoftheexamplesinfigure3cansimplybeconsideredtobealinearlytransformedinstanceoffigure6:

Figure6. 

Datawithunitcovariancematrixiscalledwhitedata.

Letthedatashownbyfigure6be 

theneachoftheexamplesshownbyfigure3canbeobtainedbylinearlytransforming 

:

（5） 

isatransformationmatrixconsistingofarotationmatrix 

andascalingmatrix 

（6） 

Thesematricesaredefinedas:

（7） 

istherotationangle,and:

（8） 

arethescalingfactorsinthexdirectionandtheydirectionrespectively.

Inthefollowingparagraphs,wewilldiscusstherelationbetweenthecovariancematrix 

andthelineartransformationmatrix 

Let’sstartwithunscaled（scaleequals1）andunrotateddata.Instatisticsthisisoftenreferedtoas‘whitedata’becauseitssamplesaredrawnfromastandardnormaldistributionandthereforecorrespondtowhite（uncorrelated）noise:

Figure7. 

Whitedataisdatawithaunitcovariancematrix.

Thecovariancematrixofthis‘white’dataequalstheidentitymatrix,suchthatthevariancesandstandarddeviationsequal1andthecovarianceequalszero:

（9） 

Nowlet’sscalethedatainthex-directionwithafactor4:

（10） 

Thedata 

nowlooksasfollows:

Figure8. 

Varianceinthex-directionresultsinahorizontalscaling.

Thecovariancematrix 

of 

isnow:

（11） 

Thus,thecovariancematrix 

oftheresultingdata 

isrelatedtothelineartransformation 

thatisappliedtotheoriginaldataasfollows:

where

（12） 

However,althoughequation（12）holdswhenthedataisscaledinthexandydirection,thequestionrisesifitalsoholdswhenarotationisapplied.Toinvestigatetherelationbetweenthelineartransformationmatrix 

andthecovariancematrix 

inthegeneralcase,wewillthereforetrytodecomposethecovariancematrixintotheproductofrotationandscalingmatrices.

Aswesawearlier,wecanrepresentthecovariancematrixbyitseigenvectorsandeigenvalues:

（13） 

Equation（13）holdsforeacheigenvector-eigenvaluepairofmatrix 

.Inthe2Dcase,weobtaintwoeigenvectorsandtwoeigenvalues.Thesystemoftwoequationsdefinedbyequation（13）canberepresentedefficientlyusingmatrixnotation:

（14） 

isthematrixwhosecolumnsaretheeigenvectorsof 

isthediagonalmatrixwhosenon-zeroelementsarethecorrespondingeigenvalues.

Thismeansthatwecanrepresentthecovariancematrixasafunctionofitseigenvectorsandeigenvalues:

（15） 

Equation（15）iscalledtheeigendecompositionofthecovariancematrixandcanbeobtainedusinga 

SingularValueDecompositionalgorithm.Whereastheeigenvectorsrepresentthedirectionsofthelargestvarianceofthedata,theeigenvaluesrepresentthemagnitudeofthisvarianceinthosedirections.Inotherwords, 

representsarotationmatrix,while 

representsascalingmatrix.Thecovariancematrixcanthusbedecomposedfurtheras:

（16） 

isarotationmatrixand 

isascalingmatrix.

Inequation（6）wedefinedalineartransformation 

.Since 

isadiagonalscalingmatrix, 

.Furthermore,since 

isanorthogonalmatrix, 

.Therefore, 

.Thecovariancematrixcanthusbewrittenas:

（17） 

Inotherwords,ifweapplythelineartransformationdefinedby 

totheoriginalwhitedata 

shownbyfigure7,weobtaintherotatedandscaleddata 

withcovariancematrix 

.Thisisillustratedbyfigure10:

Figure10. 

Thecovariancematrixrepresentsalineartransformationof