reliablility and vadility.docx

reliablility and vadility.docx

《reliablility and vadility.docx》由会员分享，可在线阅读，更多相关《reliablility and vadility.docx（15页珍藏版）》请在冰豆网上搜索。

reliablilityandvadility

Reliability&Validity

Weoftenthinkofreliabilityandvalidityasseparateideasbut,infact,they'rerelatedtoeachother.Here,Iwanttoshowyoutwowaysyoucanthinkabouttheirrelationship.

Oneofmyfavoritemetaphorsfortherelationshipbetweenreliabilityisthatofthetarget.Thinkofthecenterofthetargetastheconceptthatyouaretryingtomeasure.Imaginethatforeachpersonyouaremeasuring,youaretakingashotatthetarget.Ifyoumeasuretheconceptperfectlyforaperson,youarehittingthecenterofthetarget.Ifyoudon't,youaremissingthecenter.Themoreyouareoffforthatperson,thefurtheryouarefromthecenter.

Thefigureaboveshowsfourpossiblesituations.Inthefirstone,youarehittingthetargetconsistently,butyouaremissingthecenterofthetarget.Thatis,youareconsistentlyandsystematicallymeasuringthewrongvalueforallrespondents.Thismeasureisreliable,butnovalid（thatis,it'sconsistentbutwrong）.Thesecond,showshitsthatarerandomlyspreadacrossthetarget.Youseldomhitthecenterofthetargetbut,onaverage,youaregettingtherightanswerforthegroup（butnotverywellforindividuals）.Inthiscase,yougetavalidgroupestimate,butyouareinconsistent.Here,youcanclearlyseethatreliabilityisdirectlyrelatedtothevariabilityofyourmeasure.Thethirdscenarioshowsacasewhereyourhitsarespreadacrossthetargetandyouareconsistentlymissingthecenter.Yourmeasureinthiscaseisneitherreliablenorvalid.Finally,weseethe"RobinHood"scenario--youconsistentlyhitthecenterofthetarget.Yourmeasureisbothreliableandvalid（IbetyouneverthoughtofRobinHoodinthosetermsbefore）.

Anotherwaywecanthinkabouttherelationshipbetweenreliabilityandvalidityisshowninthefigurebelow.Here,wesetupa2x2table.Thecolumnsofthetableindicatewhetheryouaretryingtomeasurethesameordifferentconcepts.Therowsshowwhetheryouareusingthesameordifferentmethodsofmeasurement.Imaginethatwehavetwoconceptswewouldliketomeasure,studentverbalandmathability.Furthermore,imaginethatwecanmeasureeachoftheseintwoways.First,wecanuseawritten,paper-and-pencilexam（verymuchliketheSATorGREexams）.Second,wecanaskthestudent'sclassroomteachertogiveusaratingofthestudent'sabilitybasedontheirownclassroomobservation.

Thefirstcellontheupperleftshowsthecomparisonoftheverbalwrittentestscorewiththeverbalwrittentestscore.Buthowcanwecomparethesamemeasurewithitself?

Wecoulddothisbyestimatingthereliabilityofthewrittentestthroughatest-retestcorrelation,parallelforms,oraninternalconsistencymeasure（See TypesofReliability）.Whatweareestimatinginthiscellisthereliabilityofthemeasure.

Thecellonthelowerleftshowsacomparisonoftheverbalwrittenmeasurewiththeverbalteacherobservationrating.Becausewearetryingtomeasurethesameconcept,wearelookingatconvergentvalidity（See MeasurementValidityTypes）.

Thecellontheupperrightshowsthecomparisonoftheverbalwrittenexamwiththemathwrittenexam.Here,wearecomparingtwodifferentconcepts（verbalversusmath）andsowewouldexpecttherelationshiptobelowerthanacomparisonofthesameconceptwithitself（e.g.,verbalversusverbalormathversusmath）.Thus,wearetryingtodiscriminatebetweentwoconceptsandwewouldconsiderthisdiscriminantvalidity.

Finally,wehavethecellonthelowerright.Here,wearecomparingtheverbalwrittenexamwiththemathteacherobservationrating.Likethecellontheupperright,wearealsotryingtocomparetwodifferentconcepts（verbalversusmath）andsothisisadiscriminantvalidityestimate.Buthere,wearealsotryingtocomparetwodifferentmethodsofmeasurement（writtenexamversusteacherobservationrating）.So,we'llcallthis very discriminanttoindicatethatwewouldexpecttherelationshipinthiscelltobeevenlowerthanintheoneaboveit.

Thefourcellsincorporatethedifferentvaluesthatweexamineinthe multitrait-multimethod approachtoestimatingconstructvalidity.

Whenwelookatreliabilityandvalidityinthisway,weseethat,ratherthanbeingdistinct,theyactuallyformacontinuum.Ononeendisthesituationwheretheconceptsandmethodsofmeasurementarethesame（reliability）andontheotheristhesituationwhereconceptsandmethodsofmeasurementaredifferent（verydiscriminantvalidity）.

IntroductiontoValidity

Validity:

thebestavailableapproximationtothetruthofagivenproposition,inference,orconclusion

Thefirstthingwehavetoaskis:

"validityof what?

"Whenwethinkaboutvalidityinresearch,mostofusthinkaboutresearchcomponents.Wemightsaythatameasureisavalidone,orthatavalidsamplewasdrawn,orthatthedesignhadstrongvalidity.Butallofthosestatementsaretechnicallyincorrect.Measures,samplesanddesignsdon't'have'validity--onlypropositionscanbesaidtobevalid.Technically,weshouldsaythatameasureleadstovalidconclusionsorthatasampleenablesvalidinferences,andsoon.Itisaproposition,inferenceorconclusionthatcan'have'validity.

Wemakelotsofdifferentinferencesorconclusionswhileconductingresearch.Manyofthesearerelatedtotheprocessofdoingresearchandarenotthemajorhypothesesofthestudy.Nevertheless,likethebricksthatgointobuildingawall,theseintermediateprocessandmethodologicalpropositionsprovidethefoundationforthesubstantiveconclusionsthatwewishtoaddress.Forinstance,virtuallyallsocialresearchinvolvesmeasurementorobservation.And,wheneverwemeasureorobserveweareconcernedwithwhetherwearemeasuringwhatweintendtomeasureorwithhowourobservationsareinfluencedbythecircumstancesinwhichtheyaremade.Wereachconclusionsaboutthequalityofourmeasures--conclusionsthatwillplayanimportantroleinaddressingthebroadersubstantiveissuesofourstudy.Whenwetalkaboutthevalidityofresearch,weareoftenreferringtothesetothemanyconclusionswereachaboutthequalityofdifferentpartsofourresearchmethodology.

Wesubdividevalidityintofourtypes.Eachtypeaddressesaspecificmethodologicalquestion.Inordertounderstandthetypesofvalidity,youhavetoknowsomethingabouthowweinvestigatearesearchquestion.Becauseallfourvaliditytypesarereallyonlyoperativewhenstudyingcausalquestions,wewilluseacausalstudytosetthecontext.

Thefigureshowsthattherearereallytworealmsthatareinvolvedinresearch.Thefirst,onthetop,isthelandoftheory.Itiswhatgoesoninsideourheadsasresearchers.Itiswherewekeepourtheoriesabouthowtheworldoperates.Thesecond,onthebottom,isthelandofobservations.Itistherealworldintowhichwetranslateourideas--ourprograms,treatments,measuresandobservations.Whenweconductresearch,wearecontinuallyflittingbackandforthbetweenthesetworealms,betweenwhatwethinkabouttheworldandwhatisgoingoninit.Whenweareinvestigatingacause-effectrelationship,wehaveatheory（implicitorotherwise）ofwhatthecauseis（thecauseconstruct）.Forinstance,ifwearetestinganeweducationalprogram,wehaveanideaofwhatitwouldlooklikeideally.Similarly,ontheeffectside,wehaveanideaofwhatweareideallytryingtoaffectandmeasure（theeffectconstruct）.Buteachofthese,thecauseandtheeffect,hastobetranslatedintorealthings,intoaprogramortreatmentandameasureorobservationalmethod.Weusetheterm operationalization todescribetheactoftranslatingaconstructintoitsmanifestation.Ineffect,wetakeourideaanddescribeitasaseriesofoperationsorprocedures.Now,insteadofitonlybeinganideainourminds,itbecomesapublicentitythatanyonecanlookatandexamineforthemselves.Itisonething,forinstance,foryoutosaythatyouwouldliketomeasureself-esteem（aconstruct）.Butwhenyoushowaten-itempaper-and-pencilself-esteemmeasurethatyoudevelopedforthatpurpose,otherscanlookatitandunderstandmoreclearlywhatyouintendbythetermself-esteem.

Now,backtoexplainingthefourvaliditytypes.Theybuildononeanother,withtwoofthem（conclusion andinternal）referringtothelandofobservationonthebottomofthefigure,oneofthem（construct）emphasizingthelinkagesbetweenthebottomandthetop,andthelast（external）beingprimarilyconcernedabouttherangeofourtheoryonthetop.ImaginethatwewishtoexaminewhetheruseofaWorldWideWeb（WWW）VirtualClassroomimprovesstudentunderstandingofcoursematerial.Assumethatwetookthesetwoconstructs,thecauseconstruct（theWWWsite）andtheeffect（understanding）,andoperationalizedthem--turnedthemintorealitiesbyconstructingtheWWWsiteandameasureofknowledgeofthecoursematerial.Herearethefourvaliditytypesandthequestioneachaddresses:

ConclusionValidity:

Inthisstudy,istherea relationshipbetweenthetwovariables?

Inthecontextoftheexamplewe'reconsidering,thequestionmightbeworded:

inthisstudy,istherearelationshipbetweentheWWWsiteandknowledgeofcoursematerial?

Thereareseveralconclusionsorinferenceswemightdrawtoanswersuchaquestion.Wecould,forexample,concludethatthereisarelationship.Wemightconcludethatthereisapositiverelationship.Wemightinferthatthereisnorelationship.Wecanassesstheconclusionvalidityofeachoftheseconclusionsorinferences.

InternalValidity:

 Assumingthatthereisarelationshipinthisstudy, istherelationshipa causal one?

Justbecausewefindthatus