您的当前位置：首页 Frequentist statistics as a theory of inductive inference

Frequentist statistics as a theory of inductive inference

来源：智榕旅游

6002 tcO 72 ]TS.tham[ 1v6480160/tham:viXraIMSLectureNotes–MonographSeries2ndLehmannSymposium–OptimalityVol.49(2006)77–97

󰀁

cInstituteofMathematicalStatistics,2006DOI:10.1214/074921706000000400

Frequentiststatisticsasatheoryof

inductiveinference

DeborahG.Mayo1andD.R.Cox2

ViriginiaTechandNuﬃeldCollege,Oxford

Abstract:Aftersomegeneralremarksabouttheinterrelationbetweenphilo-sophicalandstatisticalthinking,thediscussioncentreslargelyonsigniﬁcancetests.Thesearedeﬁnedasthecalculationofp-valuesratherthanasformalproceduresfor“acceptance”and“rejection.”Anumberoftypesofnullhypoth-esisaredescribedandaprincipleforevidentialinterpretationsetoutgoverningtheimplicationsofp-valuesinthespeciﬁccircumstancesofeachapplication,ascontrastedwithalong-runinterpretation.Avarietyofmorecomplicatedsituationsarediscussedinwhichmodiﬁcationofthesimplep-valuemaybeessential.

1.Statisticsandinductivephilosophy1.1.WhatisthePhilosophyofStatistics?

Thephilosophicalfoundationsofstatisticsmayberegardedasthestudyoftheepistemological,conceptualandlogicalproblemsrevolvingaroundtheuseandin-terpretationofstatisticalmethods,broadlyconceived.Aswithotherdomainsofphilosophyofscience,workinstatisticalscienceprogresseslargelywithoutwor-ryingabout“philosophicalfoundations”.Nevertheless,eveninstatisticalpractice,debatesaboutthediﬀerentapproachestostatisticalanalysismayinﬂuenceandbeinﬂuencedbygeneralissuesofthenatureofinductive-statisticalinference,andthusareconcernedwithfoundationalorphilosophicalmatters.Eventhosewhoarelargelyconcernedwithapplicationsareofteninterestedinidentifyinggeneralprin-ciplesthatunderlieandjustifytheprocedurestheyhavecometovalueonrelativelypragmaticgrounds.Atonelevelofanalysisatleast,statisticiansandphilosophersofscienceaskmanyofthesamequestions.

•Whatshouldbeobservedandwhatmayjustiﬁablybeinferredfromthere-sultingdata?

•Howwelldodataconﬁrmorﬁtamodel?•Whatisagoodtest?

•DoesfailuretorejectahypothesisHconstituteevidence“conﬁrming”H?•Howcanitbedeterminedwhetheranapparentanomalyisgenuine?Howcanblameforananomalybeassignedcorrectly?

•Isitrelevanttotherelationbetweendataandahypothesisiflookingatthedatainﬂuencesthehypothesistobeexamined?

•Howcanspuriousrelationshipsbedistinguishedfromgenuineregularities?

78D.G.MayoandD.R.Cox

•Howcanacausalexplanationandhypothesisbejustiﬁedandtested?

•Howcanthegapbetweenavailabledataandtheoreticalclaimsbebridgedreliably?

Thattheseverygeneralquestionsareentwinedwithlongstandingdebatesinphilosophyofsciencehelpsexplainwhytheﬁeldofstatisticstendstocrossover,eitherexplicitlyorimplicitly,intophilosophicalterritory.Somemayevenregardstatisticsasakindof“appliedphilosophyofscience”(Fisher[10];Kempthorne[13]),andstatisticaltheoryasakindof“appliedphilosophyofinductiveinfer-ence”.AsLehmann[15]hasemphasized,Neymanregardedhisworknotonlyasacontributiontostatisticsbutalsotoinductivephilosophy.Acorequestionthatpermeates“inductivephilosophy”bothinstatisticsandphilosophyis:Whatisthenatureandroleofprobabilisticconcepts,methods,andmodelsinmakinginferencesinthefaceoflimiteddata,uncertaintyanderror?

Giventheoccasionofourcontribution,asessiononphilosophyofstatisticsforthesecondLehmannsymposium,wetakeasourspringboardtherecommendationofNeyman([22],p.17)thatweviewstatisticaltheoryasessentiallya“FrequentistTheoryofInductiveInference”.Thequestionthenarisesastowhatconception(s)ofinductiveinferencewouldallowthis.Whetherornotthisistheonlyoreventhemostsatisfactoryaccountofinductiveinference,itisinterestingtoexplorehowmuchprogresstowardsanaccountofinductiveinference,asopposedtoinductivebehavior,onemightgetfromfrequentiststatistics(withafocusontestingandassociatedmethods).Thesemethodsare,afterall,oftenusedforinferentialends,tolearnaboutaspectsoftheunderlyingdatageneratingmechanism,andmuchconfusionandcriticism(e.g.,astowhetherandwhyerrorratesaretobeadjusted)couldbeavoidediftherewasgreaterclarityontherolesininferenceofhypotheticalerrorprobabilities.

TakingasabackdropremarksbyFisher[10],Lehmann[15]onNeyman,andbyPopper[26]oninduction,weconsidertherolesofsigniﬁcancetestsinbridginginductivegapsintraditionalhypotheticaldeductiveinference.Ourgoalistoidentifyakeyprincipleofevidencebywhichhypotheticalerrorprobabilitiesmaybeusedforinductiveinferencefromspeciﬁcdata,andtoconsiderhowitmaydirectandjustify(a)diﬀerentusesandinterpretationsofstatisticalsigniﬁcancelevelsintestingavarietyofdiﬀerenttypesofnullhypotheses,and(b)whenandwhy“selectioneﬀects”needtobetakenaccountofindatadependentstatisticaltesting.1.2.Theroleofprobabilityinfrequentistinduction

Thedeﬁningfeatureofaninductiveinferenceisthatthepremises(evidencestate-ments)canbetruewhiletheconclusioninferredmaybefalsewithoutalogicalcon-tradiction:theconclusionis“evidencetranscending”.Probabilitynaturallyarisesincapturingsuchevidencetranscendinginferences,butthereismorethanonewaythiscanoccur.TwodistinctphilosophicaltraditionsforusingprobabilityininferencearesummedupbyPearson([24],p.228):

“Foroneschool,thedegreeofconﬁdenceinaproposition,aquantityvaryingwiththenatureandextentoftheevidence,providesthebasicnotiontowhichthenumericalscaleshouldbeadjusted.”Theotherschoolnotestherelevanceinordinarylifeandinmanybranchesofscienceofaknowledgeoftherelativefrequencyofoccurrenceofaparticularclassofeventsinaseriesofrepetitions,andsuggeststhat“itisthroughitslinkwithrelativefrequencythatprobabilityhasthemostdirectmeaningforthehumanmind”.

Frequentiststatistics:theoryofinductiveinference79

Frequentistinduction,whateveritsform,employsprobabilityinthesecondman-ner.Forinstance,signiﬁcancetestingappealstoprobabilitytocharacterizethepro-portionofcasesinwhichanullhypothesisH0wouldberejectedinahypotheticallong-runofrepeatedsampling,anerrorprobability.Thisdiﬀerenceintheroleofprobabilitycorrespondstoadiﬀerenceintheformofinferencedeemedappropriate:Theformeruseofprobabilitytraditionallyhasbeentiedtotheviewthataproba-bilisticaccountofinductioninvolvesquantifyingadegreeofsupportorconﬁrmationinclaimsorhypotheses.

Somefollowersofthefrequentistapproachagree,preferringtheterm“inductivebehavior”todescribetheroleofprobabilityinfrequentiststatistics.Herethein-ductivereasoner“decidestoinfer”theconclusion,andprobabilityquantiﬁestheassociatedriskoferror.Theideathatoneroleofprobabilityarisesinsciencetocharacterizethe“riskiness”orprobativenessorseverityoftheteststowhichhy-pothesesareputisreminiscentofthephilosophyofKarlPopper[26].Inparticular,Lehmann([16],p.32)hasnotedthetemporalandconceptualsimilarityoftheideasofPopperandNeymanon“ﬁnessing”theissueofinductionbyreplacinginductivereasoningwithaprocessofhypothesistesting.

ItistruethatPopperandNeymanhavebroadlyanalogousapproachesbasedontheideathatwecanspeakofahypothesishavingbeenwell-testedinsomesense,quitedistinctfromitsbeingaccordedadegreeofprobability,belieforconﬁrmation;thisis“ﬁnessinginduction”.Bothalsobroadlysharedtheviewthatinorderfordatato“conﬁrm”or“corroborate”ahypothesisH,thathypothesiswouldhavetohavebeensubjectedtoatestwithhighprobabilityorpowertohaverejecteditiffalse.Butdespitethecloseconnectionoftheideas,thereappearstobenoreferencetoPopperinthewritingsofNeyman(Lehmann[16],p.3)andthereferencesbyPoppertoNeymanarescantandscarcelyrelevant.Moreover,becausePopperdeniedthatanyinductiveclaimswerejustiﬁable,hisphilosophyforcedhimtodenythateventhemethodheespoused(conjectureandrefutations)wasreliable.AlthoughHmightbetrue,PoppermadeitclearthatheregardedcorroborationatmostasareportofthepastperformanceofH:itwarrantednoclaimsaboutitsreliabilityinfutureapplications.Bycontrast,acentralfeatureoffrequentiststatisticsistobeabletoassessandcontroltheprobabilitythatatestwouldhaverejectedahypothesis,iffalse.Theseprobabilitiescomefromformulatingthedatageneratingprocessintermsofastatisticalmodel.

Neymanthroughouthisworkemphasizestheimportanceofaprobabilisticmodelofthesystemunderstudyanddescribesfrequentiststatisticsasmodellingthephenomenonofthestabilityofrelativefrequenciesofresultsofrepeated“trials”,grantingthatthereareotherpossibilitiesconcernedwithmodellingpsychologicalphenomenaconnectedwithintensitiesofbelief,orwithreadinesstobetspeciﬁedsums,etc.citingCarnap[2],deFinetti[8]andSavage[27].InparticularNeymancriticizedtheviewof“frequentist”inferencetakenbyCarnapforoverlookingthekeyroleofthestochasticmodelofthephenomenonstudied.StatisticalworkrelatedtotheinductivephilosophyofCarnap[2]isthatofKeynes[14]and,withamoreimmediateimpactonstatisticalapplications,Jeﬀreys[12].1.3.Inductionandhypothetical-deductiveinference

While“hypothetical-deductiveinference”maybethoughtto“ﬁnesse”induction,infactinductiveinferencesoccurthroughoutempiricaltesting.Statisticaltestingideasmaybeseentoﬁlltheseinductivegaps:Ifthehypothesisweredeterministic

80D.G.MayoandD.R.Cox

wecouldﬁndarelevantfunctionofthedatawhosevalue(i)representstherelevantfeatureundertestand(ii)canbepredictedbythehypothesis.Wecalculatethefunctionandthenseewhetherthedataagreeordisagreewiththeprediction.Ifthedataconﬂictwiththeprediction,theneitherthehypothesisisinerrororsomeauxiliaryorotherbackgroundfactormaybeblamedfortheanomaly(Duhem’sproblem).

Statisticalconsiderationsenterintwoways.IfHisastatisticalhypothesis,thenusuallynooutcomestrictlycontradictsit.TherearemajorproblemsinvolvedinregardingdataasinconsistentwithHmerelybecausetheyarehighlyimprobable;allindividualoutcomesdescribedindetailmayhaveverysmallprobabilities.Rathertheissue,essentiallyfollowingPopper([26],pp.86,203),iswhetherthepossiblyanomalousoutcomerepresentssomesystematicandreproducibleeﬀect.

ThefocusonfalsiﬁcationbyPopperasthegoaloftests,andfalsiﬁcationasthedeﬁningcriterionforascientiﬁctheoryorhypothesis,clearlyisstronglyredolentofFisher’sthinking.Whileevidenceofdirectinﬂuenceisvirtuallyabsent,theviewsofPopperagreewiththestatementbyFisher([9],p.16)thateveryexperimentmaybesaidtoexistonlyinordertogivethefactsthechanceofdisprovingthenullhypothesis.However,becausePopper’spositiondenieseverhavinggroundsforinferenceaboutreliability,hedeniesthatwecaneverhavegroundsforinferringreproducibledeviations.

Theadvantageinthemodernstatisticalframeworkisthattheprobabilitiesarisefromdeﬁningaprobabilitymodeltorepresentthephenomenonofinterest.HadPoppermadeuseofthestatisticaltestingideasbeingdevelopedataroundthesametime,hemighthavebeenabletosubstantiatehisaccountoffalsiﬁcation.Thesecondissueconcernstheproblemofhowtoreasonwhenthedata“agree”withtheprediction.TheargumentfromHentailsdatay,andthatyisobserved,totheinferencethatHiscorrectis,ofcourse,deductivelyinvalid.AcentralproblemforaninductiveaccountistobeableneverthelesstowarrantinferringHinsomesense.However,theclassicalproblem,evenindeterministiccases,isthatmanyrivalhypotheses(somewouldsayinﬁnitelymany)wouldalsopredicty,andthuswouldpassaswellasH.Inorderforatesttobeprobative,onewantsthepredictionfromHtobesomethingthatatthesametimeisinsomesenseverysurprisingandnoteasilyaccountedforwereHfalseandimportantrivalstoHcorrect.Wenowconsiderhowthegapsininductivetestingmaybridgedbyaspeciﬁckindofstatisticalprocedure,thesigniﬁcancetest.2.Statisticalsigniﬁcancetests

Althoughthestatisticalsigniﬁcancetesthasbeenencircledbycontroversiesforover50years,andhasbeenmiredinmisunderstandingsintheliterature,itillustratesinsimpleformanumberofkeyfeaturesoftheperspectiveonfrequentistinductionthatweareconsidering.SeeforexampleMorrisonandHenkel[21]andGibbonsandPratt[11].Sofaraspossible,webeginwiththecoreelementsofsigniﬁcancetestinginaversionverystronglyrelatedtobutinsomerespectsdiﬀerentfrombothFisherianandNeyman-Pearsonapproaches,atleastasusuallyformulated.2.1.Generalremarksanddeﬁnition

WesupposethatwehaveempiricaldatadenotedcollectivelybyyandthatwetreattheseasobservedvaluesofarandomvariableY.Weregardyasofinter-estonlyinsofarasitprovidesinformationabouttheprobabilitydistributionof

Frequentiststatistics:theoryofinductiveinference81

Yasdeﬁnedbytherelevantstatisticalmodel.Thisprobabilitydistributionistoberegardedasanoftensomewhatabstractandcertainlyidealizedrepresentationoftheunderlyingdatageneratingprocess.Nextwehaveahypothesisabouttheprobabilitydistribution,sometimescalledthehypothesisundertestbutmoreof-tenconventionallycalledthenullhypothesisanddenotedbyH0.Weshalllatersetoutanumberofquitediﬀerenttypesofnullhypothesesbutforthemomentwedistinguishbetweenthose,sometimescalledsimple,thatcompletelyspecify(inprinciplenumerically)thedistributionofYandthose,sometimescalledcomposite,thatcompletelyspecifycertainaspectsandwhichleaveunspeciﬁedotheraspects.Inmanywaysthemostelementary,ifsomewhathackneyed,exampleisthatYconsistsofnindependentandidenticallydistributedcomponentsnormallydis-tributedwithunknownmeanµandpossiblyunknownstandarddeviationσ.Asimplehypothesisisobtainedifthevalueofσisknown,equaltoσ0,say,andthenullhypothesisisthatµ=µ0,agivenconstant.Acompositehypothesisinthesamecontextmighthaveσunknownandagainspecifythevalueofµ.

Notethatinthisformulationitisrequiredthatsomeunknownaspectofthedistribution,typicallyoneormoreunknownparameters,ispreciselyspeciﬁed.Thehypothesisthat,forexample,µ≤µ0isnotanacceptableformulationforanullhypothesisinaFisheriantest;whilethismoregeneralformofnullhypothesisisallowedinNeyman-Pearsonformulations.

TheimmediateobjectiveistotesttheconformityoftheparticulardataunderanalysiswithH0insomerespecttobespeciﬁed.Todothisweﬁndafunctiont=t(y)ofthedata,tobecalledtheteststatistic,suchthat

•thelargerthevalueoftthemoreinconsistentarethedatawithH0;

•thecorrespondingrandomvariableT=t(Y)hasa(numerically)knownprob-abilitydistributionwhenH0istrue.

Thesetworequirementsparallelthecorrespondingdeterministicones.Toassesswhetherthereisagenuinediscordancy(orreproducibledeviation)fromH0wedeﬁnetheso-calledp-valuecorrespondingtoanytas

p=p(t)=P(T≥t;H0),

regardedasameasureofconcordancewithH0intherespecttested.Inatleasttheinitialformulationalternativehypotheseslurkintheundergrowthbutarenotexplicitlyformulatedprobabilistically;alsothereisnoquestionofsettinginadvanceapreassignedthresholdvalueand“rejecting”H0ifandonlyifp≤α.Moreover,thejustiﬁcationfortestswillnotbelimitedtoappealstolongrun-behaviorbutwillinsteadidentifyaninferentialorevidentialrationale.Wenowelaborate.2.2.Inductivebehaviorvs.inductiveinference

Thereasoningmayberegardedasastatisticalversionofthevalidformofargumentcalledindeductivelogicmodustollens.ThisinfersthedenialofahypothesisHfromthecombinationthatHentailsE,togetherwiththeinformationthatEisfalse.Becausetherewasahighprobability(1−p)thatalesssigniﬁcantresultwouldhaveoccurredwereH0true,wemayjustifytakinglowp-values,properlycomputed,asevidenceagainstH0.Why?Therearetwomainreasons:

Firstlysucharuleprovideslowerrorrates(i.e.,erroneousrejections)inthelongrunwhenH0istrue,abehavioristicargument.Inlinewithanerror-assessmentviewofstatisticswemaygiveanyparticularvaluep,say,thefollowinghypothetical

82D.G.MayoandD.R.Cox

interpretation:supposethatweweretotreatthedataasjustdecisiveevidenceagainstH0.TheninhypotheticalrepetitionsH0wouldberejectedinalong-runproportionpofthecasesinwhichitisactuallytrue.However,knowledgeofthesehypotheticalerrorprobabilitiesmaybetakentounderwriteadistinctjustiﬁcation.ThisisthatsucharuleprovidesawaytodeterminewhetheraspeciﬁcdatasetisevidenceofadiscordancyfromH0.

Inparticular,alowp-value,solongasitisproperlycomputed,providesevidenceofadiscrepancyfromH0intherespectexamined,whileap-valuethatisnotsmallaﬀordsevidenceofaccordanceorconsistencywithH0(wherethisistobedistinguishedfrompositiveevidenceforH0,asdiscussedbelowinSection2.3).Interestinapplicationsistypicallyinwhetherpisinsomesuchrangeasp≥0.1whichcanberegardedasreasonableaccordancewithH0intherespecttested,orwhetherpisneartosuchconventionalnumbersas0.05,0.01,0.001.Typicalpracticeinmuchappliedworkistogivetheobservedvalueofpinratherapproximateform.Asmallvalueofpindicatesthat(i)H0isfalse(thereisadiscrepancyfromH0)or(ii)thebasisofthestatisticaltestisﬂawed,oftenthatrealerrorshavebeenunderestimated,forexamplebecauseofinvalidindependenceassumptions,or(iii)theplayofchancehasbeenextreme.

Itispartoftheobjectofgoodstudydesignandchoiceofmethodofanalysistoavoid(ii)byensuringthaterrorassessmentsarerelevant.

Thereisnosuggestionwhateverthatthesigniﬁcancetestwouldtypicallybetheonlyanalysisreported.Infact,afundamentaltenetoftheconceptionofinductivelearningmostathomewiththefrequentistphilosophyisthatinductiveinferencerequiresbuildingupincisiveargumentsandinferencesbyputtingtogetherseveraldiﬀerentpiece-mealresults.Althoughthecomplexityofthestorymakesitmorediﬃculttosetoutneatly,as,forexample,ifasinglealgorithmisthoughttocapturethewholeofinductiveinference,thepayoﬀisanaccountthatapproachesthekindoffull-bodiedargumentsthatscientistsbuildupinordertoobtainreliableknowledgeandunderstandingofaﬁeld.

Amidstthecomplexity,signiﬁcancetestreasoningreﬂectsafairlystraightfor-wardconceptionofevaluatingevidenceanomalousforH0inastatisticalcontext,theonePopperperhapshadinmindbutlackedthetoolstoimplement.Thebasicideaisthaterrorprobabilitiesmaybeusedtoevaluatethe“riskiness”ofthepre-dictionsH0isrequiredtosatisfy,byassessingthereliabilitywithwhichthetestdiscriminateswhether(ornot)theactualprocessgivingrisetothedataaccordswiththatdescribedinH0.KnowledgeofthisprobativecapacityallowsdeterminingifthereisstrongevidenceofdiscordancyThereasoningisbasedonthefollowingfrequentistprincipleforidentifyingwhetherornotthereisevidenceagainstH0:FEV(i)yis(strong)evidenceagainstH0,i.e.(strong)evidenceofdiscrepancyfromH0,ifandonlyif,whereH0acorrectdescriptionofthemechanismgener-atingy,then,withhighprobability,thiswouldhaveresultedinalessdiscordantresultthanisexempliﬁedbyy.

AcorollaryofFEVisthatyisnot(strong)evidenceagainstH0,iftheproba-bilityofamorediscordantresultisnotverylow,evenifH0iscorrect.Thatis,ifthereisamoderatelyhighprobabilityofamorediscordantresult,evenwereH0correct,thenH0accordswithyintherespecttested.

Somewhatmorecontroversialistheinterpretationofafailuretoﬁndasmallp-value;butanadequateconstrualmaybebuiltontheaboveformofFEV.

Frequentiststatistics:theoryofinductiveinference83

2.3.Failureandconﬁrmation

ThediﬃcultywithregardingamodestvalueofpasevidenceinfavourofH0isthataccordancebetweenH0andymayoccurevenifrivalstoH0seriouslydiﬀerentfromH0aretrue.Thisissueisparticularlyacutewhentheamountofdataislimited.However,sometimeswecanﬁndevidenceforH0,understoodasanassertionthataparticulardiscrepancy,ﬂaw,orerrorisabsent,andwecandothisbymeansofteststhat,withhighprobability,wouldhavereportedadiscrepancyhadonebeenpresent.AsmuchasNeymanisassociatedwithautomaticdecision-liketechniques,inpracticeatleast,bothheandE.S.Pearsonregardedtheappropriatechoiceoferrorprobabilitiesasreﬂectingthespeciﬁccontextofinterest(Neyman[23],Pearson[24]).

Therearetwodiﬀerentissuesinvolved.Oneiswhetheraparticularvalueofpistobeusedasathresholdineachapplication.ThisistheproceduresetoutinmostifnotallformalaccountsofNeyman-Pearsontheory.Thesecondissueiswhethercontroloflong-runerrorratesisajustiﬁcationforfrequentisttestsorwhethertheultimatejustiﬁcationoftestsliesintheirroleininterpretingevidenceinparticularcases.Intheaccountgivenhere,theachievedvalueofpisreported,atleastapproximately,andthe“accept-reject”accountispurelyhypotheticaltogivepanoperationalinterpretation.E.S.Pearson[24]isknowntohavedisassociatedhimselffromanarrowbehaviouristinterpretation(Mayo[17]).Neyman,atleastinhisdiscussionwithCarnap(Neyman[23])seemsalsotohintatadistinctionbetweenbehaviouralandinferentialinterpretations.

Inanattempttoclarifythenatureoffrequentiststatistics,Neymaninthisdiscussionwasconcernedwiththeterm“degreeofconﬁrmation”usedbyCarnap.Inthecontextofanexamplewhereanoptimumtesthadfailedto“reject”H0,Neymanconsideredwhetherthis“conﬁrmed”H0.Henotedthatthisdependsonthemeaningofwordssuchas“conﬁrmation”and“conﬁdence”andthatinthecontextwhereH0hadnotbeen“rejected”itwouldbe“dangerous”toregardthisasconﬁrmationofH0ifthetestinfacthadlittlechanceofdetectinganimportantdiscrepancyfromH0evenifsuchadiscrepancywerepresent.Ontheotherhandifthetesthadappreciablepowertodetectthediscrepancythesituationwouldbe“radicallydiﬀerent”.

Neymanishighlightinganinductivefallacyassociatedwith“negativeresults”,namelythatifdatayyieldatestresultthatisnotstatisticallysigniﬁcantlydif-ferentfromH0(e.g.,thenullhypothesisof’noeﬀect’),andyetthetesthassmallprobabilityofrejectingH0,evenwhenaseriousdiscrepancyexists,thenyisnotgoodevidenceforinferringthatH0isconﬁrmedbyy.Onemaybeconﬁdentintheabsenceofadiscrepancy,accordingtothisargument,onlyifthechancethatthetestwouldhavecorrectlydetectedadiscrepancyishigh.

Neymancomparesthissituationwithinterpretationsappropriateforinductivebehaviour.Hereconﬁrmationandconﬁdencemaybeusedtodescribethechoiceofaction,forexamplerefrainingfromannouncingadiscoveryorthedecisiontotreatH0assatisfactory.Therationaleisthepragmaticbehavioristiconeofcontrollingerrorsinthelong-run.ThisdistinctionimpliesthatevenforNeymanevidencefordecidingmayrequireadistinctcriterionthanevidenceforbelieving;butunfortu-natelyNeymandidnotsetoutthelatterexplicitly.WeproposethattheneededevidentialprincipleisanadaptionofFEV(i)forthecaseofap-valuethatisnotsmall:

FEV(ii):Amoderatepvalueisevidenceoftheabsenceofadiscrepancyδfrom

84D.G.MayoandD.R.Cox

H0,onlyifthereisahighprobabilitythetestwouldhavegivenaworseﬁtwithH0(i.e.,smallerpvalue)wereadiscrepancyδtoexist.FEV(ii)especiallyarisesinthecontextof“embedded”hypotheses(below).

Whatmakesthekindofhypotheticalreasoningrelevanttothecaseathandisnotsolelyorprimarilythelong-runlowerrorratesassociatedwithusingthetool(ortest)inthismanner;itisratherwhatthoseerrorratesrevealaboutthedatagener-atingsourceorphenomenon.Theerror-basedcalculationsprovidereassurancethatincorrectinterpretationsoftheevidencearebeingavoidedintheparticularcase.Todistinguishbetweenthis“evidential”justiﬁcationofthereasoningofsigniﬁcancetests,andthe“behavioristic”one,itmayhelptoconsideraveryinformalexampleofapplyingthisreasoning“tothespeciﬁccase”.Thussupposethatweightgainismeasuredbywell-calibratedandstablemethods,possiblyusingseveralmeasuringinstrumentsandobserversandtheresultsshownegligiblechangeoveratestpe-riodofinterest.Thismayberegardedasgroundsforinferringthattheindividual’sweightgainisnegligiblewithinlimitssetbythesensitivityofthescales.Why?Whileitistruethatbyfollowingsuchaprocedureinthelongrunonewouldrarelyreportweightgainserroneously,thatisnottherationalefortheparticularinference.Thejustiﬁcationisratherthattheerrorprobabilisticpropertiesoftheweighingprocedurereﬂectwhatisactuallythecaseinthespeciﬁcinstance.(ThisshouldbedistinguishedfromtheevidentialinterpretationofNeyman–Pearsonthe-orysuggestedbyBirnbaum[1],whichisnotdata-dependent.)

Thesigniﬁcancetestisameasuringdeviceforaccordancewithaspeciﬁedhy-pothesiscalibrated,aswithmeasuringdevicesingeneral,byitsperformanceinrepeatedapplications,inthiscaseassessedtypicallytheoreticallyorbysimulation.Justaswiththeuseofmeasuringinstruments,appliedtoaspeciﬁccase,weem-ploytheperformancefeaturestomakeinferencesaboutaspectsoftheparticularthingthatismeasured,aspectsthatthemeasuringtoolisappropriatelycapableofrevealing.

Ofcourseforthistoholdtheprobabilisticlong-runcalculationsmustbeasrelevantasfeasibletothecaseinhand.Theimplementationofthissurfacesinstatisticaltheoryindiscussionsofconditionalinference,thechoiceofappropriatedistributionfortheevaluationofp.Diﬃcultiessurroundingthisseemmoretechnicalthanconceptualandwillnotbedealtwithhere,excepttonotethattheexerciseofapplying(orattemptingtoapply)FEVmayhelptoguidetheappropriatetestspeciﬁcation.

3.Typesofnullhypothesisandtheircorrespondinginductiveinferences

Inthestatisticalanalysisofscientiﬁcandtechnologicaldata,thereisvirtuallyalwaysexternalinformationthatshouldenterinreachingconclusionsaboutwhatthedataindicatewithrespecttotheprimaryquestionofinterest.Typically,thesebackgroundconsiderationsenternotbyaprobabilityassignmentbutbyidentifyingthequestiontobeasked,designingthestudy,interpretingthestatisticalresultsandrelatingthoseinferencestoprimaryscientiﬁconesandusingthemtoextendandsupportunderlyingtheory.Judgmentsaboutwhatisrelevantandinformativemustbesuppliedforthetoolstobeusednon-fallaciouslyandasintended.Nevertheless,thereareaclusterofsystematicusesthatmaybesetoutcorrespondingtotypesoftestandtypesofnullhypothesis.

Frequentiststatistics:theoryofinductiveinference85

3.1.Typesofnullhypothesis

Wenowdescribeanumberoftypesofnullhypothesis.ThediscussionampliﬁesthatgivenbyCox([4],[5])andbyCoxandHinkley[6].Ourgoalhereisnottogiveaguideforthepanoplyofcontextsaresearchermightface,butrathertoelucidatesomeofthediﬀerentinterpretationsoftestresultsandtheassociatedp-values.InSection4.3,weconsiderthedeeperinterpretationofthecorrespondinginductiveinferencesthat,inourview,are(andarenot)licensedbyp-valuereasoning.1.Embeddednullhypotheses.Intheseproblemsthereisformulated,notonlyaprobabilitymodelforthenullhypothesis,butalsomodelsthatrepresentotherpossibilitiesinwhichthenullhypothesisisfalseand,usually,thereforerepresentpossibilitieswewouldwishtodetectifpresent.Amongthenumberofpossiblesituations,inthemostcommonthereisaparametricfamilyofdistributionsindexedbyanunknownparameterθpartitionedintocomponentsθ=(φ,λ),suchthatthenullhypothesisisthatφ=φ0,withλanunknownnuisanceparameterand,atleastintheinitialdiscussionwithφone-dimensional.Interestfocusesonalternativesφ>φ0.

Thisformulationhasthetechnicaladvantagethatitlargelydeterminestheap-propriateteststatistict(y)bytherequirementofproducingthemostsensitivetestpossiblewiththedataathand.

Therearetwosomewhatdiﬀerentversionsoftheaboveformulation.Inonethefullfamilyisatentativeformulationintendednottosomuchasapossiblebaseforultimateinterpretationbutasadevicefordeterminingasuitableteststatistic.Anexampleistheuseofaquadraticmodeltotestadequacyofalinearrelation;onthewholepolynomialregressionsareapoorbaseforﬁnalanalysisbutveryconvenientandinterpretablefordetectingsmalldeparturesfromagivenform.Inthesecondcasethefamilyisasolidbaseforinterpretation.Conﬁdenceintervalsforφhaveareasonableinterpretation.

Oneotherpossibility,thatarisesveryrarely,isthatthereisasimplenullhypoth-esisandasinglesimplealternative,i.e.onlytwopossibledistributionsareunderconsideration.Ifthetwohypothesesareconsideredonanequalbasistheanalysisistypicallybetterconsideredasoneofhypotheticaloractualdiscrimination,i.e.ofdeterminingwhichoneoftwo(ormore,generallyaverylimitednumber)ofpossibilitiesisappropriate,treatingthepossibilitiesonaconceptuallyequalbasis.Therearetwobroadapproachesinthiscase.Oneistousethelikelihoodratioasanindexofrelativeﬁt,possiblyinconjunctionwithanapplicationofBayestheorem.Theother,moreinaccordwiththeerrorprobabilityapproach,istotakeeachmodelinturnasanullhypothesisandtheotherasalternativeleadingtoanassessmentastowhetherthedataareinaccordwithboth,oneorneitherhypothesis.EssentiallythesameinterpretationresultsbyapplyingFEVtothiscase,whenitisframedwithinaNeyman–Pearsonframework.

Wecancallthesethreecasesthoseofaformalfamilyofalternatives,ofawell-foundedfamilyofalternativesandofafamilyofdiscretepossibilities.

2.Dividingnullhypotheses.Quiteoften,especiallybutnotonlyintechnologicalapplications,thefocusofinterestconcernsacomparisonoftwoormoreconditions,processesortreatmentswithnoparticularreasonforexpectingtheoutcometobeexactlyornearlyidentical,e.g.,comparedwithastandardanewdrugmayincreaseormaydecreasesurvivalrates.

One,ineﬀect,combinestwotests,theﬁrsttoexaminethepossibilitythatµ>µ0,

86D.G.MayoandD.R.Cox

say,theotherforµ<µ0.Inthiscase,thetwo-sidedtestcombinesbothone-sidedtests,eachwithitsownsigniﬁcancelevel.Thesigniﬁcancelevelistwicethesmallerp,becauseofa“selectioneﬀect”(CoxandHinkley[6],p.106).WereturntothisissueinSection4.Thenullhypothesisofzerodiﬀerencethendividesthepossiblesituationsintotwoqualitativelydiﬀerentregionswithrespecttothefeaturetested,thoseinwhichoneofthetreatmentsissuperiortotheotherandasecondinwhichitisinferior.

3.Nullhypothesesofabsenceofstructure.Inquiteanumberofrelativelyempir-icallyconceivedinvestigationsinﬁeldswithoutaveryﬁrmtheorybase,dataarecollectedinthehopeofﬁndingstructure,oftenintheformofdependenciesbetweenfeaturesbeyondthosealreadyknown.Inepidemiologythistakestheformoftestsofpotentialriskfactorsforadiseaseofunknownaetiology.

4.Nullhypothesesofmodeladequacy.Eveninthefullyembeddedcasewherethereisafullfamilyofdistributionsunderconsideration,richenoughpotentiallytoexplainthedatawhetherthenullhypothesisistrueorfalse,thereisthepossibilitythatthereareimportantdiscrepancieswiththemodelsuﬃcienttojustifyextension,modiﬁcationortotalreplacementofthemodelusedforinterpretation.Inmanyﬁeldstheinitialmodelsusedforinterpretationarequitetentative;inothers,notablyinsomeareasofphysics,themodelshaveaquitesolidbaseintheoryandextensiveexperimentation.Butinallcasesthepossibilityofmodelmisspeciﬁcationhastobefacedevenifonlyinformally.

Thereisthenanuneasychoicebetweenarelativelyfocusedteststatisticde-signedtobesensitiveagainstspecialkindsofmodelinadequacy(powerfulagainstspeciﬁcdirectionsofdeparture),andso-calledomnibusteststhatmakenostrongchoicesaboutthenatureofdepartures.Clearlythelatterwilltendtobeinsensitive,andoftenextremelyinsensitive,againstspeciﬁcalternatives.Thetwotypesbroadlycorrespondtochi-squaredtestswithsmallandlargenumbersofdegreesoffreedom.Forthefocusedtestwemayeitherchooseasuitableteststatisticor,almostequiv-alently,anotionalfamilyofalternatives.ForexampletoexamineagreementofnindependentobservationswithaPoissondistributionwemightineﬀecttesttheagreementofthesamplevariancewiththesamplemeanbyachi-squareddisper-siontest(oritsexactequivalent)orembedthePoissondistributionin,forexample,anegativebinomialfamily.

5.Substantively-basednullhypotheses.Incertainspecialcontexts,nullresultsmayindicatesubstantiveevidenceforscientiﬁcclaimsincontextsthatmeritaﬁfthcategory.Here,atheoryTforwhichthereisappreciabletheoreticaland/orempiricalevidencepredictsthatH0is,atleasttoaverycloseapproximation,thetruesituation.

(a)Inoneversion,theremayberesultsapparentlyanomalousforT,andatestisdesignedtohaveampleopportunitytorevealadiscordancywithH0iftheanomalousresultsaregenuine.

(b)InasecondversionarivaltheoryT∗predictsaspeciﬁeddiscrepancyfromH0.andthesigniﬁcancetestisdesignedtodiscriminatebetweenTandtherivaltheoryT∗(inathusfarnottesteddomain).

Foranexampleof(a)physicaltheorysuggeststhatbecausethequantumofen-ergyinnonionizingelectro-magneticﬁelds,suchasthosefromhighvoltagetrans-missionlines,ismuchlessthanisrequiredtobreakamolecularbond,thereshouldbenocarcinogeniceﬀectfromexposuretosuchﬁelds.Thusinarandomizedex-

Frequentiststatistics:theoryofinductiveinference87

perimentinwhichtwogroupsofmiceareunderidenticalconditionsexceptthatonegroupisexposedtosuchaﬁeld,thenullhypothesisthatthecancerincidenceratesinthetwogroupsareidenticalmaywellbeexactlytrueandwouldbeaprimefocusofinterestinanalysingthedata.Ofcoursethenullhypothesisofthisgeneralkinddoesnothavetobeamodelofzeroeﬀect;itmightrefertoagreementwithpreviouswell-establishedempiricalﬁndingsortheory.3.2.Somegeneralpoints

Wehaveintheabovedescribedessentiallyone-sidedtests.Theextensiontotwo-sidedtestsdoesinvolvesomeissuesofdeﬁnitionbutweshallnotdiscussthesehere.

Severalofthetypesofnullhypothesisinvolveanincompleteprobabilityspeci-ﬁcation.Thatis,wemayhaveonlythenullhypothesisclearlyspeciﬁed.Itmightbearguedthatafullprobabilityformulationshouldalwaysbeattemptedcoveringbothnullandfeasiblealternativepossibilities.Thismayseemsensibleinprinciplebutasastrategyfordirectuseitisoftennotfeasible;inanycasemodelsthatwouldcoverallreasonablepossibilitieswouldstillbeincompleteandwouldtendtomakeevensimpleproblemscomplicatedwithsubstantialharmfulside-eﬀects.

Note,however,thatinalltheformulationsusedheresomenotionofexplanationsofthedataalternativetothenullhypothesisisinvolvedbythechoiceofteststatis-tic;theissueiswhenthischoiceismadeviaanexplicitprobabilisticformulation.ThegeneralprincipleofevidenceFEVhelpsustoseethatinspeciﬁedcontexts,theformersuﬃcesforcarryingoutanevidentialappraisal(seeSection3.3).

Itis,however,sometimesarguedthatthechoiceofteststatisticcanbebasedonthedistributionofthedataunderthenullhypothesisalone,ineﬀectchoosingminusthelogprobabilityasteststatistic,thussummingprobabilitiesoverallsamplepointsasorlessprobablethanthatobserved.Whilethisoftenleadstosensibleresultsweshallnotfollowthatroutehere.

3.3.Inductiveinferencesbasedonoutcomesoftests

Howdoessigniﬁcancetestreasoningunderwriteinductiveinferencesorevidentialevaluationsinthevariouscases?Thehypotheticaloperationalinterpretationofthep-valueisclearbutwhatarethedeeperimplicationseitherofamodestorofasmallvalueofp?Thesedependsstronglybothon(i)thetypeofnullhypothesis,and(ii)thenatureofthedepartureoralternativebeingprobed,aswellas(iii)whetherweareconcernedwiththeinterpretationofparticularsetsofdata,asinmostdetailedstatisticalwork,orwhetherweareconsideringabroadmodelforanalysisandinterpretationinaﬁeldofstudy.ThelatterisclosetothetraditionalNeyman-Pearsonformulationofﬁxingacriticallevelandaccepting,insomesense,H0ifp>αandrejectingH0otherwise.Weconsidersomeofthefamiliarshortcomingsofaroutineormechanicaluseofp-values.3.4.Theroutine-behavioruseofp-values

Imagineonesetsα=0.05andthatresultsleadtoapublishablepaperifandonlyfortherelevantp,thedatayieldp<0.05.Therationaleisthebehavioristiconeoutlinedearlier.Nowthegreatmajorityofstatisticaldiscussion,goingbacktoYates

88D.G.MayoandD.R.Cox

[32]andearlier,deploressuchanapproach,bothoutofaconcernthatitencouragesmechanical,automaticandunthinkingprocedures,aswellasadesiretoemphasizeestimationofrelevanteﬀectsovertestingofhypotheses.Indeedafewjournalsinsomeﬁeldshaveineﬀectbannedtheuseofp-values.Inothers,suchasanumberofareasofepidemiology,itisconventionaltoemphasize95%conﬁdenceintervals,asindeedisinlinewithmuchmainstreamstatisticaldiscussion.Ofcourse,thisdoesnotfreeonefromneedingtogiveaproperfrequentistaccountoftheuseandinterpretationofconﬁdencelevels,whichwedonotdohere(thoughseeSection3.6).Neverthelesstherelativelymechanicaluseofp-values,whileopentoparody,isnotfarfrompracticeinsomeﬁelds;itdoesserveasascreeningdevice,recognizingthepossibilityoferror,anddecreasingthepossibilityofthepublicationofmislead-ingresults.Asomewhatsimilarroleoftestsarisesintheworkofregulatoryagents,inparticulartheFDA.Whilerequiringstudiestoshowplessthansomepreas-signedlevelbyapreordainedtestmaybeinﬂexible,andthechoiceofcriticallevelarbitrary,neverthelesssuchprocedureshavevirtuesofimpartialityandrelativeindependencefromunreasonablemanipulation.Whileadheringtoaﬁxedp-valuemayhavethedisadvantageofbiasingtheliteraturetowardspositiveconclusions,itoﬀersanappealingassuranceofsomeknownanddesirablelong-runproperties.TheywillbeseentobeparticularlyappropriateforExample3ofSection4.2.3.5.Theinductive-evidenceuseofp-values

Wenowturntotheuseofsigniﬁcancetestswhich,whilemorecommon,isatthesametimemorecontroversial;namelyasonetooltoaidtheanalysisofspeciﬁcsetsofdata,and/orbaseinductiveinferencesondata.Thediscussionpresupposesthattheprobabilitydistributionusedtoassessthep-valueisasappropriateaspossibletothespeciﬁcdataunderanalysis.

Thegeneralfrequentistprincipleforinductivereasoning,FEV,orsomethinglikeit,providesaguidefortheappropriatestatementaboutevidenceorinfer-enceregardingeachtypeofnullhypothesis.Muchasonemakesinferencesaboutchangesinbodymassbasedonperformancecharacteristicsofvariousscales,onemaymakeinferencesfromsigniﬁcancetestresultsbyusingerrorratepropertiesoftests.Theyindicatethecapacityoftheparticulartesttohaverevealedinconsis-tenciesanddiscrepanciesintherespectsprobed,andthisinturnallowsrelatingp-valuestohypothesesabouttheprocessasstatisticallymodelled.ItfollowsthatanadequatefrequentistaccountofinferenceshouldstrivetosupplytheinformationtoimplementFEV.

EmbeddedNulls.Inthecaseofembeddednullhypotheses,itisstraightforwardtousesmallp-valuesasevidenceofdiscrepancyfromthenullinthedirectionofthealternative.Suppose,however,thatthedataarefoundtoaccordwiththenullhypothesis(pnotsmall).Onemay,ifitisofinterest,regardthisasevidencethatanydiscrepancyfromthenullislessthanδ,usingthesamelogicinsigniﬁcancetesting.Insuchcasesconcordancewiththenullmayprovideevidenceoftheabsenceofadiscrepancyfromthenullofvarioussizes,asstipulatedinFEV(ii).

ToinfertheabsenceofadiscrepancyfromH0aslargeasδwemayexaminetheprobabilityβ(δ)ofobservingaworseﬁtwithH0ifµ=µ0+δ.Ifthatprobabilityisnearonethen,followingFEV(ii),thedataaregoodevidencethatµ<µ0+δ.Thusβ(δ)mayberegardedasthestringencyorseveritywithwhichthetesthasprobedthediscrepancyδ;equivalentlyonemightsaythatµ<µ0+δhaspassedaseveretest(Mayo[17]).

Frequentiststatistics:theoryofinductiveinference89

ThisavoidsunwarrantedinterpretationsofconsistencywithH0withinsensitivetests.Suchanassessmentismorerelevanttospeciﬁcdatathanisthenotionofpower,whichiscalculatedrelativetoapredesignatedcriticalvaluebeyondwhichthetest“rejects”thenull.Thatis,powerappertainstoaprespeciﬁedrejectionregion,nottothespeciﬁcdataunderanalysis.

Althoughoversensitivityisusuallylesslikelytobeaproblem,ifatestissosensitivethatap-valueasorevensmallerthantheoneobserved,isprobableevenwhenµ<µ0+δ,thenasmallvalueofpisnotevidenceofdeparturefromH0inexcessofδ.

Ifthereisanexplicitfamilyofalternatives,itwillbepossibletogiveasetofconﬁdenceintervalsfortheunknownparameterdeﬁningH0andthiswouldgiveamoreextendedbasisforconclusionsaboutthedeﬁningparameter.

Dividingandabsenceofstructurenulls.Inthecaseofdividingnulls,discordancywiththenull(usingthetwo-sidedvalueofp)indicatesdirectionofdeparture(e.g.,whichoftwotreatmentsissuperior);accordancewithH0indicatesthatthesedatadonotprovideadequateevidenceevenofthedirectionofanydiﬀerence.Oneoftenhearscriticismsthatitispointlesstotestanullhypothesisknowntobefalse,butevenifwedonotexpecttwomeans,say,tobeequal,thetestisinformativeinordertodividethedeparturesintoqualitativelydiﬀerenttypes.Theinterpretationisanalogouswhenthenullhypothesisisoneofabsenceofstructure:amodestvalueofpindicatesthatthedataareinsuﬃcientlysensitivetodetectstructure.Ifthedataarelimitedthismaybenomorethanawarningagainstover-interpretationratherthanevidenceforthinkingthatindeedthereisnostructurepresent.Thatisbecausethetestmayhavehadlittlecapacitytohavedetectedanystructurepresent.Asmallvalueofp,however,indicatesevidenceofagenuineeﬀect;thattolookforasubstantiveinterpretationofsuchaneﬀectwouldnotbeintrinsicallyerror-prone.

Analogousreasoningapplieswhenassessmentsabouttheprobativenessorsen-sitivityoftestsareinformal.Ifthedataaresoextensivethataccordancewiththenullhypothesisimpliestheabsenceofaneﬀectofpracticalimportance,andarea-sonablyhighp-valueisachieved,thenitmaybetakenasevidenceoftheabsenceofaneﬀectofpracticalimportance.Likewise,ifthedataareofsuchalimitedextentthatitcanbeassumedthatdatainaccordwiththenullhypothesisareconsistentalsowithdeparturesofscientiﬁcimportance,thenahighp-valuedoesnotwarrantinferringtheabsenceofscientiﬁcallyimportantdeparturesfromthenullhypothesis.Nullsofmodeladequacy.Whennullhypothesesareassertionsofmodeladequacy,theinterpretationoftestresultswilldependonwhetheronehasarelativelyfocusedteststatisticdesignedtobesensitiveagainstspecialkindsofmodelinadequacy,orsocalledomnibustests.Concordancewiththenullintheformercasegivesevidenceofabsenceofthetypeofdeparturethatthetestissensitiveindetecting,whereas,withtheomnibustest,itislessinformative.Inbothtypesoftests,asmallp-valueisevidenceofsomedeparture,butsolongasvariousalternativemodelscouldaccountfortheobservedviolation(i.e.,solongasthistesthadlittleabilitytodiscriminatebetweenthem),thesedatabythemselvesmayonlyprovideprovisionalsuggestionsofalternativemodelstotry.

Substantivenulls.Intheprecedingcases,accordancewithanullcouldatmostprovideevidencetoruleoutdiscrepanciesofspeciﬁedamountsortypes,accordingtotheabilityofthetesttohaverevealedthediscrepancy.Morecanbesaidinthecaseofsubstantivenulls.Ifthenullhypothesisrepresentsapredictionfrom

90D.G.MayoandD.R.Cox

sometheorybeingcontemplatedforgeneralapplicability,consistencywiththenullhypothesismayberegardedassomeadditionalevidenceforthetheory,especiallyifthetestanddataaresuﬃcientlysensitivetoexcludemajordeparturesfromthetheory.AnaspectisencapsulatedinFisher’saphorism(Cochran[3])thattohelpmakeobservationalstudiesmorenearlybearacausalinterpretation,oneshouldmakeone’stheorieselaborate,bywhichhemeantoneshouldplanavarietyoftestsofdiﬀerentconsequencesofatheory,toobtainacomprehensivecheckofitsimplications.Thelimitedresultthatonesetofdataaccordswiththetheoryaddsonepiecetotheevidencewhoseweightstemsfromaccumulatinganabilitytorefutealternativeexplanations.

Intheﬁrsttypeofexampleunderthisrubric,theremaybeapparentlyanomalousresultsforatheoryorhypothesisT,whereThassuccessfullypassedappreciabletheoreticaland/orempiricalscrutiny.WeretheapparentlyanomalousresultsforTgenuine,itisexpectedthatH0willberejected,sothatwhenitisnot,theresultsarepositiveevidenceagainsttherealityoftheanomaly.Inasecondtypeofcase,oneagainhasawell-testedtheoryT,andarivaltheoryT∗isdeterminedtoconﬂictwithTinathusfaruntesteddomain,withrespecttoaneﬀect.ByidentifyingthenullwiththepredictionfromT,anydiscrepanciesinthedirectionofT∗aregivenaverygoodchancetobedetected,suchthat,ifnosigniﬁcantdepartureisfound,thisconstitutesevidenceforTintherespecttested.

Althoughthegeneraltheoryofrelativity,GTR,wasnotfacinganomaliesinthe1960s,rivalstotheGTRpredictedabreakdownoftheWeakEquivalencePrincipleformassiveself-gravitatingbodies,e.g.,theearth-moonsystem:thiseﬀect,calledtheNordvedteﬀectwouldbe0forGTR(identiﬁedwiththenullhypothesis)andnon-0forrivals.Measurementsoftheroundtriptraveltimesbetweentheearthandmoon(between1969and1975)enabledtheexistenceofsuchananomalyforGTRtobeprobed.FindingnoevidenceagainstthenullhypothesissetupperboundstothepossibleviolationoftheWEP,andbecausethetestsweresuﬃcientlysensitive,thesemeasurementsprovidedgoodevidencethattheNordvedteﬀectisabsent,andthusevidenceforthenullhypothesis(Will[31]).NotethatsuchanegativeresultdoesnotprovideevidenceforallofGTR(inallitsareasofprediction),butitdoesprovideevidenceforitscorrectnesswithrespecttothiseﬀect.Thelogicisthis:theoryTpredictsH0isatleastaverycloseapproximationtothetruesituation;rivaltheoryT∗predictsaspeciﬁeddiscrepancyfromH0,andthetesthashighprobabilityofdetectingsuchadiscrepancyfromTwereT∗correct.Detectingnodiscrepancyisthusevidenceforitsabsence.3.6.Conﬁdenceintervals

Asnotedaboveinmanyproblemstheprovisionofconﬁdenceintervals,inprincipleatarangeofprobabilitylevels,givesthemostproductivefrequentistanalysis.Ifso,thenconﬁdenceintervalanalysisshouldalsofallunderourgeneralfrequentistprinciple.Itdoes.Inonesidedtestingofµ=µ0againstµ>µ0,asmallp-valuecorrespondstoµ0being(just)excludedfromthecorresponding(1−2p)(two-sided)conﬁdenceinterval(or1−pfortheone-sidedinterval).Wereµ=µL,thelowerconﬁdencebound,thenalessdiscordantresultwouldoccurwithhighprobability(1−p).ThusFEVlicensestakingthisasevidenceofinconsistencywithµ=µL(inthepositivedirection).Moreover,thisreasoningshowstheadvantageofconsideringseveralconﬁdenceintervalsatarangeoflevels,ratherthanjustreportingwhetherornotagivenparametervalueiswithintheintervalataﬁxedconﬁdencelevel.

Frequentiststatistics:theoryofinductiveinference91

Neymandevelopedthetheoryofconﬁdenceintervalsabinitioi.e.relyingonlyimplicitlyratherthanexplicitlyonhisearlierworkwithE.S.Pearsononthetheoryoftests.Itistosomeextentamatterofpresentationwhetheroneregardsintervalestimationassodiﬀerentinprinciplefromtestinghypothesesthatitisbestdevel-opedseparatelytopreservetheconceptualdistinction.Ontheotherhandthereareconsiderableadvantagestoregardingaconﬁdencelimit,intervalorregionasthesetofparametervaluesconsistentwiththedataatsomespeciﬁedlevel,asassessedbytestingeachpossiblevalueinturnbysomemutuallyconcordantprocedures.Inparticularthisapproachdealspainlesslywithconﬁdenceintervalsthatarenullorwhichconsistofallpossibleparametervalues,atsomespeciﬁedsigniﬁcancelevel.Suchnullorinﬁniteregionssimplyrecordthatthedataareinconsistentwithallpossibleparametervalues,orareconsistentwithallpossiblevalues.Itiseasytoconstructexampleswheretheseseementirelyappropriateconclusions.4.Somecomplications:selectioneﬀects

Theidealizedformulationinvolvedintheinitialdeﬁnitionofasigniﬁcancetestinprinciplestartswithahypothesisandateststatistic,thenobtainsdata,thenappliesthetestandlooksattheoutcome.Thehypotheticalprocedureinvolvedinthedeﬁnitionofthetestthenmatchesreasonablycloselywhatwasdone;thepossibleoutcomesarethediﬀerentpossiblevaluesofthespeciﬁedteststatistic.Thispermitsfeaturesofthedistributionoftheteststatistictoberelevantforlearningaboutcorrespondingfeaturesofthemechanismgeneratingthedata.Therearevariousreasonswhytheprocedureactuallyfollowedmaybediﬀerentandwenowconsideronebroadaspectofthat.

Itoftenhappensthateitherthenullhypothesisortheteststatisticareinﬂuencedbypreliminaryinspectionofthedata,sothattheactualproceduregeneratingtheﬁnaltestresultisaltered.Thisinturnmayalterthecapabilitiesofthetesttodetectdiscrepanciesfromthenullhypothesesreliably,callingforadjustmentsinitserrorprobabilities.

Totheextentthatpisviewedasanaspectofthelogicalormathematicalrelationbetweenthedataandtheprobabilitymodelsuchpreliminarychoicesareirrelevant.Thiswillnotsuﬃceinordertoensurethatthep-valuesservetheirintendedpurposeforfrequentistinference,whetherinbehavioralorevidentialcontexts.Totheextentthatonewantstheerror-basedcalculationsthatgivethetestitsmeaningtobeapplicabletothetasksoffrequentiststatistics,thepreliminaryanalysisandchoicemaybehighlyrelevant.

Thegeneralpointinvolvedhasbeendiscussedextensivelyinbothphilosophicalandstatisticalliteratures,intheformerundersuchheadingsasrequiringnoveltyoravoidingadhochypotheses,underthelatter,asrulesagainstpeekingatthedataorshoppingforsigniﬁcance,andthusrequiringselectioneﬀectstobetakenintoaccount.ThegeneralissueiswhethertheevidentialbearingofdatayonaninferenceorhypothesisH0isalteredwhenH0hasbeeneitherconstructedorselectedfortestinginsuchawayastoresultinaspeciﬁcobservedrelationbetweenH0andy,whetherthatisagreementordisagreement.Thosewhofavourlogicalapproachestoconﬁrmationsayno(e.g.,Mill[20],Keynes[14]),whereasthoseclosertoanerrorstatisticalconceptionsayyes(Whewell[30],Pierce[25]).Followingthelatterphilosophy,PopperrequiredthatscientistssetoutinadvancewhatoutcomestheywouldregardasfalsifyingH0,arequirementthatevenhecametoreject;theentireissueinphilosophyremainsunresolved(Mayo[17]).

92D.G.MayoandD.R.Cox

Errorstatisticalconsiderationsallowgoingfurtherbyprovidingcriteriaforwhenvariousdatadependentselectionsmatterandhowtotakeaccountoftheirinﬂuenceonerrorprobabilities.Inparticular,ifthenullhypothesisischosenfortestingbecausetheteststatisticislarge,theprobabilityofﬁndingsomesuchdiscordanceorothermaybehighevenunderthenull.Thus,followingFEV(i),wewouldnothavegenuineevidenceofdiscordancewiththenull,andunlessthep-valueismodiﬁedappropriately,theinferencewouldbemisleading.Totheextentthatonewantstheerror-basedcalculationsthatgivethetestitsmeaningtosupplyreassurancethatapparentinconsistencyintheparticularcaseisgenuineandnotmerelyduetochance,adjustingthep-valueiscalledfor.

Suchadjustmentsoftenariseincasesinvolvingdatadependentselectionseitherinmodelselectionorconstruction;oftenthequestionofadjustingparisesincasesinvolvingmultiplehypothesestesting,butitisimportantnottoruncasestogethersimplybecausethereisdatadependenceormultiplehypothesistesting.Wenowoutlinesomespecialcasestobringoutthekeypointsindiﬀerentscenarios.Thenweconsiderwhetherallowanceforselectioniscalledforineachcase.4.1.Examples

Example1.Aninvestigatorhas,say,20independentsetsofdata,eachreportingondiﬀerentbutcloselyrelatedeﬀects.Theinvestigatordoesall20testsandreportsonlythesmallestp,whichinfactisabout0.05,anditscorrespondingnullhypoth-esis.Thekeypointsaretheindependenceofthetestsandthefailuretoreporttheresultsfrominsigniﬁcanttests.

Example2.AhighlyidealizedversionoftestingforaDNAmatchwithagivenspecimen,perhapsofacriminal,isthatasearchthroughadata-baseofpossiblematchesisdoneoneatatime,checkingwhetherthehypothesisofagreementwiththespecimenisrejected.Supposethatsensitivityandspeciﬁcityarebothveryhigh.Thatis,theprobabilitiesoffalsenegativesandfalsepositivesarebothverysmall.Theﬁrstindividual,ifany,fromthedata-baseforwhichthehypothesisisrejectedisdeclaredtobethetruematchandtheprocedurestopsthere.

Example3.AmicroarraystudyexaminesseveralthousandgenesforpotentialexpressionofsayadiﬀerencebetweenType1andType2diseasestatus.Therearethusseveralthousandhypothesesunderinvestigationinonestep,eachwithitsassociatednullhypothesis.

Example4.Tostudythedependenceofaresponseoroutcomevariableyonanexplanatoryvariablexitisintendedtousealinearregressionanalysisofyonx.Inspectionofthedatasuggeststhatitwouldbebettertousetheregressionoflogyonlogx,forexamplebecausetherelationismorenearlylinearorbecausesecondaryassumptions,suchasconstancyoferrorvariance,aremorenearlysatisﬁed.Example5.Tostudythedependenceofaresponseoroutcomevariableyonaconsiderablenumberofpotentialexplanatoryvariablesx,adata-dependentproce-dureofvariableselectionisusedtoobtainarepresentationwhichisthenﬁttedbystandardmethodsandrelevanthypothesestested.

Example6.Supposethatpreliminaryinspectionofdatasuggestssometotallyunexpectedeﬀectorregularitynotcontemplatedattheinitialstages.Byaformaltesttheeﬀectisvery“highlysigniﬁcant”.Whatisitreasonabletoconclude?

Frequentiststatistics:theoryofinductiveinference93

4.2.Needforadjustmentsforselection

Thereisnotspacetodiscussalltheseexamplesindepth.Akeyissueconcernswhichofthesesituationsneedanadjustmentformultipletestingordatadependentselectionandwhatthatadjustmentshouldbe.Howdoesthegeneralconceptionofthecharacterofafrequentisttheoryofanalysisandinterpretationhelptoguidetheanswers?

Weproposethatitdoessointhefollowingmanner:Firstlyitmustbeconsideredwhetherthecontextisonewherethekeyconcernisthecontroloferrorratesinaseriesofapplications(behavioristicgoal),orwhetheritisacontextofmakingaspeciﬁcinductiveinferenceorevaluatingspeciﬁcevidence(inferentialgoal).Therelevanterrorprobabilitiesmaybealteredfortheformercontextandnotforthelatter.Secondly,therelevantsequenceofrepetitionsonwhichtobasefrequenciesneedstobeidentiﬁed.Thegeneralrequirementisthatwedonotreportdiscordancewithanullhypothesisbymeansaprocedurethatwouldreportdiscordanciesfairlyfrequentlyeventhoughthenullhypothesisistrue.Ascertainmentoftherelevanthypotheticalseriesonwhichthiserrorfrequencyistobecalculateddemandscon-siderationofthenatureoftheproblemorinference.Morespeciﬁcally,onemustidentifytheparticularobstaclesthatneedtobeavoidedforareliableinferenceintheparticularcase,andthecapacityofthetest,asameasuringinstrument,tohaverevealedthepresenceoftheobstacle.

Whenthegoalisappraisingspeciﬁcevidence,ourmaininterest,FEVgivessomeguidance.MorespeciﬁcallytheproblemariseswhendataareusedtoselectahypothesistotestoralterthespeciﬁcationofanunderlyingmodelinsuchawaythatFEViseitherviolatedoritcannotbedeterminedwhetherFEVissatisﬁed(MayoandKruse[18]).

Example1(Huntingforstatisticalsigniﬁcance).Thetestprocedureisverydiﬀerentfromthecaseinwhichthesinglenullfoundstatisticallysigniﬁcantwaspresetasthehypothesistotest,perhapsitisH0,13,the13thnullhypothesisoutofthe20.InExample1,thepossibleresultsarethepossiblestatisticallysigniﬁcantfactorsthatmightbefoundtoshowa“calculated”statisticalsigniﬁcantdeparturefromthenull.Hencethetype1errorprobabilityistheprobabilityofﬁndingatleastonesuchsigniﬁcantdiﬀerenceoutof20,eventhoughtheglobalnullistrue(i.e.,alltwentyobserveddiﬀerencesareduetochance).Theprobabilitythatthisprocedureyieldsanerroneousrejectiondiﬀersfrom,andwillbemuchgreaterthan,0.05(andisapproximately0.64).Therearediﬀerent,andindeedmanymore,waysonecanerrinthisexamplethanwhenonenullisprespeciﬁed,andthisisreﬂectedintheadjustedp-value.

Thismuchiswellknown,butshouldthisinﬂuencetheinterpretationofthere-sultinacontextofinductiveinference?AccordingtoFEVitshould.Howevertheconcernisnottheavoidanceofoftenannouncinggenuineeﬀectserroneouslyinaseries,theconcernisthatthistestperformspoorlyasatoolfordiscriminatinggenuinefromchanceeﬀectsinthisparticularcase.Becauseatleastonesuchim-pressivedeparture,weknow,iscommonevenifallareduetochance,thetesthasscarcelyreassuredusthatithasdoneagoodjobofavoidingsuchamistakeinthiscase.Evenifthereareothergroundsforbelievingthegenuinenessoftheoneeﬀectthatisfound,wedenythatthistestalonehassuppliedsuchevidence.

Frequentistcalculationsservetoexaminetheparticularcase,wehavebeensay-ing,bycharacterizingthecapabilityofteststohaveuncoveredmistakesininference,andonthosegrounds,the“huntingprocedure”haslowcapacitytohavealertedus

94D.G.MayoandD.R.Cox

to,ineﬀect,temperourenthusiasm,evenwheresuchtemperingiswarranted.If,ontheotherhand,oneadjuststhep-valuetoreﬂecttheoverallerrorrate,thetestagainbecomesatoolthatservesthispurpose.

Example1maybecontrastedtoastandardfactorialexperimentsetuptoinves-tigatetheeﬀectsofseveralexplanatoryvariablessimultaneously.Herethereareanumberofdistinctquestions,eachwithitsassociatedhypothesisandeachwithitsassociatedp-value.Thatweaddressthequestionsviathesamesetofdataratherthanviaseparatesetsofdataisinasenseatechnicalaccident.Eachpiscorrectlyinterpretedinthecontextofitsownquestion.Diﬃcultiesariseforparticularinfer-encesonlyifweineﬀectthrowawaymanyofthequestionsandconcentrateonlyonone,ormoregenerallyasmallnumber,chosenjustbecausetheyhavethesmallestp.Forthenwehavealteredthecapacityofthetesttohavealertedus,bymeansofacorrectlycomputedp-value,whetherwehaveevidencefortheinferenceofinterest.Example2(Explainingaknowneﬀectbyeliminativeinduction).Ex-ample2issuperﬁciallysimilartoExample1,ﬁndingaDNAmatchbeingsome-whatakintoﬁndingastatisticallysigniﬁcantdeparturefromanullhypothesis:onesearchesthroughdataandconcentratesontheonecasewherea“match”withthecriminal’sDNAisfound,ignoringthenon-matches.Ifoneadjustsfor“hunting”inExample1,shouldn’tonedosoinbroadlythesamewayinExample2?No.

InExample1theconcernisthatofinferringagenuine,“reproducible”eﬀect,wheninfactnosucheﬀectexists;inExample2,thereisaknowneﬀectorspeciﬁcevent,thecriminal’sDNA,andreliableproceduresareusedtotrackdownthespeciﬁccauseorsource(asconveyedbythelow“erroneous-match”rate.)Theprobabilityishighthatwewouldnotobtainamatchwithpersoni,ifiwerenotthecriminal;so,byFEV,ﬁndingthematchis,ataqualitativelevel,goodevidencethatiisthecriminal.Moreover,eachnon-matchfound,bythestipulationsoftheexample,virtuallyexcludesthatperson;thus,themoresuchnegativeresultsthestrongeristheevidencewhenamatchisﬁnallyfound.Themorenegativeresultsfound,themoretheinferred“match”isfortiﬁed;whereasinExample1thisisnotso.

Becauseatmostonenullhypothesisofinnocenceisfalse,evidenceofinnocenceononeindividualincreases,evenifonlyslightly,thechanceofguiltofanother.Anassessmentoferrorratesiscertainlypossibleoncethesamplingprocedurefortestingisspeciﬁed.Detailswillnotbegivenhere.

AbroadlyanalogoussituationconcernstheanomalyoftheorbitofMercury:thenumerousfailedattemptstoprovideaNewtonianinterpretationmadeitallthemoreimpressivewhenEinstein’stheorywasfoundtopredicttheanomalousresultspreciselyandwithoutanyadhocadjustments.

Example3(Micro-arraydata).Intheanalysisofmicro-arraydata,areasonablestartingassumptionisthataverylargenumberofnullhypothesesarebeingtestedandthatsomefairlysmallproportionofthemare(strictly)false,aglobalnullhypothesisofnorealeﬀectsatalloftenbeingimplausible.Theproblemisthenoneofselectingthesiteswhereaneﬀectcanberegardedasestablished.Here,theneedforanadjustmentformultipletestingiswarrantedmainlybyapragmaticconcerntoavoid“toomuchnoiseinthenetwork”.Themaininterestisinhowbesttoadjusterrorratestoindicatemosteﬀectivelythegenehypothesesworthfollowingup.Anerror-basedanalysisoftheissuesisthenviathefalse-discoveryrate,i.e.essentiallythelongrunproportionofsitesselectedaspositiveinwhichnoeﬀectispresent.AnalternativeformulationisviaanempiricalBayesmodelandtheconclusionsfromthiscanbelinkedtothefalsediscoveryrate.Thelattermethodmaybepreferable

Frequentiststatistics:theoryofinductiveinference95

becauseanerrorratespeciﬁctoeachselectedgenemaybefound;theevidenceinsomecasesislikelytobemuchstrongerthaninothersandthisdistinctionisblurredinanoverallfalse-discoveryrate.SeeShaﬀer[28]forasystematicreview.Example4(Redeﬁningthetest).Iftestsarerunwithdiﬀerentspeciﬁcations,andtheonegivingthemoreextremestatisticalsigniﬁcanceischosen,thenadjust-mentforselectionisrequired,althoughitmaybediﬃculttoascertainthepreciseadjustment.Byallowingtheresulttoinﬂuencethechoiceofspeciﬁcation,oneisalteringtheproceduregivingrisetothep-value,andthismaybeunacceptable.Whilethesubstantiveissueandhypothesisremainunchangedtheprecisespeciﬁca-tionoftheprobabilitymodelhasbeenguidedbypreliminaryanalysisofthedatainsuchawayastoalterthestochasticmechanismactuallyresponsibleforthetestoutcome.

Ananalogymightbetestingasharpshooter’sabilitybyhavinghimshootandthendrawingabull’s-eyearoundhisresultssoastoyieldthehighestnumberofbull’s-eyes,theso-calledprincipleoftheTexasmarksman.Theskillthatoneisallegedlytestingandmakinginferencesaboutishisabilitytoshootwhenthetargetisgivenandﬁxed,whilethatisnottheskillactuallyresponsiblefortheresultinghighscore.

Bycontrast,ifthechoiceofspeciﬁcationisguidednotbyconsiderationsofthestatisticalsigniﬁcanceofdeparturefromthenullhypothesis,butratherbecausethedataindicatestheneedtoallowforchangestoachievelinearityorconstancyoferrorvariance,noallowanceforselectionseemsneeded.Quitethecontrary:choosingthemoreempiricallyadequatespeciﬁcationgivesreassurancethatthecalculatedp-valueisrelevantforinterpretingtheevidencereliably.(MayoandSpanos[19]).Thismightbejustiﬁedmoreformallybyregardingthespeciﬁcationchoiceasaninformalmaximumlikelihoodanalysis,maximizingoveraparameterorthogonaltothosespecifyingthenullhypothesisofinterest.

Example5(Datamining).ThisexampleisanalogoustoExample1,althoughhowtomaketheadjustmentforselectionmaynotbeclearbecausetheprocedureusedinvariableselectionmaybetortuous.Heretoo,thediﬃcultiesofselectivereportingarebypassedbyspecifyingallthosereasonablysimplemodelsthatareconsistentwiththedataratherthanbychoosingonlyonemodel(CoxandSnell[7]).Thediﬃcultiesofimplementingsuchastrategyarepartlycomputationalratherthanconceptual.Examplesofthissortareimportantinmuchrelativelyelaboratestatisticalanalysisinthatseriesofveryinformallyspeciﬁedchoicesmaybemadeaboutthemodelformulationbestforanalysisandinterpretation(Spanos[29]).Example6(Thetotallyunexpectedeﬀect).Thisraisesmajorproblems.Inlaboratoryscienceswithdataobtainablereasonablyrapidly,anattempttoobtainindependentreplicationoftheconclusionswouldbevirtuallyobligatory.Inothercontextsasearchforotherdatabearingontheissuewouldbeneeded.Highstatis-ticalsigniﬁcanceonitsownwouldbeverydiﬃculttointerpret,essentiallybecauseselectionhastakenplaceanditistypicallyhardorimpossibletospecifywithanyrealismthesetoverwhichselectionhasoccurred.TheconsiderationsdiscussedinExamples1-5,however,maygiveguidance.If,forexample,thesituationisasinExample2(explainingaknowneﬀect)thesourcemaybereliablyidentiﬁedinaprocedurethatfortiﬁes,ratherthandetractsfrom,theevidence.InacaseakintoExample1,thereisaselectioneﬀect,butitisreasonablyclearwhatisthesetofpossibilitiesoverwhichthisselectionhastakenplace,allowingcorrectionofthep-value.Inotherexamples,thereisaselectioneﬀect,butitmaynotbeclearhow

96D.G.MayoandD.R.Cox

tomakethecorrection.Inshort,itwouldbeveryunwisetodismissthepossibilityoflearningfromdatasomethingnewinatotallyunanticipateddirection,butonemustdiscriminatethecontextsinordertogainguidanceforwhatfurtheranalysis,ifany,mightberequired.5.Concludingremarks

Wehavearguedthaterrorprobabilitiesinfrequentisttestsmaybeusedtoevalu-atethereliabilityorcapacitywithwhichthetestdiscriminateswhetherornottheactualprocessgivingrisetodataisinaccordancewiththatdescribedinH0.Knowl-edgeofthisprobativecapacityallowsdeterminationofwhetherthereisstrongevi-denceagainstH0basedonthefrequentistprinciplewesetoutFEV.Whatmakesthekindofhypotheticalreasoningrelevanttothecaseathandisnotthelong-runlowerrorratesassociatedwithusingthetool(ortest)inthismanner;itisratherwhatthoseerrorratesrevealaboutthedatageneratingsourceorphenomenon.WehavenotattemptedtoaddresstherelationbetweenthefrequentistandBayesiananalysesofwhatmayappeartobeverysimilarissues.Afundamentaltenetoftheconceptionofinductivelearningmostathomewiththefrequentistphilosophyisthatinductiveinferencerequiresbuildingupincisiveargumentsandinferencesbyputtingtogetherseveraldiﬀerentpiece-mealresults;wehavesetoutconsiderationstoguidethesepieces.Althoughthecomplexityoftheissuesmakesitmorediﬃculttosetoutneatly,as,forexample,onecouldbyimaginingthatasinglealgorithmencompassesthewholeofinductiveinference,thepayoﬀisanaccountthatap-proachesthekindofargumentsthatscientistsbuildupinordertoobtainreliableknowledgeandunderstandingofaﬁeld.References

[1]Birnbaum,A.(1977).TheNeyman–Pearsontheoryasdecisiontheory,andas

inferencetheory;withacriticismoftheLindley–SavageargumentforBayesiantheory.Synthese36,19–49.MR0652320

[2]Carnap,R.(1962).LogicalFoundationsofProbability.UniversityofChicago

Press.MR0184839

[3]Cochran,W.G.(1965).Theplanningofobservationalstudiesinhuman

populations(withdiscussion).J.R.Statist.Soc.A128,234–265.

[4]Cox,D.R.(1958).Someproblemsconnectedwithstatisticalinference.Ann.

Math.Statist.29,357–372.MR0094890

[5]Cox,D.R.(1977).Theroleofsigniﬁcancetests(withdiscussion).Scand.J.

Statist.4,49–70.MR0448666

[6]Cox,D.R.andHinkley,D.V.(1974).TheoreticalStatistics.Chapman

andHall,London.MR0370837

[7]Cox,D.R.andSnell,E.J.(1974).Thechoiceofvariablesinobservational

studies.J.R.Statist.Soc.C23,51–59.MR0413333

[8]DeFinetti,B.(1974).TheoryofProbability,2vols.Englishtranslationfrom

Italian.Wiley,NewYork.

[9]Fisher,R.A.(1935a).DesignofExperiments.OliverandBoyd,Edinburgh.[10]Fisher,R.A.(1935b).Thelogicofinductiveinference.J.R.Statist.Soc.

98,39–54.

[11]Gibbons,J.D.andPratt,J.W.(1975).P-values:Interpretationand

methodology.AmericanStatistician29,20–25.

Frequentiststatistics:theoryofinductiveinference97

[12]Jeffreys,H.(1961).TheoryofProbability,Thirdedition.OxfordUniversity

Press.MR0187257

[13]Kempthorne,O.(1976).Statisticsandthephilosophers.InFoundationsof

ProbabilityTheory,StatisticalInference,andStatisticalTheoriesofScienceHarperandHooker(eds.),Vol.2,273–314.MR0488407

[14]Keynes,J.M.[1921](1952).ATreatiseonProbability.Reprint.St.Martin’s

press,NewYork.MR1113699

[15]Lehmann,E.L.(1993).TheFisherandNeyman–Pearsontheoriesoftest-inghypotheses:Onetheoryortwo?J.Amer.Statist.Assoc.88,1242–1249.MR1245356

[16]Lehmann,E.L.(1995).Neyman’sstatisticalphilosophy.Probabilityand

MathematicalStatistics15,29–36.MR1369789

[17]Mayo,D.G.(1996).ErrorandtheGrowthofExperimentalKnowledge.

UniversityofChicagoPress.

[18]Mayo,D.G.andM.Kruse(2001).Principlesofinferenceandtheircon-sequences.InFoundationsofBayesianism,D.CornﬁeldandJ.Williamson(eds.).KluwerAcademicPublishers,Netherlands,381–403.MR1889643

[19]Mayo,D.G.andSpanos,A.(2006).Severetestingasabasicconceptin

aNeyman–Pearsonphilosophyofinduction.BritishJournalofPhilosophyofScience57,323–357.MR2249183

[20]Mill,J.S.(1988).ASystemofLogic,Eighthedition.HarperandBrother,

NewYork.

[21]Morrison,D.andHenkel,R.(eds.)(1970).TheSigniﬁcanceTestContro-versy.Aldine,Chicago.

[22]Neyman,J.(1955).Theproblemofinductiveinference.Comm.Pureand

AppliedMaths8,13–46.MR0068145

[23]Neyman,J.(1957).Inductivebehaviorasabasicconceptofphilosophyof

science.Int.Statist.Rev.25,7–22.

[24]Pearson,E.S.(1955).Statisticalconceptsintheirrelationtoreality.J.R.

Statist.Soc.B17,204–207.MR0076234

[25]Pierce,C.S.[1931-5].CollectedPapers,Vols.1–6,HartshorneandWeiss,P.

(eds.).HarvardUniversityPress,Cambridge.MR0110632

[26]Popper,K.(1959).TheLogicofScientiﬁcDiscovery.BasicBooks,NewYork.

MR0107593

[27]Savage,L.J.(1964).Thefoundationsofstatisticsreconsidered.InStudies

inSubjectiveProbability,KyburgH.E.andH.E.Smokler(eds.).Wiley,NewYork,173–188.MR0179814

[28]Shaffer,J.P.(2005).Thisvolume.

[29]Spanos,A.(2000).Revisitingdatamining:‘hunting’withorwithoutalicense.

JournalofEconomicMethodology7,231–264.

[30]Whewell,W.[1847](1967).ThePhilosophyoftheInductiveSciences.

FoundedUponTheirHistory,Secondedition,Vols.1and2.Reprint.John-sonReprint,London.

[31]Will,C.(1993).TheoryandExperimentinGravitationalPhysics.Cambridge

UniversityPress.MR0778909

[32]Yates,F.(1951).TheinﬂuenceofStatisticalMethodsforResearchWorkers

onthedevelopmentofthescienceofstatistics.J.Amer.Statist.Assoc.46,19–34.

因篇幅问题不能全部显示，请点此查看更多更全内容

查看全文