The posterior probability of a null hypothesis given a statistically significant result
- When researchers carry out a null hypothesis significance test, it is tempting to assume that a statistically significant result lowers Prob(H0), the probability of the null hypothesis being true. Technically, such a statement is meaningless for various reasons: e.g., the null hypothesis does not have a probability associated with it. However, it is possible to relax certain assumptions to compute the posterior probability Prob(H0) under repeated sampling. We show in a step-by-step guide that the intuitively appealing belief, that Prob(H0) is low when significant results have been obtained under repeated sampling, is in general incorrect and depends greatly on: (a) the prior probability of the null being true; (b) type-I error rate, (c) type-II error rate, and (d) replication of a result. Through step-by-step simulations using open-source code in the R System of Statistical Computing, we show that uncertainty about the null hypothesis being true often remains high despite a significant result. To help the reader develop intuitionsWhen researchers carry out a null hypothesis significance test, it is tempting to assume that a statistically significant result lowers Prob(H0), the probability of the null hypothesis being true. Technically, such a statement is meaningless for various reasons: e.g., the null hypothesis does not have a probability associated with it. However, it is possible to relax certain assumptions to compute the posterior probability Prob(H0) under repeated sampling. We show in a step-by-step guide that the intuitively appealing belief, that Prob(H0) is low when significant results have been obtained under repeated sampling, is in general incorrect and depends greatly on: (a) the prior probability of the null being true; (b) type-I error rate, (c) type-II error rate, and (d) replication of a result. Through step-by-step simulations using open-source code in the R System of Statistical Computing, we show that uncertainty about the null hypothesis being true often remains high despite a significant result. To help the reader develop intuitions about this common misconception, we provide a Shiny app (https://danielschad.shinyapps.io/probnull/). We expect that this tutorial will help researchers better understand and judge results from null hypothesis significance tests.…
Verfasserangaben: | Daniel SchadORCiDGND, Shravan VasishthORCiDGND |
---|---|
DOI: | https://doi.org/10.20982/tqmp.18.2.p011 |
ISSN: | 1913-4126 |
ISSN: | 2292-1354 |
Titel des übergeordneten Werks (Englisch): | The quantitative methods for psychology |
Verlag: | University of Montreal, Department of Psychology |
Verlagsort: | Montreal |
Publikationstyp: | Wissenschaftlicher Artikel |
Sprache: | Englisch |
Datum der Erstveröffentlichung: | 16.04.2022 |
Erscheinungsjahr: | 2022 |
Datum der Freischaltung: | 27.11.2023 |
Freies Schlagwort / Tag: | Bayesian inference; Null hypothesis significance testing; power; statistical |
Band: | 18 |
Ausgabe: | 2 |
Seitenanzahl: | 12 |
Erste Seite: | 130 |
Letzte Seite: | 141 |
Organisationseinheiten: | Humanwissenschaftliche Fakultät / Strukturbereich Kognitionswissenschaften / Department Linguistik |
Humanwissenschaftliche Fakultät / Strukturbereich Kognitionswissenschaften / Department Psychologie | |
DDC-Klassifikation: | 1 Philosophie und Psychologie / 15 Psychologie / 150 Psychologie |
4 Sprache / 41 Linguistik / 410 Linguistik | |
Peer Review: | Referiert |
Publikationsweg: | Open Access / Gold Open-Access |
DOAJ gelistet | |
Lizenz (Deutsch): | CC-BY - Namensnennung 4.0 International |