TY  - JOUR
A1  - Schad, Daniel
A1  - Vasishth, Shravan
T1  - The posterior probability of a null hypothesis given a statistically significant result
T2  - The quantitative methods for psychology
N2  - When researchers carry out a null hypothesis significance test, it is tempting to assume that a statistically significant result lowers Prob(H0), the probability of the null hypothesis being true. Technically, such a statement is meaningless for various reasons: e.g., the null hypothesis does not have a probability associated with it. However, it is possible to relax certain assumptions to compute the posterior probability Prob(H0) under repeated sampling. We show in a step-by-step guide that the intuitively appealing belief, that Prob(H0) is low when significant results have been obtained under repeated sampling, is in general incorrect and depends greatly on: (a) the prior probability of the null being true; (b) type-I error rate, (c) type-II error rate, and (d) replication of a result. Through step-by-step simulations using open-source code in the R System of Statistical Computing, we show that uncertainty about the null hypothesis being true often remains high despite a significant result. To help the reader develop intuitions about this common misconception, we provide a Shiny app (https://danielschad.shinyapps.io/probnull/). We expect that this tutorial will help researchers better understand and judge results from null hypothesis significance tests.
KW  - Null hypothesis significance testing
KW  - Bayesian inference
KW  - statistical
KW  - power
Y1  - 2022
UR  - https://publishup.uni-potsdam.de/frontdoor/index/index/docId/61572
SN  - 1913-4126
SN  - 2292-1354
VL  - 18
IS  - 2
SP  - 130
EP  - 141
PB  - University of Montreal, Department of Psychology
CY  - Montreal
ER  -