The statistical significance filter leads to overoptimistic expectations of replicability
- It is well-known in statistics (e.g., Gelman & Carlin, 2014) that treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These effects get published, leading to an overconfident belief in replicability. We demonstrate the adverse consequences of this statistical significance filter by conducting seven direct replication attempts (268 participants in total) of a recent paper (Levy & Keller, 2013). We show that the published claims are so noisy that even non-significant results are fully compatible with them. We also demonstrate the contrast between such small-sample studies and a larger-sample study; the latter generally yields a less noisy estimate but also a smaller effect magnitude, which looks less compelling but is more realistic. We reiterate several suggestions from the methodology literature for improving current practices.
Author details: | Shravan VasishthORCiDGND, Daniela MertzenORCiD, Lena A. Jaeger, Andrew Gelman |
---|---|
DOI: | https://doi.org/10.1016/j.jml.2018.07.004 |
ISSN: | 0749-596X |
ISSN: | 1096-0821 |
Title of parent work (English): | Journal of memory and language |
Publisher: | Elsevier |
Place of publishing: | San Diego |
Publication type: | Article |
Language: | English |
Year of first publication: | 2018 |
Publication year: | 2018 |
Release date: | 2021/06/28 |
Tag: | Bayesian data analysis; Expectation; Locality; Parameter estimation; Replicability; Surprisal; Type M error |
Volume: | 103 |
Number of pages: | 25 |
First page: | 151 |
Last Page: | 175 |
Funding institution: | Volkswagen FoundationVolkswagen [89 953]; Deutsche Forschungsgemeinschaft, Collaborative Research CenterGerman Research Foundation (DFG) [(SFB) 1287]; U.S. Office of Naval ResearchOffice of Naval Research [N00014-15-1-2541] |
Organizational units: | Humanwissenschaftliche Fakultät / Strukturbereich Kognitionswissenschaften / Department Linguistik |
DDC classification: | 4 Sprache / 41 Linguistik / 410 Linguistik |
Peer review: | Referiert |
Publishing method: | Open Access / Hybrid Open-Access |
License (German): | CC-BY-NC-ND - Namensnennung, nicht kommerziell, keine Bearbeitungen 4.0 International |