Aggregate and individual replication probability within an explicit model of the research process
- We study a model of the research process in which the true effect size, the replication jitter due to changes in experimental procedure, and the statistical error of effect size measurement are all normally distributed random variables. Within this model, we analyze the probability of successfully replicating an initial experimental result by obtaining either a statistically significant result in the same direction or any effect in that direction. We analyze both the probability of successfully replicating a particular experimental effect (i.e., the individual replication probability) and the average probability of successful replication across different studies within some research context (i.e., the aggregate replication probability), and we identify the conditions under which the latter can be approximated using the formulas of Killeen (2005a, 2007). We show how both of these probabilities depend on parameters of the research context that would rarely be known in practice. In addition, we show that the statistical uncertaintyWe study a model of the research process in which the true effect size, the replication jitter due to changes in experimental procedure, and the statistical error of effect size measurement are all normally distributed random variables. Within this model, we analyze the probability of successfully replicating an initial experimental result by obtaining either a statistically significant result in the same direction or any effect in that direction. We analyze both the probability of successfully replicating a particular experimental effect (i.e., the individual replication probability) and the average probability of successful replication across different studies within some research context (i.e., the aggregate replication probability), and we identify the conditions under which the latter can be approximated using the formulas of Killeen (2005a, 2007). We show how both of these probabilities depend on parameters of the research context that would rarely be known in practice. In addition, we show that the statistical uncertainty associated with the size of an initial observed effect would often prevent accurate estimation of the desired individual replication probability even if these research context parameters were known exactly. We conclude that accurate estimates of replication probability are generally unattainable.…
Author details: | Jeff Miller, Wolfgang SchwarzORCiDGND |
---|---|
DOI: | https://doi.org/10.1037/a0023347 |
ISSN: | 1082-989X |
Title of parent work (English): | Psychological methods |
Publisher: | American Psychological Association |
Place of publishing: | Washington |
Publication type: | Article |
Language: | English |
Year of first publication: | 2011 |
Publication year: | 2011 |
Release date: | 2017/03/26 |
Tag: | Bayesian inference; posterior statistical power; probability of replication; random-effects model; statistical estimation |
Volume: | 16 |
Issue: | 3 |
Number of pages: | 24 |
First page: | 337 |
Last Page: | 360 |
Organizational units: | Humanwissenschaftliche Fakultät / Strukturbereich Kognitionswissenschaften / Department Psychologie |
Peer review: | Referiert |
Institution name at the time of the publication: | Humanwissenschaftliche Fakultät / Institut für Psychologie |