Accounting for stimulus and participant effects in event-related potential analyses to increase the replicability of studies
- Background: Event-related potentials (ERPs) are increasingly used in cognitive science. With their high temporal resolution, they offer a unique window into cognitive processes and their time course. In this paper, we focus on ERP experiments whose designs involve selecting participants and stimuli amongst many. Recently, Westfall et al. (2017) highlighted the drastic consequences of not considering stimuli as a random variable in fMRI studies with such designs. Most ERP studies in cognitive psychology suffer from the same drawback. New method: We advocate the use of the Quasi-F or Mixed-effects models instead of the classical ANOVA/by-participant F1 statistic to analyze ERP datasets in which the dependent variable is reduced to one measure per trial (e.g., mean amplitude). We combine Quasi-F statistic and cluster mass tests to analyze datasets with multiple measures per trial. Doing so allows us to treat stimulus as a random variable while correcting for multiple comparisons. Results: Simulations show that the use of Quasi-FBackground: Event-related potentials (ERPs) are increasingly used in cognitive science. With their high temporal resolution, they offer a unique window into cognitive processes and their time course. In this paper, we focus on ERP experiments whose designs involve selecting participants and stimuli amongst many. Recently, Westfall et al. (2017) highlighted the drastic consequences of not considering stimuli as a random variable in fMRI studies with such designs. Most ERP studies in cognitive psychology suffer from the same drawback. New method: We advocate the use of the Quasi-F or Mixed-effects models instead of the classical ANOVA/by-participant F1 statistic to analyze ERP datasets in which the dependent variable is reduced to one measure per trial (e.g., mean amplitude). We combine Quasi-F statistic and cluster mass tests to analyze datasets with multiple measures per trial. Doing so allows us to treat stimulus as a random variable while correcting for multiple comparisons. Results: Simulations show that the use of Quasi-F statistics with cluster mass tests allows maintaining the family wise error rates close to the nominal alpha level of 0.05. Comparison with existing methods: Simulations reveal that the classical ANOVA/F1 approach has an alarming FWER, demonstrating the superiority of models that treat both participant and stimulus as random variables, like the Quasi-F approach. Conclusions: Our simulations question the validity of studies in which stimulus is not treated as a random variable. Failure to change the current standards feeds the replicability crisis.…
Author details: | Audrey Damaris Bürki-FoschiniORCiDGND, Jaromil Frossard, Olivier RenaudORCiD |
---|---|
DOI: | https://doi.org/10.1016/j.jneumeth.2018.09.016 |
ISSN: | 0165-0270 |
ISSN: | 1872-678X |
Pubmed ID: | https://pubmed.ncbi.nlm.nih.gov/30232038 |
Title of parent work (English): | Journal of neuroscience methods |
Publisher: | Elsevier |
Place of publishing: | Amsterdam |
Publication type: | Article |
Language: | English |
Date of first publication: | 2018/09/16 |
Publication year: | 2018 |
Release date: | 2021/07/19 |
Tag: | Cluster mass; ERP; Mixed-effects model; Quasi-F; Replicability crisis; Stimulus as fixed-effect fallacy |
Volume: | 309 |
Number of pages: | 10 |
First page: | 218 |
Last Page: | 227 |
Funding institution: | Deutsche Forschungsgemeinschaft (DFG)German Research Foundation (DFG) [SFB 1287] |
Organizational units: | Humanwissenschaftliche Fakultät / Strukturbereich Kognitionswissenschaften / Department Linguistik |
DDC classification: | 4 Sprache / 41 Linguistik / 410 Linguistik |
Peer review: | Referiert |