TY - JOUR A1 - Schad, Daniel A1 - Betancourt, Michael A1 - Vasishth, Shravan T1 - Toward a principled Bayesian workflow in cognitive science JF - Psychological methods N2 - Experiments in research on memory, language, and in other areas of cognitive science are increasingly being analyzed using Bayesian methods. This has been facilitated by the development of probabilistic programming languages such as Stan, and easily accessible front-end packages such as brms. The utility of Bayesian methods, however, ultimately depends on the relevance of the Bayesian model, in particular whether or not it accurately captures the structure of the data and the data analyst's domain expertise. Even with powerful software, the analyst is responsible for verifying the utility of their model. To demonstrate this point, we introduce a principled Bayesian workflow (Betancourt, 2018) to cognitive science. Using a concrete working example, we describe basic questions one should ask about the model: prior predictive checks, computational faithfulness, model sensitivity, and posterior predictive checks. The running example for demonstrating the workflow is data on reading times with a linguistic manipulation of object versus subject relative clause sentences. This principled Bayesian workflow also demonstrates how to use domain knowledge to inform prior distributions. It provides guidelines and checks for valid data analysis, avoiding overfitting complex models to noise, and capturing relevant data structure in a probabilistic model. Given the increasing use of Bayesian methods, we aim to discuss how these methods can be properly employed to obtain robust answers to scientific questions. KW - workflow KW - prior predictive checks KW - posterior predictive checks KW - model KW - building KW - Bayesian data analysis Y1 - 2021 U6 - https://doi.org/10.1037/met0000275 SN - 1082-989X SN - 1939-1463 VL - 26 IS - 1 SP - 103 EP - 126 PB - American Psychological Association CY - Washington ER - TY - JOUR A1 - Jäger, Lena Ann A1 - Mertzen, Daniela A1 - Van Dyke, Julie A. A1 - Vasishth, Shravan T1 - Interference patterns in subject-verb agreement and reflexives revisited BT - a large-sample study JF - Journal of memory and language N2 - Cue-based retrieval theories in sentence processing predict two classes of interference effect: (i) Inhibitory interference is predicted when multiple items match a retrieval cue: cue-overloading leads to an overall slowdown in reading time; and (ii) Facilitatory interference arises when a retrieval target as well as a distractor only partially match the retrieval cues; this partial matching leads to an overall speedup in retrieval time. Inhibitory interference effects are widely observed, but facilitatory interference apparently has an exception: reflexives have been claimed to show no facilitatory interference effects. Because the claim is based on underpowered studies, we conducted a large-sample experiment that investigated both facilitatory and inhibitory interference. In contrast to previous studies, we find facilitatory interference effects in reflexives. We also present a quantitative evaluation of the cue-based retrieval model of Engelmann, Jager, and Vasishth (2019). KW - Sentence processing KW - Cue-based retrieval KW - Similarity-based interference KW - Reflexives KW - Agreement KW - Bayesian data analysis KW - Replication Y1 - 2020 U6 - https://doi.org/10.1016/j.jml.2019.104063 SN - 0749-596X SN - 1096-0821 VL - 111 PB - Elsevier CY - San Diego ER - TY - JOUR A1 - Nicenboim, Bruno A1 - Roettger, Timo B. A1 - Vasishth, Shravan T1 - Using meta-analysis for evidence synthesis BT - the case of incomplete neutralization in German JF - Journal of phonetics N2 - Within quantitative phonetics, it is common practice to draw conclusions based on statistical significance alone Using incomplete neutralization of final devoicing in German as a case study, we illustrate the problems with this approach. If researchers find a significant acoustic difference between voiceless and devoiced obstruents, they conclude that neutralization is incomplete, and if they find no significant difference, they conclude that neutralization is complete. However, such strong claims regarding the existence or absence of an effect based on significant results alone can be misleading. Instead, the totality of available evidence should be brought to bear on the question. Towards this end, we synthesize the evidence from 14 studies on incomplete neutralization in German using a Bayesian random-effects meta-analysis. Our meta-analysis provides evidence in favor of incomplete neutralization. We conclude with some suggestions for improving the quality of future research on phonetic phenomena: ensure that sample sizes allow for high-precision estimates of the effect; avoid the temptation to deploy researcher degrees of freedom when analyzing data; focus on estimates of the parameter of interest and the uncertainty about that parameter; attempt to replicate effects found; and, whenever possible, make both the data and analysis available publicly. (c) 2018 Elsevier Ltd. All rights reserved. KW - Meta-analysis KW - Incomplete neutralization KW - Final devoicing KW - German KW - Bayesian data analysis Y1 - 2018 U6 - https://doi.org/10.1016/j.wocn.2018.06.001 SN - 0095-4470 VL - 70 SP - 39 EP - 55 PB - Elsevier CY - London ER - TY - JOUR A1 - Vasishth, Shravan A1 - Nicenboim, Bruno A1 - Beckman, Mary E. A1 - Li, Fangfang A1 - Kong, Eun Jong T1 - Bayesian data analysis in the phonetic sciences BT - a tutorial introduction JF - Journal of phonetics N2 - This tutorial analyzes voice onset time (VOT) data from Dongbei (Northeastern) Mandarin Chinese and North American English to demonstrate how Bayesian linear mixed models can be fit using the programming language Stan via the R package brms. Through this case study, we demonstrate some of the advantages of the Bayesian framework: researchers can (i) flexibly define the underlying process that they believe to have generated the data; (ii) obtain direct information regarding the uncertainty about the parameter that relates the data to the theoretical question being studied; and (iii) incorporate prior knowledge into the analysis. Getting started with Bayesian modeling can be challenging, especially when one is trying to model one’s own (often unique) data. It is difficult to see how one can apply general principles described in textbooks to one’s own specific research problem. We address this barrier to using Bayesian methods by providing three detailed examples, with source code to allow easy reproducibility. The examples presented are intended to give the reader a flavor of the process of model-fitting; suggestions for further study are also provided. All data and code are available from: https://osf.io/g4zpv. KW - Bayesian data analysis KW - Linear mixed models KW - Voice onset time KW - Gender effects KW - Vowel duration Y1 - 2018 U6 - https://doi.org/10.1016/j.wocn.2018.07.008 SN - 0095-4470 VL - 71 SP - 147 EP - 161 PB - Elsevier CY - London ER - TY - JOUR A1 - Vasishth, Shravan A1 - Mertzen, Daniela A1 - Jaeger, Lena A. A1 - Gelman, Andrew T1 - The statistical significance filter leads to overoptimistic expectations of replicability JF - Journal of memory and language N2 - It is well-known in statistics (e.g., Gelman & Carlin, 2014) that treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These effects get published, leading to an overconfident belief in replicability. We demonstrate the adverse consequences of this statistical significance filter by conducting seven direct replication attempts (268 participants in total) of a recent paper (Levy & Keller, 2013). We show that the published claims are so noisy that even non-significant results are fully compatible with them. We also demonstrate the contrast between such small-sample studies and a larger-sample study; the latter generally yields a less noisy estimate but also a smaller effect magnitude, which looks less compelling but is more realistic. We reiterate several suggestions from the methodology literature for improving current practices. KW - Type M error KW - Replicability KW - Surprisal KW - Expectation KW - Locality KW - Bayesian data analysis KW - Parameter estimation Y1 - 2018 U6 - https://doi.org/10.1016/j.jml.2018.07.004 SN - 0749-596X SN - 1096-0821 VL - 103 SP - 151 EP - 175 PB - Elsevier CY - San Diego ER - TY - JOUR A1 - Sorensen, Tanner A1 - Hohenstein, Sven A1 - Vasishth, Shravan T1 - Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists JF - Tutorials in Quantitative Methods for Psychology N2 - With the arrival of the R packages nlme and lme4, linear mixed models (LMMs) have come to be widely used in experimentally-driven areas like psychology, linguistics, and cognitive science. This tutorial provides a practical introduction to fitting LMMs in a Bayesian framework using the probabilistic programming language Stan. We choose Stan (rather than WinBUGS or JAGS) because it provides an elegant and scalable framework for fitting models in most of the standard applications of LMMs. We ease the reader into fitting increasingly complex LMMs, using a two-condition repeated measures self-paced reading study. KW - Bayesian data analysis KW - linear mixed models Y1 - 2016 U6 - https://doi.org/10.20982/tqmp.12.3.p175 SN - 2292-1354 VL - 12 SP - 175 EP - 200 PB - University of Montreal, Department of Psychology CY - Montreal ER -