TY - JOUR A1 - Matuschek, Hannes A1 - Kliegl, Reinhold A1 - Vasishth, Shravan A1 - Baayen, Harald R. A1 - Bates, Douglas T1 - Balancing Type I error and power in linear mixed models JF - Journal of memory and language N2 - Linear mixed-effects models have increasingly replaced mixed-model analyses of variance for statistical inference in factorial psycholinguistic experiments. Although LMMs have many advantages over ANOVA, like ANOVAs, setting them up for data analysis also requires some care. One simple option, when numerically possible, is to fit the full variance covariance structure of random effects (the maximal model; Barr, Levy, Scheepers & Tily, 2013), presumably to keep Type I error down to the nominal a in the presence of random effects. Although it is true that fitting a model with only random intercepts may lead to higher Type I error, fitting a maximal model also has a cost: it can lead to a significant loss of power. We demonstrate this with simulations and suggest that for typical psychological and psycholinguistic data, higher power is achieved without inflating Type I error rate if a model selection criterion is used to select a random effect structure that is supported by the data. (C) 2017 The Authors. Published by Elsevier Inc. KW - Power KW - Linear mixed effect model KW - Hypothesis testing Y1 - 2017 U6 - https://doi.org/10.1016/j.jml.2017.01.001 SN - 0749-596X SN - 1096-0821 VL - 94 SP - 305 EP - 315 PB - Elsevier CY - San Diego ER - TY - JOUR A1 - Vasishth, Shravan A1 - Nicenboim, Bruno T1 - Statistical Methods for Linguistic Research: Foundational Ideas - Part I JF - Language and linguistics compass N2 - We present the fundamental ideas underlying statistical hypothesis testing using the frequentist framework. We start with a simple example that builds up the one-sample t-test from the beginning, explaining important concepts such as the sampling distribution of the sample mean, and the iid assumption. Then, we examine the meaning of the p-value in detail and discuss several important misconceptions about what a p-value does and does not tell us. This leads to a discussion of Type I, II error and power, and Type S and M error. An important conclusion from this discussion is that one should aim to carry out appropriately powered studies. Next, we discuss two common issues that we have encountered in psycholinguistics and linguistics: running experiments until significance is reached and the ‘garden-of-forking-paths’ problem discussed by Gelman and others. The best way to use frequentist methods is to run appropriately powered studies, check model assumptions, clearly separate exploratory data analysis from planned comparisons decided upon before the study was run, and always attempt to replicate results. Y1 - 2016 U6 - https://doi.org/10.1111/lnc3.12201 SN - 1749-818X VL - 10 SP - 349 EP - 369 PB - Wiley-Blackwell CY - Hoboken ER - TY - JOUR A1 - Vasishth, Shravan A1 - Lewis, Richard L. T1 - Argument-head distance and processing complexity: Explaining both locality and antilocality effects JF - Language : journal of the Linguistic Society of America N2 - Although proximity between arguments and verbs (locality) is a relatively robust determinant of sentence-processing difficulty (Hawkins 1998, 2001, Gibson 2000), increasing argument-verb distance can also facilitate processing (Konieczny 2000). We present two self-paced reading (SPR) experiments involving Hindi that provide further evidence of antilocality, and a third SPR experiment which suggests that similarity-based interference can attenuate this distance-based facilitation. A unified explanation of interference, locality, and antilocality effects is proposed via an independently motivated theory of activation decay and retrieval interference (Anderson et al. 2004).* Y1 - 2006 U6 - https://doi.org/10.1353/lan.2006.0236 SN - 0097-8507 VL - 82 IS - 4 SP - 767 EP - 794 PB - Linguistic Society of America CY - Washington ER - TY - JOUR A1 - Sorensen, Tanner A1 - Hohenstein, Sven A1 - Vasishth, Shravan T1 - Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists JF - Tutorials in Quantitative Methods for Psychology N2 - With the arrival of the R packages nlme and lme4, linear mixed models (LMMs) have come to be widely used in experimentally-driven areas like psychology, linguistics, and cognitive science. This tutorial provides a practical introduction to fitting LMMs in a Bayesian framework using the probabilistic programming language Stan. We choose Stan (rather than WinBUGS or JAGS) because it provides an elegant and scalable framework for fitting models in most of the standard applications of LMMs. We ease the reader into fitting increasingly complex LMMs, using a two-condition repeated measures self-paced reading study. KW - Bayesian data analysis KW - linear mixed models Y1 - 2016 U6 - https://doi.org/10.20982/tqmp.12.3.p175 SN - 2292-1354 VL - 12 SP - 175 EP - 200 PB - University of Montreal, Department of Psychology CY - Montreal ER - TY - JOUR A1 - Vasishth, Shravan A1 - Nicenboim, Bruno A1 - Engelmann, Felix A1 - Burchert, Frank T1 - Computational Models of Retrieval Processes in Sentence Processing JF - Trends in Cognitive Sciences N2 - Sentence comprehension requires that the comprehender work out who did what to whom. This process has been characterized as retrieval from memory. This review summarizes the quantitative predictions and empirical coverage of the two existing computational models of retrieval and shows how the predictive performance of these two competing models can be tested against a benchmark data-set. We also show how computational modeling can help us better understand sources of variability in both unimpaired and impaired sentence comprehension. Y1 - 2019 U6 - https://doi.org/10.1016/j.tics.2019.09.003 SN - 1364-6613 SN - 1879-307X VL - 23 IS - 11 SP - 968 EP - 982 PB - Elsevier CY - London ER - TY - JOUR A1 - Metzner, Paul A1 - von der Malsburg, Titus Raban A1 - Vasishth, Shravan A1 - Roesler, Frank T1 - The Importance of Reading Naturally: Evidence From Combined Recordings of Eye Movements and Electric Brain Potentials JF - Cognitive science : a multidisciplinary journal of anthropology, artificial intelligence, education, linguistics, neuroscience, philosophy, psychology ; journal of the Cognitive Science Society KW - Reading KW - Sentence comprehension KW - ERP KW - Eye movements KW - Regressions Y1 - 2017 U6 - https://doi.org/10.1111/cogs.12384 SN - 0364-0213 SN - 1551-6709 VL - 41 SP - 1232 EP - 1263 PB - Wiley CY - Hoboken ER - TY - JOUR A1 - Baayen, Harald R. A1 - Vasishth, Shravan A1 - Kliegl, Reinhold A1 - Bates, Douglas T1 - The cave of shadows: Addressing the human factor with generalized additive mixed models JF - Journal of memory and language KW - Generalized additive mixed models KW - Within-experiment adaptation KW - Autocorrelation KW - Experimental time series KW - Confirmatory versus exploratory data analysis KW - Model selection Y1 - 2017 U6 - https://doi.org/10.1016/j.jml.2016.11.006 SN - 0749-596X SN - 1096-0821 VL - 94 SP - 206 EP - 234 PB - Elsevier CY - San Diego ER - TY - JOUR A1 - Vasishth, Shravan A1 - Nicenboim, Bruno A1 - Beckman, Mary E. A1 - Li, Fangfang A1 - Kong, Eun Jong T1 - Bayesian data analysis in the phonetic sciences BT - a tutorial introduction JF - Journal of phonetics N2 - This tutorial analyzes voice onset time (VOT) data from Dongbei (Northeastern) Mandarin Chinese and North American English to demonstrate how Bayesian linear mixed models can be fit using the programming language Stan via the R package brms. Through this case study, we demonstrate some of the advantages of the Bayesian framework: researchers can (i) flexibly define the underlying process that they believe to have generated the data; (ii) obtain direct information regarding the uncertainty about the parameter that relates the data to the theoretical question being studied; and (iii) incorporate prior knowledge into the analysis. Getting started with Bayesian modeling can be challenging, especially when one is trying to model one’s own (often unique) data. It is difficult to see how one can apply general principles described in textbooks to one’s own specific research problem. We address this barrier to using Bayesian methods by providing three detailed examples, with source code to allow easy reproducibility. The examples presented are intended to give the reader a flavor of the process of model-fitting; suggestions for further study are also provided. All data and code are available from: https://osf.io/g4zpv. KW - Bayesian data analysis KW - Linear mixed models KW - Voice onset time KW - Gender effects KW - Vowel duration Y1 - 2018 U6 - https://doi.org/10.1016/j.wocn.2018.07.008 SN - 0095-4470 VL - 71 SP - 147 EP - 161 PB - Elsevier CY - London ER - TY - JOUR A1 - Nicenboim, Bruno A1 - Roettger, Timo B. A1 - Vasishth, Shravan T1 - Using meta-analysis for evidence synthesis BT - the case of incomplete neutralization in German JF - Journal of phonetics N2 - Within quantitative phonetics, it is common practice to draw conclusions based on statistical significance alone Using incomplete neutralization of final devoicing in German as a case study, we illustrate the problems with this approach. If researchers find a significant acoustic difference between voiceless and devoiced obstruents, they conclude that neutralization is incomplete, and if they find no significant difference, they conclude that neutralization is complete. However, such strong claims regarding the existence or absence of an effect based on significant results alone can be misleading. Instead, the totality of available evidence should be brought to bear on the question. Towards this end, we synthesize the evidence from 14 studies on incomplete neutralization in German using a Bayesian random-effects meta-analysis. Our meta-analysis provides evidence in favor of incomplete neutralization. We conclude with some suggestions for improving the quality of future research on phonetic phenomena: ensure that sample sizes allow for high-precision estimates of the effect; avoid the temptation to deploy researcher degrees of freedom when analyzing data; focus on estimates of the parameter of interest and the uncertainty about that parameter; attempt to replicate effects found; and, whenever possible, make both the data and analysis available publicly. (c) 2018 Elsevier Ltd. All rights reserved. KW - Meta-analysis KW - Incomplete neutralization KW - Final devoicing KW - German KW - Bayesian data analysis Y1 - 2018 U6 - https://doi.org/10.1016/j.wocn.2018.06.001 SN - 0095-4470 VL - 70 SP - 39 EP - 55 PB - Elsevier CY - London ER - TY - JOUR A1 - Vasishth, Shravan A1 - Mertzen, Daniela A1 - Jaeger, Lena A. A1 - Gelman, Andrew T1 - The statistical significance filter leads to overoptimistic expectations of replicability JF - Journal of memory and language N2 - It is well-known in statistics (e.g., Gelman & Carlin, 2014) that treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These effects get published, leading to an overconfident belief in replicability. We demonstrate the adverse consequences of this statistical significance filter by conducting seven direct replication attempts (268 participants in total) of a recent paper (Levy & Keller, 2013). We show that the published claims are so noisy that even non-significant results are fully compatible with them. We also demonstrate the contrast between such small-sample studies and a larger-sample study; the latter generally yields a less noisy estimate but also a smaller effect magnitude, which looks less compelling but is more realistic. We reiterate several suggestions from the methodology literature for improving current practices. KW - Type M error KW - Replicability KW - Surprisal KW - Expectation KW - Locality KW - Bayesian data analysis KW - Parameter estimation Y1 - 2018 U6 - https://doi.org/10.1016/j.jml.2018.07.004 SN - 0749-596X SN - 1096-0821 VL - 103 SP - 151 EP - 175 PB - Elsevier CY - San Diego ER -