TY  - JOUR
A1  - Vasishth, Shravan
A1  - Mertzen, Daniela
A1  - Jaeger, Lena A.
A1  - Gelman, Andrew
T1  - The statistical significance filter leads to overoptimistic expectations of replicability
JF  - Journal of memory and language
N2  - It is well-known in statistics (e.g., Gelman & Carlin, 2014) that treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These effects get published, leading to an overconfident belief in replicability. We demonstrate the adverse consequences of this statistical significance filter by conducting seven direct replication attempts (268 participants in total) of a recent paper (Levy & Keller, 2013). We show that the published claims are so noisy that even non-significant results are fully compatible with them. We also demonstrate the contrast between such small-sample studies and a larger-sample study; the latter generally yields a less noisy estimate but also a smaller effect magnitude, which looks less compelling but is more realistic. We reiterate several suggestions from the methodology literature for improving current practices.
KW  - Type M error
KW  - Replicability
KW  - Surprisal
KW  - Expectation
KW  - Locality
KW  - Bayesian data analysis
KW  - Parameter estimation
Y1  - 2018
U6  - https://doi.org/10.1016/j.jml.2018.07.004
SN  - 0749-596X
SN  - 1096-0821
VL  - 103
SP  - 151
EP  - 175
PB  - Elsevier
CY  - San Diego
ER  - 
TY  - JOUR
A1  - Jäger, Lena Ann
A1  - Chen, Zhong
A1  - Li, Qiang
A1  - Lin, Chien-Jer Charles
A1  - Vasishth, Shravan
T1  - The subject-relative advantage in Chinese: Evidence for expectation-based processing
JF  - Journal of memory and language
N2  - Chinese relative clauses are an important test case for pitting the predictions of expectation-based accounts against those of memory-based theories. The memory-based accounts predict that object relatives are easier to process than subject relatives because, in object relatives, the distance between the relative clause verb and the head noun is shorter. By contrast, expectation-based accounts such as surprisal predict that the less frequent object relative should be harder to process. In previous studies on Chinese relative clause comprehension, local ambiguities may have rendered a comparison between relative clause types uninterpretable. We designed experimental materials in which no local ambiguities confound the comparison. We ran two experiments (self-paced reading and eye-tracking) to compare reading difficulty in subject and object relatives which were placed either in subject or object modifying position. The evidence from our studies is consistent with the predictions of expectation-based accounts but not with those of memory-based theories. (C) 2014 Elsevier Inc. All rights reserved.
KW  - Sentence processing
KW  - Relative clause
KW  - Structural expectation
KW  - Working-memory
KW  - Surprisal
KW  - Chinese
Y1  - 2015
U6  - https://doi.org/10.1016/j.jml.2014.10.005
SN  - 0749-596X
SN  - 1096-0821
VL  - 79
SP  - 97
EP  - 120
PB  - Elsevier
CY  - San Diego
ER  - 
TY  - JOUR
A1  - Engelmann, Felix
A1  - Vasishth, Shravan
A1  - Engbert, Ralf
A1  - Kliegl, Reinhold
T1  - A framework for modeling the interaction of syntactic processing and eye movement control
JF  - Topics in cognitive science
N2  - We explore the interaction between oculomotor control and language comprehension on the sentence level using two well-tested computational accounts of parsing difficulty. Previous work (Boston, Hale, Vasishth, & Kliegl, 2011) has shown that surprisal (Hale, 2001; Levy, 2008) and cue-based memory retrieval (Lewis & Vasishth, 2005) are significant and complementary predictors of reading time in an eyetracking corpus. It remains an open question how the sentence processor interacts with oculomotor control. Using a simple linking hypothesis proposed in Reichle, Warren, and McConnell (2009), we integrated both measures with the eye movement model EMMA (Salvucci, 2001) inside the cognitive architecture ACT-R (Anderson et al., 2004). We built a reading model that could initiate short Time Out regressions (Mitchell, Shen, Green, & Hodgson, 2008) that compensate for slow postlexical processing. This simple interaction enabled the model to predict the re-reading of words based on parsing difficulty. The model was evaluated in different configurations on the prediction of frequency effects on the Potsdam Sentence Corpus. The extension of EMMA with postlexical processing improved its predictions and reproduced re-reading rates and durations with a reasonable fit to the data. This demonstration, based on simple and independently motivated assumptions, serves as a foundational step toward a precise investigation of the interaction between high-level language processing and eye movement control.
KW  - Sentence comprehension
KW  - Eye movements
KW  - Reading
KW  - Parsing difficulty
KW  - Working memory
KW  - Surprisal
KW  - Computational modeling
Y1  - 2013
U6  - https://doi.org/10.1111/tops.12026
SN  - 1756-8757
VL  - 5
IS  - 3
SP  - 452
EP  - 474
PB  - Wiley-Blackwell
CY  - Hoboken
ER  -