Refine
Has Fulltext
- no (11)
Year of publication
Language
- English (11)
Is part of the Bibliography
- yes (11)
Keywords
- Activation suppression model (2)
- RT models (2)
- Simon effect (2)
- 2AFC (1)
- Bayesian inference (1)
- Consciousness (1)
- Decision making (1)
- Delta plot (1)
- Delta plots (1)
- Libet (1)
- Neuroscience (1)
- ROC curve (1)
- compound detection or discrimination (1)
- diffusion model (1)
- display size effect (1)
- hypothesis testing (1)
- individual differences (1)
- null effects (1)
- posterior statistical power (1)
- probability of replication (1)
- probability summation (1)
- psychological models (1)
- psychometric functions (1)
- random-effects model (1)
- redundancy gain (1)
- signal prevalence (1)
- statistical estimation (1)
- strategic termination (1)
- visual search (1)
Institute
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Most psychological models are intended to describe processes that operate within each individual. In many research areas, however, models are tested by looking at results averaged across many individuals, despite the fact that such averaged results may give a misleading picture of what is true for each one. We consider this conundrum with respect to the interpretation of on-average null effects. Specifically, even though an experimental manipulation might have no effect on average across individuals, it might still have demonstrable effects-albeit in opposite directions-for many or all of the individuals tested. We discuss several examples of research questions for which it would be theoretically crucial to determine whether manipulations really have no effect at the individual level, and we present a method of testing for individual-level effects.
Dissociations between reaction times and temporal order judgments : a diffusion model approach
(2006)
A diffusion model for simple reaction time (RT) and temporal order judgment (TOJ) tasks was developed to account for a commonly observed dissociation between these 2 tasks: Most stimulus manipulations (e.g., intensity) have larger effects in RT tasks than in TOJ tasks. The model assumes that a detection criterion determines the level of sensory evidence needed to conclude that a stimulus has been presented. Analysis of the performance that would be achieved with different possible criterion settings revealed that performance was optimal with a lower criterion setting for the TOJ task than for the RT task. In addition, the model predicts that effects of stimulus manipulations should increase with the size of the detection criterion. Thus, the model suggests that commonly observed dissociations between RT and TOJ tasks may simply be due to performance optimization in the face of conflicting task demands
We describe a mathematically simple yet precise model of activation suppression that can explain the negative-going delta plots often observed in standard Simon tasks. The model postulates a race between the identification of the relevant stimulus attribute and the suppression of irrelevant location-based activation, with the irrelevant activation only having an effect if the irrelevant activation is still present at the moment when central processing of the relevant attribute starts. The model can be fitted by maximum likelihood to observed distributions of RTs in congruent and incongruent trials, and it provides good fits to two previously-reported data sets with plausible parameter values. R and MATLAB software for use with the model is provided.
We compared individual-participant and jackknife-based methods for scoring the onset latencies of event-related potential (ERP) components using a diffusion process as a model for an ERP. We studied "ramp-like" components in which the true ERP increases or decreases monotonically, except for noise. If the growth rates of such components vary across participants, the jackknife-based measure can easily have only 10%-20% as much error variance as the traditional method, and this advantage is magnified with more participants. We also studied boolean AND-shaped or "bump-like" components. Jackknifing generally yielded smaller error variances with these components too, especially when the component's peak amplitude varied across participants, but less so if the component's peak latency varied. These results help illuminate the reasons for the superiority of jackknife-based onset latency measures over traditional measures in recent simulations.
We consider the specific transformation of a Wiener process {X(t), t >= 0} in the presence of an absorbing barrier a that results when this process is "time-locked" with respect to its first passage time T-a through a criterion level a, and the evolution of X(t) is considered backwards ( retrospectively) from T-a. Formally, we study the random variables defined by Y(t) = X(T-a - t) and derive explicit results for their density and mean, and also for their asymptotic forms. We discuss how our results can aid interpretations of time series "response-locked" to their times of crossing a criterion level.
Many perceptual and cognitive tasks permit or require the integrated cooperation of specialized sensory channels, detectors, or other functionally separate units. In compound detection or discrimination tasks, 1 prominent general mechanism to model the combination of the output of different processing channels is probability summation. The classical example is the binocular summation model of Pirenne (1943), according to which a weak visual stimulus is detected if at least 1 of the 2 eyes detects this stimulus; as we review briefly, exactly the same reasoning is applied in numerous other fields. It is generally accepted that this mechanism necessarily predicts performance based on 2 (or more) channels to be superior to single channel performance, because 2 separate channels provide "2 chances" to succeed with the task. We argue that this reasoning is misleading because it neglects the increased opportunity with 2 channels not just for hits but also for false alarms and that there may well be no redundancy gain at all when performance is measured in terms of receiver operating characteristic curves. We illustrate and support these arguments with a visual detection experiment involving different spatial uncertainty conditions. Our arguments and findings have important implications for all models that, in one way or another, rest on, or incorporate, the notion of probability summation for the analysis of detection tasks, 2-alternative forced-choice tasks, and psychometric functions.
Neuroscientific studies have shown that brain activity correlated with a decision to move can be observed before a person reports being consciously aware of having made that decision (e.g., Libet, Gleason, Wright, & Pearl, 1983; Soon, Brass, Heinze, & Haynes, 2008). Given that a later event (i.e., conscious awareness) cannot cause an earlier one (i.e., decision-related brain activity), such results have been interpreted as evidence that decisions are made unconsciously (e.g., Libet, 1985). We argue that this interpretation depends upon an all-or-none view of consciousness, and we offer an alternative interpretation of the early decision-related brain activity based on models in which conscious awareness of the decision to move develops gradually up to the level of a reporting criterion. Under this interpretation, the early brain activity reflects sub-criterion levels of awareness rather than complete absence of awareness and thus does not suggest that decisions are made unconsciously.
Delta plots (DPs) graphically compare reaction time (RT) quantiles obtained under two experimental conditions. In some research areas (e.g., Simon effects), decreasing delta plots (nDPs) have consistently been found, indicating that the experimental effect is largest at low quantiles and decreases for higher quantiles. nDPs are unusual and intriguing: They imply that RT in the faster condition is more variable, a pattern predicted by few standard RT models. We describe and analyze five classes of well-established latency mechanisms that are consistent with nDPs-exhaustive processing models, correlated stage models, mixture models, cascade models, and parallel channels models-and discuss the implications of our analyses for the interpretation of DPs. DPs generally do not imply any specific processing model; therefore, it is more fruitful to start from a specific quantitative model and to compare the DP it predicts with empirical data.
Aggregate and individual replication probability within an explicit model of the research process
(2011)
We study a model of the research process in which the true effect size, the replication jitter due to changes in experimental procedure, and the statistical error of effect size measurement are all normally distributed random variables. Within this model, we analyze the probability of successfully replicating an initial experimental result by obtaining either a statistically significant result in the same direction or any effect in that direction. We analyze both the probability of successfully replicating a particular experimental effect (i.e., the individual replication probability) and the average probability of successful replication across different studies within some research context (i.e., the aggregate replication probability), and we identify the conditions under which the latter can be approximated using the formulas of Killeen (2005a, 2007). We show how both of these probabilities depend on parameters of the research context that would rarely be known in practice. In addition, we show that the statistical uncertainty associated with the size of an initial observed effect would often prevent accurate estimation of the desired individual replication probability even if these research context parameters were known exactly. We conclude that accurate estimates of replication probability are generally unattainable.