Share the code, not just the data
- In 2019 the Journal of Memory and Language instituted an open data and code policy; this policy requires that, as a rule, code and data be released at the latest upon publication. How effective is this policy? We compared 59 papers published before, and 59 papers published after, the policy took effect. After the policy was in place, the rate of data sharing increased by more than 50%. We further looked at whether papers published under the open data policy were reproducible, in the sense that the published results should be possible to regenerate given the data, and given the code, when code was provided. For 8 out of the 59 papers, data sets were inaccessible. The reproducibility rate ranged from 34% to 56%, depending on the reproducibility criteria. The strongest predictor of whether an attempt to reproduce would be successful is the presence of the analysis code: it increases the probability of reproducing reported results by almost 40%. We propose two simple steps that can increase the reproducibility of published papers: shareIn 2019 the Journal of Memory and Language instituted an open data and code policy; this policy requires that, as a rule, code and data be released at the latest upon publication. How effective is this policy? We compared 59 papers published before, and 59 papers published after, the policy took effect. After the policy was in place, the rate of data sharing increased by more than 50%. We further looked at whether papers published under the open data policy were reproducible, in the sense that the published results should be possible to regenerate given the data, and given the code, when code was provided. For 8 out of the 59 papers, data sets were inaccessible. The reproducibility rate ranged from 34% to 56%, depending on the reproducibility criteria. The strongest predictor of whether an attempt to reproduce would be successful is the presence of the analysis code: it increases the probability of reproducing reported results by almost 40%. We propose two simple steps that can increase the reproducibility of published papers: share the analysis code, and attempt to reproduce one's own analysis using only the shared materials.…
Author details: | Anna LaurinavichyuteORCiDGND, Himanshu YadavORCiD, Shravan VasishthORCiDGND |
---|---|
DOI: | https://doi.org/10.1016/j.jml.2022.104332 |
ISSN: | 0749-596X |
ISSN: | 1096-0821 |
Title of parent work (English): | Journal of memory and language |
Subtitle (English): | a case study of the reproducibility of articles published in the Journal of Memory and Language under the open data policy |
Publisher: | Elsevier |
Place of publishing: | San Diego |
Publication type: | Article |
Language: | English |
Date of first publication: | 2022/06/02 |
Publication year: | 2022 |
Release date: | 2022/11/04 |
Tag: | Journal policy; Meta-research; Open; Open data; Reproducibility; Reproducible statistical analyses; science |
Volume: | 125 |
Article number: | 104332 |
Number of pages: | 12 |
Funding institution: | Deutsche Forschungsgemeinschaft (DFG, German Research Foundation); [317633480, SFB 1287] |
Organizational units: | Humanwissenschaftliche Fakultät / Strukturbereich Kognitionswissenschaften / Department Linguistik |
DDC classification: | 1 Philosophie und Psychologie / 15 Psychologie / 150 Psychologie |
3 Sozialwissenschaften / 37 Bildung und Erziehung / 370 Bildung und Erziehung | |
4 Sprache / 40 Sprache / 400 Sprache | |
Peer review: | Referiert |
Publishing method: | Open Access / Hybrid Open-Access |
License (German): | CC-BY - Namensnennung 4.0 International |