Open Science interventions to improve reproducibility and replicability of research: a scoping review

Manuscript title: Open Science interventions to improve reproducibility and replicability of research: a scoping review
Invitation to Review date13.12.2024
Review submission date: 10.01.2025
Review type: original submission / revision / other

I have reviewed the manuscript “Open Science interventions to improve reproducibility and replicability of research: a scoping review”. The authors summarize evidence on the impact of numerous open science practices on the repeatability of research across different disciplines. When I was asked to review it, I had already been aware of it, read through it, and cited it in my own work. Open science and replicability are the topics that I spend most of my research time on, which is why I felt qualified to review the article. For the review, I read through the text and took brief looks at the preregistration and supplementary material.

Evaluation: I believe that this review is exactly what the field needs (apart from more evaluations of open science interventions). The method of a scoping review is a good fit for summarizing such heterogeneous articles and the results are a great help for researchers to identify gaps and plan their research. There are several comments that I would like to see the authors address before publication listed below. I hope that they can help the authors improve the manuscript.

Most importantly, I could not access the data via the link provided by the authors as it requires the creation of an account: https://knowcenterat.sharepoint.com/:x:/r/sites/ECReproducibilityprojects/Freigegebene%20Dokumente/Scoping%20review/Intervention%20Scoping%20Review/Data/Scoping%20review%20data_2024_02_20/Full_data_2024_04_02.xlsx?d=w025028e364f04a1b8ad647f89d37276b&csf=1&web=1&e=Phds9i
I do not recommend publication of research articles unless data and code are available and results are reproducible.

Recommendation: Revision

General Issues

1. Data and code should be made openly available on the OSF before submission
2. As soon as #1 is met, a reviewer should verify that the provided code leads to the reported results
3. I think you should consider updating the review or adding a section on more recent research since a lot has been happening since August 2023 as you note yourself (e.g., https://dx.doi.org/10.2139/ssrn.4790780, https://doi.org/10.1177/25152459241258945).
4. I like that you included proxy outcomes but I would have liked to see a more detailed discussion about which is which and have some part of the table in the main manuscript. Moreover, I recommend that you add a note that some interventions do not need empirical evidence and some may increase trust and transparency without affecting reproducibility (e.g., diamond open access publications). For example, I do not think that we require a study to show that:
   1. better documentation and open data could have enabled Errington et al. (2021) to repeat more than 50 out of 193 experiments (https://doi.org/10.7554/elife.67995).
   2. Indicating version numbers for software facilitate reproduction.
   3. Using open source software such as R will of course make it easier to re-run the code.
   4. Open peer review allows to check if review actually happens and is necessary to identify paper mills or review mills.

I recommend that you discuss these and other things and set them apart from interventions where we do need an empirical assessment or at least discuss this in the limitations. Currently, I am afraid that readers of your manuscript could be led to think that everything that changed through open science is a waste of time, which I personally find dangerous and to which I disagree.

1. Preregistration:
   1. I found it a bit confusing that there is a preregistration file on the OSF and also a published protocol on ORE. I recommend using only one of these tools as there are now three versions of the registration (OSF version dated 2023-07-27, ORE v1 2023-10-18, and ORE v2 2024-07-10). I assumed that the first two versions are identical and used the oldest version from the OSF to assess adherence to the preregistration.
   2. Moreover, I would have expected the preregistration to be registered via the OSF registration-function (and thus findable in the OSF-Registries at osf.io/registries). This would have been findable here (https://osf.io/7ef5h/registrations) or as a frozen version here (https://osf.io/d65ys/registrations). To be clear, I acknowledge that you did upload the protocol (https://osf.io/rhe9k) before executing the study but strongly recommend using the registration function next time. This way, changes are not possible anymore. The current version would allow you to change or delete the file. For example, the date in the file (https://osf.io/rhe9k) says 6^th May 2023 but the OSF version history indicates that this was uploaded on 2023-07-27.
2. Citing software: I very much like that you report the used software for all steps of the research process. For example, you wrote “We used Zotero reference management software” on p.7. I suggest that you add a citations and version numbers in these cases. If possible, please also indicate when you used Connected Papers, EPPI reviewer or any other programs and what versions you used. Thus, other researchers can, if they want to, reproduce your results.

Further remarks in no specific order

1. I recommend exporting the Zotero folder as a literature list and saving it in the OSF just in case the Zotero link is not working anymore (https://www.zotero.org/groups/4983756/osirisconsortium/collections/5Q3IJKZ2). Ideally, all hyperlinks should be permanent links (i.e., via internet archive or OSF registration function) although I acknowledge that this may not be possible in many cases.
2. P. 10 line 30, I think there is a “by” missing in “but they were double checked __ another author”
3. P. 12 top: How can you include one preprint and one editorial and have 85/86 articles be peer reviewed? Was the editorial peer reviewed?
4. On multiple occasions you write “cross-sectional, precluding any causal inference” (e.g., p. 26): Running the risk of embarrassing myself, I must admit that I do not understand this. If reproducibility increases over time and preregistration rate does too, aren’t there numerous potential confounders prohibiting me from concluding that there is a causal relationship between the two? On the other hand, I could assign to a group of studies that are to be conducted (e.g., student projects) that half of the studies are randomly selected to require a preregistration and the other half must not preregister and then have a next group of researchers replicate all studies. I would get a cross-sectional estimate of the effect of preregistration on replicability that would allow for causal inference just as all RCT studies do. Am I missing something here? I think my problem will resolve if you write “cross sectional, non-experimental design” or “observational”.
5. P. 26 “many … have been investigated by fewer than 10 studies, often with sub-optimal designs”: To me, this sounds like ten studies in an area are not good enough, but one could argue that we do not need 10 studies but only one well-designed study that could even test investigate multiple areas. I suggest writing this less critically (at least the number – I agree on the design).
6. I recommend revising the “Deviations to the original protocol” section. For example, it should say “deviations from”, there is a reference to “figure xx”, and I recommend using the structure from Heirene et al. (2024; https://doi.org/10.15626/MP.2021.2909): For each deviation, describe it, write how and why you deviated, and how this affected the results.
7. For future preregistrations and based on https://dx.doi.org/10.2139/ssrn.4790780, I strongly recommend including the analysis code in the preregistration.

Disclaimers

1. I did not conduct a reproducibility check. This was not possible due to the inaccessibility of data and code.
2. I did not check for deviations from the preregistrations in detail.
3. Publication of this review: To track my review activities and to be transparent about them, I have uploaded this review on OSF reviews project.

Recommendations for the editor

1. Have somebody conduct a reproducibility check (e.g., via CODECHECK, https://codecheck.org.uk).
2. Let authors create a frozen version of the OSF project to prevent data from being altered or deleted or archive all materials via the journal.
3. Transition to a model with much lower and justified APCs or Diamond Open Access (for justified publication costs see https://doi.org/10.12688/f1000research.27468.2).

PRO Initiative Statements

1. I request that the authors add [or retain] a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes. The authors should, of course, add any additional text to ensure the statement is accurate. This is the standard reviewer disclosure request endorsed by the Center for Open Science [see http://osf.io/project/hadz3]. I include it in every review.
2. I also suggest preregistering all future studies (e.g. on osf.io). Note, on osf.io, one can anonymize the document so blind reviewers can see it.
3. I only ever recommend accepting a paper if I have access to the full survey materials (in absence of exceptional circumstances). I recommend posting them for reviewers and readers if the authors have not already done so.