Open Review Tracker

Review Template

This is the template I use for most of my reviews. Shared under CC0 1.0: reuse and adapt it freely.

Manuscript title: _____
Invitation to Review date: xx.xx.xxxx
Review submission date: xx.xx.xxxx
Review type: original submission / revision / other

I have reviewed the manuscript “_____”. The authors ______.

Evaluation: _____

Recommendation: _____

General Issues

Remarks in no specific order

Methodological Checks (see also https://www.opennessinitiative.org/the-initiative/)

  1. Preregistration
    1. Existence:
    2. Specificity:
    3. Inclusion of Analysis Plan (i.e., R-code):
    4. Discussion of all deviations:
  2. Open Data
    1. Existence:
    2. Code sheet:
    3. Raw AND Processed data:
  3. Open Code
    1. Existence:
    2. Open Source Software:
    3. Version control or disclosure of versions:
    4. Well-commented:
  4. Results
    1. Statcheck results:
    2. Indicators of QRPs:
  5. Open Materials
    1. Analysis code
    2. Stimuli or questionnaires
    3. Open source software
  6. 21 word solution or anything similar (e.g., file-drawer statement)
  7. Replicability
    1. FReD Annotator results:
  8. Pre-print
    1. Should be published if allowed by the journal and if the original article is not published as diamond open access (see https://v2.sherpa.ac.uk/romeo/search.html for guidance)

Disclaimers

  1. I did not conduct a reproducibility check
  2. I did not check for deviations from the preregistrations in detail
  3. [Have I collaborated with the author? If so, describe project, if there was a close collaboration, consider declining to review]
    1. Publication of this review: I have published this review at [hypothes.is, ASAPbio, on OSF project of mine, or Pubpeer]. You can access it here [Link to public version]

Recommendations for the journal

  1. Reproducibility check
  2. Statistical expert necessary?
  3. Let authors create a frozen version of the OSF project to prevent data from being altered or deleted or archive all materials via the journal

Additional recommendations for replication studies

  1. In the case of discrepancies, I recommend that the authors of the replication study contact the authors of the original study. Due to the replication success I do not think that any insight will be gained from this procedure here.
  2. If you have not done so yet, please list all deviations from the original study so that readers can easily evaluate the replication closeness. Please discuss for each difference whether it should affect the results.
  3. I recommend that the authors make a comment on the original study via pubpeer.com or alternative systems and include information about the replication study, materials, and outcomes.
  4. Ideally, journals that published original studies should also publish corresponding replications (pottery barn rule, https://thehardestscience.com/2012/09/27/a-pottery-barn-rule-for-scientific-journals/). If the authors have already submitted the manuscript to the original journal and it was rejected, I (as a reader) would like to know about the reasons. If they did not submit the study to the original study’s journal, I would like to know why they decided against this.