Can I trust this paper?

Manuscript title: Can I trust this paper?
Invitation to Review date: 24.03.2025
Review submission date: 28.03.2025 [I made minor changes on 31.03. to respect the journal’s confidentiality clause]
Review type: original submission / revision / other
Review available online at: Röseler, L. (2025). Reviews. https://doi.org/10.17605/OSF.IO/PBNW7

Dear Andrey Anikin,

I have reviewed the manuscript “Can I trust this paper?” available at https://www.cogsci.se/publications/pdf/anikin_2025_can_I_trust_this_paper.pdf (accessed on March 28, 2025). I have read through it once and did not reproduce the simulation as no code was linked in the manuscript.

I feel qualified to evaluate the preprint as I have conducted research in that area myself (e.g., https://doi.org/10.1016/j.jbusres.2023.114189 - although I understand if this is not cited due to it being paywalled). I must say, however, that I am no expert in forensic meta-science – a discipline which I find highly relevant for that. I encourage the you to seek reviews by experts such as James Heathers (see for example his introduction to forensic meta-science: https://jamesheathers.curve.space). Like he writes himself “Peer review, traditionally seen as a secure bulwark against shoddy research, is not robust enough to ensure high standards”.

While a commercial journal reached out to me about reviewing this manuscript, I declined to provide my work to a commercial journal that charges very high fees for open access (https://www.nature.com/nathumbehav/submission-guidelines/publishing-options; for context, justified APCs would lie at around US$400, https://f1000research.com/articles/10-20) and thus uploaded it to an OSF repository under a CC BY 4.0 Attribution license. I thank you for uploading their manuscript to their website and recommend to make it available via Green Open Access (e.g., a preprint server that provides DOIs) or Diamond Open Access via metaror.org.

Sincerely,
Lukas Röseler

Evaluation: You provide a narrative overview of tools to identify untrustworthy or erroneous studies. I very much like the idea of empowering researchers to recognize potential shortcomings and think that this is an important contribution to meta-science that should be published in a peer-reviewed journal. However, I think that there are a few points that could strongly increase the quality of your manuscript:

1. Publish the simulation code and let others conduct an independent reproduction of the simulation study mentioned in the manuscript.
2. Stress that these are rules of thumb, so just because a p-value is interpreted incorrectly, it does not mean that one should reach out to COPE.
3. Complement the techniques with tools that help researchers implement these heuristics. There are several communities trying to build such tools and many are already available (e.g., statcheck.io).
4. Distinguish more carefully in the text (specifically the recommendation at the end) between different types of problems (errors, QRPs, fraud).

Below I have listed all my remarks in no specific order and elaborate on the points mentioned above. I hope that my feedback helps you improve the manuscript and invite Nature Human Behavior to rely on this independent review in their assessment.

1. The citation of reference #13 is incomplete. I think it refers to this article: https://doi.org/10.3758/s13428-023-02277-0. I recommend that you add DOIs to all references.
2. P. 2 “the estimated prevalence is much higher”: I recommend also referring to this review here: https://metaror.org/kotahi/articles/18/index.html
3. P. 2: In the section about retractions, I encourage you to also cite the retractiondatabase that I assume is the basis for the cited studies and relevant here: https://retractiondatabase.org/RetractionSearch.aspx?
4. “retracted papers often continue to circulate online and to be cited”: I suggest you add a reference to this statement, e.g., https://doi.org/10.1080/08989621.2021.1886933
5. Link to retractionwatch: I recommend archiving any links that are not DOIs via the Internet Archive and also using direct links. Otherwise, the link may not work anymore in a few years.
6. “evidence that there are more retractions and overblown claims in higher-ranked journals”: maybe also add another study by Brembs on journal prestige and quality: https://doi.org/10.3389/fnhum.2018.00037
7. P. 3: “preprints can be withdrawn much more rapidly than publications in case a problem is discovered”: they can also be changed more easily and changes are usually documented unlike stealth-corrections in journals (https://doi.org/10.1002/leap.1660). Of course, researchers could still withdraw and repost the preprint.
8. “performing simple statistical integrity checks, detecting plagiarism, etc.”: I recommend also referring to Heathers’ book here: https://jamesheathers.curve.space
9. P. 4 “1600 observations per condition”: Can you please add a page number to the reference?
10. P. 5 “Major smoking guns in the analysis scripts”: I understand that idioms can make text more appealing, however, I recommend not using this one or adding a brief explanation.
11. P. 7 :”large p-value, in contrast”: I would have expected a brief mention of equivalence testing (https://doi.org/10.1177/2515245918770963). I recommend that you add it.
12. Figure 2:
    1. I recommend describing the simulation in more detail. It took me some time to understand the plot. In my opinion, ideally, every plot of a manuscript should clear for people who read the abstract.
    2. Please add a link to the simulation study so that it can be reproduced. You are writing that having no data is an indicator of untrustworthy research so I am surprised that you do not provide data yourself – but hopefully it is linked with your submission and simply missed it.
    3. “recipe for disaster”: I find this a bit too colloquial and recommend writing it more formally. This may, however, be a matter of personal taste.
13. “salami publishing“: I suggest you use the formal term instead (selective reporting, see also https://osf.io/preprints/psyarxiv/fhk98_v2) or the one coined by Fanelli, 2018 “salami slicing” (https://forrt.org/glossary/english/salami_slicing/).
14. “based on fabricated data and retracted”: I recommend that you cite the retraction notice here or at least add a page number to the reference. With the current secondary citation, it is too costly for me to check if that example is actually mentioned in the cited paper.
15. P. 8: last paragraph: You could mention heterogeneity here, which is what meta-analyses should correct for (like you mention publication bias correction).
16. P. 9: “Bahnik“ should say Bahník
17. P. 10: I would recommend researchers to first reach out to the author. Also, I recommend that you more carefully distinguish between error, QRP, and fraud. Errors can happen to anyone and QRPs can be applied unconsciously. For example, I would not expect papers to be retracted due to questionable research practices such as selective reporting.
18. Table 1: I recommend putting the full references or DOIs in the last column because scrolling back and forth from numbers is tedious.
19. I suggest adding more existing tools to your review that can researchers carrying out the checks you review:
    1. Statcheck.io
    2. Scienverse/Papercheck tools
    3. Forensic Meta-Science tools (see book from Heathers)
    4. FReD Annotator (https://forrt-replications.shinyapps.io/fred_annotator/)
    5. Many significant findings (https://doi.org/10.1177/1948550617693058)
20. I suggest adding more correlates of quality
    1. Interaction effect size power
    2. Preregistration with pre-analysis plan (https://www.journals.uchicago.edu/doi/abs/10.1086/730455)
    3. Registered report
    4. Reproducibiltiy check by journal / data editor / https://codecheck.org.uk/ Institute for Replication (I4R)
    5. Independent replication studies