Open Review Tracker
← Back to Manuscript Reviews

review rsos ospractices

Manuscript Reviews

Recommended citation (APA 7)

Röseler, L. (2023, November 13). Review: review rsos ospractices [Peer review]. Open Review Tracker. https://lroesele.zivgitlabpages.uni-muenster.de/reviews/manuscript-reviews/review-rsos-ospractices.html (accessed September 16, 2026).

I have reviewed the manuscript titled “Exploring Open Science Practices in Behavioural Public Policy Research”. The authors investigated reporting biases in an exploratory secondary analysis of evidence from two large datasets on nudging. I believe that their methods and findings are valuable for the quality assurance of nudge units’ work and that they should be published. Reading the manuscript, I learned a lot about the economics and politics of nudging and I thank the authors for that! However, I had difficulties understanding the methods, which, although complete, were written very densely and exceeded my knowledge on meta-analytical methods. I believe that I (and hopefully other readers) would benefit from more thorough descriptions of the meta-analytical models and I am certain that this would resolve my main issues. Apart from this central issue, I have numerous minor remarks listed below. I hope that they can help the authors improve the manuscript.

Central Issue

  1. I understand that this as an explorative approach wherein a specific model makes data from an important look as if it is biased. I believe that using exploratory models, bias can occur in almost any dataset. I understand the significance of the dataset used but I do not see why the mixture models are better than previous models (e.g., classical Hedges and Vevea Selection Models), why no other prominent models were used (e.g., RoBMA) and why this is not framed as a comment submitted to Econometrica and linked directly to DellaVigna and Linos’ paper. I expect that clarifying the meta-analytical models in the methods section as well in the R-script would resolve these problems. I apologize if this is obvious or if I missed something here! Frankly, I only have experience with “out of the box” meta-analytical models and recommend letting a specialist statistical reviewer check the R-code.

Remarks in no specific order

  1. This is probably just for review but the link listed in the “Author-supplied statements” did not work (https://osf.io/f3rx/). The “t” got lost somewhere and it should say https://osf.io/f3rxt/ instead.
  2. P. 3, l. 12 [referring to the “Page x of 8” numbers] “nudge units: government agencies”: Can you list which government(s) have created such agencies or collaborated with them? I think this would underline your first sentence (one of the most widespread applications”).
  3. P. 3 footnote 1 and other links: I recommend that you archive or screenshot the website as links tend to stop working or websites are altered over time. I think that the best way to do this is via the internet archive / waybackmachine (https://archive.org/web/).
  4. P. 3 “8.7 percentage points) was larger than in trials run by OES and BIT (average 1.4 percentage points).”: I believe percentages are misleading most of the times they are used and I do not understand what they mean here. Is it possible for you to provide different types of effect sizes or to elaborate on them in a footnote?
  5. P. 4 footnote: Can you please provide a more specific reference (e.g., a page number)? There are two supplements and after skimming the online appendix, I found a wealth of plots suggesting that there are no differences in publication biases so you could write something like pp. 12-16.
  6. P. 4 “This means that instead of assuming all effects to come from a single distribution as is commonly done in meta-analysis, their framework allows the effect sizes to come from two separate distributions”: I believe that it would help me a lot if explained in simple terms what the different between 2-mixture and 3-mixture is.
  7. Figure 2: The size of the bars exceeds the scale which seems a bit odd to me.
  8. File “Figure.R”: The code is not reproducible. Please add instructions of how to install the jaspGraphs package. Running install.packages("jaspGraphs") before executing the ggplot() code led to the error that there is no such package. Can it only be installed via git.install?
  9. I recommend that you cite R and all packages used including version numbers.
  10. File “analysis.R”: I found the code not well commented and suggest adding comments about what is happening there and that running everything may take an hour or two. There were approximately 1 or 2 comments per 100 rows of code. Still, despite the density in the code and the manuscript, I found the object names (e.g., “fit_3n”) to be well chosen and self-explanatory. Most importantly, I am not familiar with the mathematics behind the applied models and could thus not evaluate whether these functions were correctly programmed. I strongly recommend letting a specialist statistical reviewer check the code. In general, I think the way this is currently written would be perfectly fine for an audience that is highly experienced with statistical methods and is aware of all the operations happening in all sorts of models. I trust the editor knows their audience and whether this level of expertise can be assumed.
  11. The declaration of AI use is missing, please add it.

Notes for the editors

  1. I only checked some of the references.
  2. I did not run a reproducibility check but highly recommend the journal to do so (see Meta-Psychology for a state-of-the-art example). For example, I did not test whether the tables here (https://osf.io/cegnh) could be reproduced with the provided R-code. Note, however, that I could successfully reproduce Figure 1 and a version of Figure 2 that had different formatting.
  3. Switching between the text and references was a bit more difficult than in cases where APA recommendations are used. An alternative that would have facilitated my work would have been a working link from numbers to the reference list.
  4. I was very impressed with the detailed instructions for reviewers, thank you! There is one very easy to fix issue, though: Viewing the submission file with a pdf-viewer, I have different page numbers per page:
    1. The pdf-page number (5/10),
    2. the text number in the top right corner (4 of 8)
    3. and the black box with a number right next to the paragraphs (3).

This makes referring to specific passages unnecessarily complicated and I did not notice the differences until halfway through the review. I tried to refer to the number in the top right corner in my comments (“Page X of 8”).

Download originalWord document