Data from CREP Replication of Does Money Buy Happiness, Diener et al. (2010)

Manuscript title: Data from CREP Replication of Does Money Buy Happiness, Diener et al. (2010)
Invitation to Review date: 15.12.2025
Review submission date: 16.01.2026
Review type: original submission / revision / other

I have reviewed the manuscript Data from CREP Replication of Does Money Buy Happiness, Diener et al. (2010). I read through the entire manuscript once, took a brief look at the dataset and codebook, and ran the markdown script on my computer. I spent about 2.5 hours for the review. The authors describe aggregated data from multiple replications of the Diener et al. study and central results that did not converge with the original findings.

Evaluation: The authors provide a clear codebook, easy to navigate dataset, and an analysis script (markdown) to easily reproduce the results. I applaud the authors to their project. I find their data valuable and think that it has the potential to be published at JOPD. I point out numerous problems from the report that I recommend the authors to fix, such as inconsistent descriptions of the license, and a slightly “messy” repository and script. I think that these problems can easy be resolved and hope that the authors find them helpful in improving their manuscript – and hopefully they do not seem too nitpicky.

Recommendation: Revision

Remarks in no specific order

1. There are other replications related to money and happiness, that you may want to refer to (found via FReD Explorer, Keyword “money”):
   1. Spending Money on Others Promotes Happiness: Aknin, L. B., Dunn, E. W., Proulx, J., Lok, I., & Norton, M. I. (2020). Does spending money on others promote happiness?: A registered replication report. Journal of Personality and Social Psychology, 119(2), e15–e26.
   2. time scarcity underlies asymmetric wage rates: O’Donnell, M., Dev, A. S., Antonoplis, S., Baum, S. M., Benedetti, A. H., Brown, N. D., Carrillo, B., Choi, A. L., Connor, P., Donnelly, K., Ellwood-Lowe, M. E., Foushee, R., Jansen, R., Jarvis, S. N., Lundell-Creagh, R., Ocampo, J. M., Okafor, G. N., Azad, Z. R., Rosenblum, M., … Nelson, L. D. (2021). Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity. Proceedings of the National Academy of Sciences, 118(44), e2103313118.
   3. Replication of another finding from Myers & Diener, 1995: Ford, T.E., Lappi, S.K., & Holden, C.J. (2016). Personality, humor styles and happiness: Happy people have positive humor styles. Europe’s Journal of Psychology, 12(3), 320-337.
2. I found it particularly sad that a replication did not provide any data “because the Open Science Framework Project page would not load”. There are lots of ways to share data beyond the OSF (for an index of repositories, see re3data.org). I strongly recommend that you follow up on this study to see if maybe the project page is now loading.
3. Since the majority of people did not provide income data, I would expect a brief discussion (or link to one, e.g., did you ask for exact income or provide categories?).
4. I think that a description of the different lab’s studies could also be helpful. Maybe I missed the part, I do not understand whether all projects had the exact same study design, language, and variables or whether each laboratory did adjustments.
5. “All data, regardless of whether participants provided income values, are located in the datasheet available on our OSF page (https://osf.io/qdx7p/).” – I would prefer a direct link to the dataset or at least a revised folder structure. I first looked at the “Results” folder, which did not include a dataset. Then I opened the Methods and Materials folder but wasn’t sure whether to open meta_analysisDat.xlsx or Meta Analysis_ First Steps Spreadsheet.xlsx. For better or worse, I could not find a .csv file that you mentioned earlier.
   1. Create a folder that is called “data” and includes raw, aggregated, and processed data with clear names.
   2. Add a Codebook to the folder for all datasets. If necessary, divide study and aggregated data into different folders.
   3. Revise names (e.g., “replication_survey full export (1).pdf” looks like it is a copy of another file and you mixed underscore (_) and space ( ) in the names as well as lower and uppercase variables.
   4. Replace the instance of “.csv” with the actual file format (“.xlsx”).
6. Since most people you list in Table 1 are excluded, I suggest adding a column for N with reported income. This would also allow researchers to check if missing income data is related to site and modality.
7. “Did you experience depression during a lot of the day yesterday?” I am not an expert on these matters but I would expect that researchers asking such a question would also have to provide information for respondents potentially suffering of depression.
8. “Data cleaning was conducted using excel and RStudio”:
   1. Can you please explain what part you used Excel for? Ideally, all of it would be done in R or another code-based program and thus be reproducible.
   2. Please add links to the exact files whenever possible (e.g., the RMarkdown file for processing).
   3. I recommend that you cite all software that you used and include the versions (including versions of R packages) to credit the creators of the programs/packages and facilitate reproducibility (e.g., get the recommended citation via the R Code citation("readxl")). Getting the version is quite easy with a command such as sessionInfo(), see this thread’s top answer for a simple function to save everything: https://stackoverflow.com/questions/21967254/how-to-write-a-reader-friendly-sessioninfo-to-text-file
   4. I recommend that you cite all of the individual datasets (every OSF project provides a recommended citation that you could use for that).
   5. I recommend that you describe when you accessed the data because it is possible for OSF projects to be changed unless they are registered.
9. Publication of replication study beyond data paper: Under 2.8 you describe that there is no documented use of the data apart from a poster. I strongly recommend to publish the replication as a stand-alone article in addition to the dataset in an open journal that welcomes replications, such as Meta-Psychology (though I have to admit that I have a strong COI with respect to wanting to publish replications). With my experience from JOPD, I would not expect the regression analyses in the data paper but in a more traditional research article. At the same time, for a research article, I think a lot is missing and unclear (e.g., sample size justification such as sensitivity power analysis or a posteriori power analysis, comparison between studies and effect sizes, justification why you did not attempt a reproduction first). I think this limits the educational value.
10. I do not quite understand why you included the Vienna site as a predictor and left all others out. You wrote that there was a difference but the table row of Vienne consists of NAs. Could you please explain this decision and the missing values?
11. I like the meta_analysisDat.xlsx file. Identifying the variables and codebook after seeing which filenames were relevant felt very simple.
12. Meta_analysis_4802.Rmd:
    1. I recommend also sharing the rendered (HTML or PDF) version of the .Rmd file since you automatically create it with the script and it is slightly easier to read.
    2. I was wondering what “4802” stands for. Is this part of the filename important for the dataset?
    3. Although it is slightly entertaining, I recommend cleaning the file up a bit. For example, it says in one line “#Option 2: Do what Dr Woodward told me to do”. There are lots of comments that are not useful for me as a user of the dataset (“##graphing this another way”) and lots of places where I would like more thorough comments (e.g., why and when are you excluding high-income cases?).
    4. I would also expect a list of all needed R packages at the top of the script.
    5. Maybe (in the future) you could also name RMarkdown chunks to facilitate navigation.
    6. I ran the code on my machine and it did not produce any error. However, I found it difficult to determine where the values in Table 2 come from. For example, I found one model with basic needs and positive feelings and beta = 0.053. However, SE was 0.13802 instead of 0.053 (Table 2, row 1, col 4).
13. You did not replace the JOPD template text under 3.6. It says “The open license under which the data has been deposited (e.g. CC0).”. Please clearly determine the license here, for example by removing the “e.g.”, though under reusability and in the project metadata (https://osf.io/qdx7p/metadata/osf), you write CC BY 4.0, which is inconsistent with you listing no limits to sharing (term for sharing is attribution for CC BY: https://creativecommons.org/licenses/by/4.0/deed.en).
14. “The data is stored in a generic CSV file”: This is incorrect. It is stored as an .xlsx file.
15. “the metadata is given in a README file”: I recommend changing the file’s name that you refer to to “Readme”.

Remarks as per the JOPD Reviewer Guidelines

Finally, these are statements that I include in (almost) every review.

Disclaimers

1. I did not check for the existence of a preregistration and did not review the quality of the replication study or deviations from a potential preregistrations.
2. I did not check numerical reproducibility of the reported results.
3. Like all my reviews, I will upload this document to https://osf.io/pbnw7/files/osfstorage.

Recommendations for the journal

1. As for all empirical articles that I review, I recommend the journal to conduct a Reproducibility check.
2. I recommend the journal to let authors create a frozen version of the OSF project to prevent data from being altered or deleted or archive all materials via the journal.

Additional recommendations for replication studies

1. In the case of discrepancies, I recommend that the authors of the replication study consider contacting the authors of the original study. In the present study, I did not find the description of replication success very clear, since the significant effects were different and there was no discussion as to whether null-effects were due to lack of power, deviations of the studies, or absence of effects.
2. If you have not done so yet, please list all deviations from the original study so that readers can easily evaluate the replication closeness. Please discuss for each difference whether it should affect the results.
3. I recommend that the authors make a comment on the original study via pubpeer.com or alternative systems and include information about the replication study, materials, and outcomes.

PRO Initiative Statements [all requests are met or they do not apply]

—