Reality Television Shows Focusing on Sexual Relationships and College Students Engagement in One-Night Stands: A Replication Report

Manuscript title: Reality Television Shows Focusing on Sexual Relationships and College Students Engagement in One-Night Stands: A Replication Report
Invitation to Review date: 07.05.2025
Review submission date: 21.05.2025
Review type: original submission / revision / other

I have reviewed the manuscript “Reality Television Shows Focusing on Sexual Relationships and College Students Engagement in One-Night Stands: A Replication Report”. The authors conducted a close replication with extension of a cross-sectional study on self-reported media consumption (specifically reality TV shows) and self-reported sexual behavior.

Evaluation: I find the authors’ report very transparent and comprehensive and think that it generally matches the scope of Meta-Psychology. However, there are several issues that I would like the authors to address: Most importantly, there is no statistical comparisons between the two studies apart from if both effects were significant or not. The preregistration is problematic: While I acknowledge that the present preregistration matches the standard in psychology, it is far from ideal in that it lacked details in several places, it was not registered before data collection, and no deviations were discussed. Also, I did not understand the decision to look at dating versus sexual reality TV shows separately. Moreover, the authors did not share their data but only Jamovi outputs and a codebook.

Finally, given that this research is directly linked to potential public health interventions I strongly advise the editor and authors to carefully think about communication about the original and replication finding: While I think that the current report could be used by policymakers to propose limiting reality TV show viewing, I do not think that it should be. The authors should carefully discuss policy implications in the discussion sections and highlight the non-causal relationship of their findings.

Below, I listed all my remarks. I hope that they can help the authors improve their manuscript.

Thank you for inviting me to review this exciting replication study!
Sincerely,
Lukas Röseler

Recommendation: Revision

Remarks in no specific order

1. Please also discuss the societal relevance of the topic and address potential misinterpretations. I highly recommend clearer communication about associative and non-causal relationship and a more extensive discussion of the original “argument for public health interventions to limit reality TV viewership”.
2. p. 7: I recommend that you cite Qualtrics as software and a direct link to the preregistration (https://osf.io/sg7au).
3. P. 12: I also recommend citing Jamovi and including the Jamovi version and all packages with versions. You can save these by running the following function in R (and upload them to the OSF): writeLines(capture.output(sessionInfo()), "sessionInfo.txt").
4. Preregistration timestamp: The preregistration has the timestamp April 11, 2022 and you wrote that data collection occurred from March to April. Can you please explain why you did not preregister the study before data collection, how much data was collected by April 11, and if you already had looked at parts of the data by the time of the preregistration?
5. Gift card draw: For researchers to be able to evaluate the incentive, I suggest that you explain how many gift cards there were, what their values were, and what they were for. For example, different people could have been motivated to participate in your study depending on if the gift card was for a store where you can buy contraceptives versus books.
6. The preregistration lacks specificity with respect to:
   1. the statistical procedures: Hypothesis I speaks of percentages but in your power analysis you only report a Cohen’s d. Then in the section below you report ANOVA and Mann-Whitney test, which do not match with any of the two statistical values mentioned before. It is also unclear why you oversampled and why you chose the medium effect instead of the actual effect from the original study. I strongly recommend preregistering the entire analysis script next time to anticipate these things and effectively deprive yourself from numerous degrees of freedom.
   2. data coding (what counts as a dating reality TV show?). I recommend that you report an exploratory analysis where you drop this distinction.
   3. Sample size: You wrote “at least 600”, which does not rule out the QRP “optional stopping”. I recommend that you run your analyses with the first 600 participants and report them as an exploratory robustness check.
   4. Exclusions: You report on p. 13 that you excluded participants who did not provide demographic data. This is a clear deviation from your preregistration and you should discuss all deviations.
7. I recommend exporting the Jamovi code so that your analyses can be reproduced. In the “Data and Result” folder I could not find an R script. Here is a guide on how to do that: https://docs.jamovi.org/_pages/jmv_overview.html#:~:text=Use%20of%20jamovi%20syntax%20in%20R&text=Set%20a%20tick%20at%20syntax,export%20or%20copy%20the%20syntax.
8. Figure 1 should include more information: 95% confidence intervals or one-sided Cis as your hypothesis is one-tailed; a y-axis label (I guess these are minutes?), a note with the group sample sizes; possibly an indicator of which differences were significant.
9. P. 12: “All p-values were two-tailed”: I think it is a common misconception that ANOVA p-values can be two-tailed. As it is based on an asymmetric distribution (F) and is only testing for variance, ANOVA’s are always one-tailed tests. Moreover, please report eta² for ANOVAs (including 90% Cis). For guidance, I recommend this guide (full discloruse: I am a co-author on it): https://matthewbjane.quarto.pub If you prefer not to cite me, there is also an informative book by Lakens: https://lakens.github.io/statistical_inferences/07-CI.html
10. P. 13: Please also report the sample size before the application of exclusion criteria.
11. Provided that you made sure to have the rights to publish the data, please share raw and processed anonymized datasets via your OSF project and assign a license. I could only find Jamovi output files in the “Data and Results” folder.
12. P. 13: Please report together with the Chi squared tests the cross-tables. Also, I am a bit uncertain about the values in parentheses. Maybe clarify this by indicating that the second one is the sample size (e.g., Chi²(18, N = 686) = 48.63)?
13. P. 14: “U tests were performed on skewed variables .. used a value of +/- 2”: please also discuss that this was not part of the preregistration and possibly run exploratory robustness checks to see how different cut-off values affect the results. For guidance regarding deviations from the preregistration, see https://doi.org/10.1525/collabra.117094 or https://doi.org/10.15626/MP.2021.2909.
14. P. 15: Please also report the effect size found in the original study. You should be able to compute Phi coefficients for both studies including confidence intervals and compare them using a criterion that makes sense in your case (see for example https://forrt.org/FReD/articles/success_criteria.html for an overview by a colleague and myself or https://lakens.github.io/statistical_inferences/17-replication.html#analyzing-replication-studies. by Lakens).
15. Your hypotheses 1 and 2 would, together, form an interaction hypothesis – but you do not test it. Moreover, just because the dating hypothesis is not significant with p = .072, does not mean that it is wrong or that the effect is significantly smaller than the one for H1. I am not familiar with equivalence tests for Chi-Squared but maybe you could compute a 3 dimensional chi-squared test here or compare the Phi coefficients to see if the difference between p < .001 and p = .072 is something that you can interpret in favor of your hypotheses. In a nutshell: When you use statistics, absence of evidence is not evidence of absence.
16. P. 16: “reproducing” would mean that you have their data (and maybe their code) and check if you get their results. I think that this should say “recreate”. If possible, I recommend that you request the original data from the original authors and try to reproduce their findings.
17. In general, I recommend reporting effect sizes together with confidence intervals (see also APA’s journal article reporting standards). Also, “r = -.01, ns”: Please adhere to APA 7 in that you always report 3 digits of p-values. This should look something like r(df) = .xx, p = .xxx, 95% CI [.xxx, .xxx]. For t-tests, please report Cohen’s d (with CI).
18. I like the transparency with the additional variables. I think Table 4 would be more informative if you juxtaposed original and replication correlations.
19. P. 20 “we felt it was important”: This is nitpicky but I generally favor rational arguments over feeling when justifying analyses.
20. I could not open the “Ethics Approval” file, can you please check what file type it is (e.g., .pdf)? https://osf.io/enp3t

Statements and checks that I include in every review:

Methodological Checks (see also https://www.opennessinitiative.org/the-initiative/)

1. Preregistration
   1. Existence: https://osf.io/hyzkv
   2. Specificity: medium (see comments above)
   3. Inclusion of Analysis Plan (i.e., R-code): no
   4. Discussion of all deviations: no
2. Open Data
   1. Existence: no
   2. Code sheet: osf.io/594wb/
   3. Raw AND Processed data: no
3. Open Code
   1. Existence: no
   2. Open Source Software: yes (Jamovi)
   3. Version control or disclosure of versions: no
   4. Well-commented: not applicable
4. Results
   1. Statcheck results: [reproducibility check will be conducted by journal]
   2. Indicators of QRPs: possible optional stopping (preregistered N = 600, actual N = 680; discrepancy not yet explained), possible HARKing (preregistration did not precede data collection, hypothesis intransparently distinguishes between two categories)
5. Open Materials
   1. Analysis code: does not apply
   2. Stimuli or questionnaires: all listed very transparently
   3. Open source software: does not apply
6. 21 word solution or anything similar (e.g., file-drawer statement): no
7. Replicability
   1. FReD results: none for “reality” or “TV”
8. Pre-print: public as per MP’s submission policy

Disclaimers

1. I did not conduct a reproducibility check.
2. I did not check for deviations from the preregistrations in detail.
3. Publication of this review: Like all of my reviews, I have published this review on my OSF reviews project. You can access it here: https://osf.io/hvbqz

Additional recommendations for replication studies

1. If possible, I advise the authors to conduct a reproduction based on the original study’s data (see https://bitss.github.io/ACRE/ for guidance).
2. In the case of discrepancies, I recommend that the authors of the replication study contact the authors of the original study. Due to the replication success, I do not think that much insight will be gained from this procedure here.
3. If you have not done so yet, please list all deviations from the original study so that readers can easily evaluate the replication closeness. If possible, please discuss for each difference whether it affects the results.

PRO Initiative Statements

1. I request that the authors add [or retain] a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes. The authors should, of course, add any additional text to ensure the statement is accurate. This is the standard reviewer disclosure request endorsed by the Center for Open Science [see http://osf.io/project/hadz3]. I include it in every review.
2. I also suggest preregistering all future studies (e.g. on osf.io). Note, on osf.io, one can anonymize the document so blind reviewers can see it.
3. I only ever recommend accepting a paper if I have access to the full survey materials (in absence of exceptional circumstances). I recommend posting them for reviewers and readers if the authors have not already done so.