The Fragility Principle: Quantifying When Scientific Conclusions Depend on Analytical Choice

Manuscript title: The Fragility Principle: Quantifying When Scientific Conclusions Depend on Analytical Choice
Invitation to Review date: 14.8.2026
Review submission date: 16.09.2026
Review type: original submission / revision / other

I have reviewed the manuscript “The Fragility Principle: Quantifying When Scientific Conclusions Depend on Analytical Choice”. The review request from RSOS included the title and abstract, no preprint was available. Based on this data, it reminded me of discussions that I have actively participated in, namely on analytic flexibility (https://doi.org/10.31222/osf.io/v259t_v2). When reading the manuscript, I quickly noticed that it was outside of my expertise, contained lots of specialized terms that I had never heard of (probably from EEG research) and had nothing to do with the topic that I expected from the title.

In general, I had great difficulty understanding many parts (see also comments below). Many terms are never explained (e.g., OKIMO), and basic requirements such as a sample description are missing. Given that I consider RSOS a multidisciplinary outlet that publishes paper that I as a somewhat experienced meta-researcher would understand, I do not think that this article fits the journal. I apologize for having agreed to review a manuscript that turned out to be outside of my area of expertise.

Recommendation: Reject

Remarks for the authors in no specific order

1. There is a spelling error in the title of the manuscript “Scientic” instead of “Scientific” though not in the manuscript
2. I recommend that you include information about what AI technologies (provider, specific model, purpose) you used.
3. Here is a list of terms that I did not understand. It seems that some of them are common in research on EEG but they are not in the area of open science or meta-science.
   1. Nuisance-to-signal axis sounds like a tortured phrase for signal-noise ratio but maybe this is just because I am no expert on EEG methods
   2. “a geometry computable before data collection”
   3. “directional fragility index with its decision mass”
   4. “coordinated analysis studies“
   5. „multiverse displays“
   6. „A conclusion whose magnitude wobbles across defensible specications is normal science“
   7. “VR was unremarkable under permutation”
4. Why is there a Generative model B but no model A? What was the actual purpose of the study beyond “accelerometer monitoring”? What are the participant demographics? What is OKIMO?
5. Key literature on analytic flexibility and multiverse analysis is not engaged with (Simmons, Nelson & Simonsohn, 2011, 10.1177/0956797611417632; Lakens, 2026, 10.31234/osf.io/hvjxf_v1)