Prove it or call it null: the registered protocol
The short answer. A line that goes up after some work is not proof the work did it; engines move on their own. So the product works one half of a brand’s prompts, deliberately leaves the other half alone, and only credits a lift when the worked half separates from its own baseline while the untouched half stays put. The rule is written before day zero and the readouts land on fixed days. A null is a result.
Why it matters. Every tool in this category can draw a before-and-after line. Few will show you the half they did not touch.
The decision rule, verbatim from the software
Positive when the worked half's since-window interval separates upward from its own before-window AND the untouched half stays inside its interval. Null when neither half moves. Inconclusive when both halves move, or the untouched half moves more: something other than the work moved the numbers.
Readouts at day 30, 45, 60 after the split opens. The rule and the dates are printed on the experiment card inside the product on the day the split is opened, and the card cannot be edited.
The protocol
- 01The split is stratified by prompt type and by the current mention rate, so both halves start comparable. It is proposed by the software and reviewed by a person before anything is worked.
- 02The untouched half is measured on exactly the same cadence as the worked half, and the protocol is that nobody works it. That is a rule the operator keeps, not a lock the software holds: the product does not block a brief, a draft or a placement on a held out prompt today, and the console mentions the arms rather than fencing them. Its movement is the noise floor for the experiment, so a fix shipped on it quietly contaminates the floor the whole result rests on.
- 03The decision rule is fixed before day zero and printed on the card. It cannot be changed once the split is opened; a different rule means a different registration.
- 04Readouts are declared for fixed days before day zero, and a readout that lands inside the intervals is written up as null, with the same prominence as a positive. What the card does between those days is stated rather than implied: while an experiment runs, its verdict is recomputed on every read over a window ending today, so it can read positive on day 12 and turn by day 30. Only a concluded experiment is settled, and settling a running verdict at the declared readouts is on the Planned list of our methodology page.
- 05Both arms report counts and intervals, never a single rate. A verdict is positive, null or inconclusive; there is no fourth word.
Published readouts
None yet. No registration is published on this page, and no readout has been published here. When a split is opened for publication, its arms, its rule and its readout days go here before day zero, and its verdict goes here on the readout day, null or not.
What a verdict does not prove
- A positive verdict shows that the worked half moved and the untouched half did not, over this window, on these engines. It does not show which of the things done in the worked half did the moving.
- A null verdict is a result about this window and this amount of work. It is not evidence that no work could move the answers.
- Sixty days is a compromise between engine refresh cycles and a client's patience. Slower-moving surfaces may settle later than the last readout.
- The arms are halves of one brand's prompts, so they share the brand's sources and reputation. A lift that spills from the worked half into the untouched half reads as inconclusive, which is the honest reading, not a failure of the design.
The noise each engine adds on its own, which the untouched half measures in practice, is quantified in The noise floor of every engine.