Prove it or call it null: the registered protocol
The short answer. A line that goes up after some work is not proof the work did it; engines move on their own. So the product works one half of a brand’s questions, deliberately leaves the other half alone, and only credits a lift when the worked half separates from its own baseline while the untouched half stays put. The rule is written before day zero and the readouts land on fixed days. A null is a result.
Why it matters. Every tool in this category can draw a before-and-after line. Few will show you the half they did not touch.
The decision rule, verbatim from the software
Positive when the worked half's since-window interval separates upward from its own before-window AND the untouched half stays inside its interval. Null when neither half moves. Inconclusive when both halves move, or the untouched half moves more: something other than the work moved the numbers.
Readouts at day 30, 45, 60 after the split opens. The rule and the dates are printed on the experiment card inside the product on the day the split is opened, and the card cannot be edited.
The protocol
- 01The split is stratified by question type and by the current named rate, so both halves start comparable. It is proposed by the software and reviewed by a person before anything is worked.
- 02The untouched half is measured on exactly the same cadence as the worked half and is never acted on. Its movement is the noise floor for the experiment.
- 03The decision rule is fixed before day zero and printed on the card. It cannot be changed once the split is opened; a different rule means a different registration.
- 04Readouts happen on fixed days. There is no peeking, and a readout that lands inside the intervals is written up as null, with the same prominence as a positive.
- 05Both arms report counts and intervals, never a single rate. A verdict is positive, null or inconclusive; there is no fourth word.
Current registrations
Every open split, with its arms frozen at the size they had on the day it opened. Verdicts are appended here on their readout days.
| Registration | Opened | Worked / untouched | Readouts | Verdict |
|---|---|---|---|---|
| SavvyCal Q3 holdout | 27 August 2026 | 31 / 30 questions | 26 September · 11 October · 26 October | pending (running) |
What a verdict does not prove
- A positive verdict shows that the worked half moved and the untouched half did not, over this window, on these engines. It does not show which of the things done in the worked half did the moving.
- A null verdict is a result about this window and this amount of work. It is not evidence that no work could move the answers.
- Sixty days is a compromise between engine refresh cycles and a client's patience. Slower-moving surfaces may settle later than the last readout.
- The arms are halves of one brand's questions, so they share the brand's sources and reputation. A lift that spills from the worked half into the untouched half reads as inconclusive, which is the honest reading, not a failure of the design.
The noise each engine adds on its own, which the untouched half measures in practice, is quantified in The noise floor of every engine.