A flip rate is not a stability metric

Why the raw number cannot be read

The reason repeated measurement matters in AI search is well established and we are not disputing it. Answers are generated rather than retrieved, so a single observation is a draw from a distribution. Our own reporting contract has required repeated runs, denominators and intervals since we started.

The consecutive flip rate has become the convenient headline for that instability: count how often a mention changes between one run and the next, and report the share. It is easy to compute and intuitive to read. It is also, on its own, not interpretable, because the null model is not zero.

The function is not monotonic in the way intuition suggests. It peaks at p = 0.5 and falls toward zero at both extremes. A brand that is almost never named and a brand that is almost always named will both show very low flip rates, and neither result says anything about whether the engine is stable. Only the comparison between observed and expected carries information.

What we measured

All figures come from Ansengine production measurement between 11 July and 31 August 2026, a 51 day window. A mention is counted when the stored answer names the brand under our verified matching rules. Branded prompts are excluded throughout: a prompt containing the brand name guarantees the mention and would inflate every rate. The 32 brand records are a record count, not a claim of 32 distinct companies; the cohort includes duplicate and internal test records, and we have not removed them, because removing them after seeing the result would be a choice made on the outcome.

Answers that failed, timed out or have not yet been graded are excluded and never counted as absent. That is 3,361 ungraded answers sitting outside this analysis entirely. Counting an ungraded answer as a non-mention would depress the base rate, which would lower the expected flip rate, which would manufacture the exact stability this study is trying to test.

Finding 1: our answers are about three times more stable than chance

Across the pooled cohort the brand was named in 821 of 8,060 answers, a base rate of 10.19%. Mentions flipped on 259 of 4,174 consecutive pairs, an observed rate of 6.21% with a 95% Wilson interval of 5.5% to 7.0%. Chance alone predicts 18.30%. That is a stability ratio of 2.95x, and the observed interval sits far below the expected value, so the effect is real rather than sampling noise.

Consecutive answers are not independent draws. Knowing that an engine named a brand yesterday tells you a great deal about today. That is the finding a raw flip rate cannot express, because 6.21% looks like instability until you know that chance predicts 18.30%.

Pairs within a cell are correlated, so a pair-level analysis overstates the effective sample. We repeated the test with the cell as the unit, which is much closer to independent. Of the 916 cells with three or more observations, 114 (12.4%) contained both a mention and a non-mention. Using each cell’s own observation count and the pooled base rate, independence predicts 39.8%. The correction survives the change of unit and the effect size is materially the same.

Finding 2: two engines, one number, two meanings

Google AI Mode flipped on 4.94% of pairs. Copilot flipped on 5.33%. Those two figures are four tenths of a point apart. Published side by side as raw flip rates they would look like the same result, and a reader would reasonably conclude the two engines behave alike.

They do not. Google AI Mode names the brand in 4.90% of answers, so chance alone would produce a 9.31% flip rate. Copilot names the brand in 2.56% of answers, so chance alone would produce 4.98%.

One of these engines is meaningfully more consistent than chance. The other is not distinguishable from flipping a biased coin at every observation. The raw numbers cannot tell them apart, and the raw numbers are what almost everyone publishes, ourselves included until now.

Finding 3: the correction inverts the engine ranking

Every engine in the cohort, with its base rate, its observed flip rate and interval, and the value chance predicts. “Proven” means the observed interval sits entirely below the expected value.

Where we cannot prove stability, we say so

Four of the nine engines clear the bar. Four do not, and one, Copilot, sits at chance with a ratio of 0.93x. The four that fail are the four with the lowest base rates and the smallest samples, which is exactly what the arithmetic predicts. When a brand is named in 2.4% of answers, chance itself only produces a 4.6% flip rate, so separating real stability from noise needs far more pairs than we currently have on those surfaces. We are not claiming 2.95x everywhere. We claim it where the interval supports it and report the rest as unproven.

The same pattern appears across brands. Every brand in the cohort with a meaningful base rate lands between 2.07x and 3.40x, which says the pooled result is not carried by one client. Every brand with a base rate near zero lands at or below 1.0x, which says the statistic stops working exactly where the mention rate collapses.

How this relates to our August study

On 27 August we published per-engine verdict stability with different figures, and the two are not in conflict. They measure different things, and saying so is the point of versioning a measurement.

That study grouped runs by brand, prompt, engine and measurement period, then reported the share of same-period cells that contained both verdicts. It captures within-period variation: the same prompt asked repeatedly under the same conditions. This study drops the period from the key, orders every observation in a cell by capture time, and compares each one to the one before it. It therefore captures within-period variation and day-to-day drift across a 51 day window.

Both are ours, both are correct, and the gap between them is not an error to reconcile away. It is the drift component, and it is the reason a number measured within one day should never be compared with a number measured across seven weeks without saying which is which.

The reporting rule we are adopting

Any published claim about AI answer stability should carry four fields. Three are already common. The fourth is the one that makes the other three mean something.

• The observed flip rate, as flips over pairs.
• The marginal mention rate, as mentions over valid runs. Without it the flip rate has no null model and cannot be read.
• The expected flip rate under independence, 2p(1 minus p).
• An interval on the observed rate, per engine, so a ratio near 1.0 is reported as unproven rather than as stability.

Engines have to be reported separately for this to work at all. Pooling them averages together base rates that differ by a factor of seven in our cohort, which makes the pooled expected value meaningless.

What this does not prove

• It is not a random sample. This is an observational production cohort of our customers' prompts, weighted toward B2B software with a small number of local businesses. Base rates in another cohort will differ, and every number here moves with them.
• It does not establish that any engine is stable in general. It establishes that in this cohort, over this window, consecutive answers are not independent draws.
• The independence null is a floor, not a model of engine behaviour. Real answers are correlated through prompt difficulty, brand stature and index state. Beating independence is the minimum bar for calling something stability, not a full account of it.
• It says nothing about causality. No intervention was tested and no control arm was run. We run concurrent held-out prompt arms separately for that purpose.
• A stability ratio is not a quality score, and it is not a rank. An engine being predictable is not the same as an engine being favourable.
• Pair-level counts are clustered, which is why the cell-level replication is reported beside them. Both are given so the reader can choose.

Reproduce it

The calculation is four lines of SQL over any store that keeps one row per answer with a timestamp and a mention flag: partition by brand, prompt and engine, order by capture time, compare each observation to the one before, and divide. The null model is arithmetic. Anyone holding repeated-measurement data can run this against their own cohort today, and we would rather they did.

The aggregate table behind every figure above is published as a CSV: per engine, the observation count, the mention numerator and denominator, the pair count, the flip count, the interval, the expected value and the ratio. No prompts, no answers, no customer identifiers.