The ChatGPT you measure is not the ChatGPT your buyers use.

Almost every tool that claims to track ChatGPT is querying the OpenAI API. It is the reasonable engineering choice: it is documented, it is stable, and it returns clean structured output. We built our own ChatGPT adapter that way too.

The assumption underneath it is that the API answers roughly like the product. We wanted to know whether that assumption survives contact with evidence, because if it does not, then a whole category of tools is confidently reporting the wrong thing.

Method

What the three runs returned

The second prompt is the one that matters

Counting shared domains understates the gap. On the second prompt the two answers did share two sources, so a naive overlap metric would score it as partial agreement. But both shared sources were Trustpilot and Reddit, which are review aggregators rather than providers. Of the businesses actually recommended to a buyer, the overlap was zero.

That is the distinction a visibility tool has to get right. Being cited as a source and being recommended as the answer are separate outcomes, and a brand can win one while losing the other.

It is why we never collapse them into a single visibility score. Presence is measured as three separate rates for every prompt: how often an answer names you, how often it cites one of your own domains, and how often it does both in the same answer. Then a second layer grades how the answer actually presented you, and that grade only counts when the classifier can produce a verbatim sentence from the answer that names you. If it cannot, the run stays unevaluated and shrinks the sample rather than quietly counting as “not recommended”.

Two things we found that we were not looking for

Ads are already inside consumer answers. Two of the three consumer responses carried a sponsored item. Nothing in the API surface shows them. If you are benchmarking your category through an API, paid placement in the answer your buyer reads is invisible to you.

The consumer surface can return nothing. One of three scrapes came back empty. We record that as a failed run and exclude it from every denominator. The tempting alternative, counting it as an answer that mentioned nobody, would quietly depress the measured rate of every brand in the category and make our own product look more useful than it is.

What we changed because of it

We ship the consumer surface as its own engine rather than folding it into our ChatGPT number. Averaging the two would have produced a figure that describes neither. They appear as separate rows, measured separately, and a brand can see that it is winning one and losing the other.

We also wrote the finding into our published refusals: we will not sell AI answer measurement drawn from model APIs alone. That page lists the other things we decline to build and why, including our position that schema markup is hygiene rather than a citation lever.

How to check this yourself

You do not need our product to reproduce the shape of this. Ask ChatGPT the question your best customer would ask, in the app, and write down every business it names. Then ask the same question through the API, or through any tool that says it tracks ChatGPT, and compare the two lists. If they differ, the number your tool reports is describing a surface your buyers do not use.

If you would rather have it measured repeatedly and with sample sizes attached, that is what we built. A free report runs your own buyer questions across the engines and shows the rates with their ranges, including the runs that failed.