Documentation
AI traffic and crawler logs
Connect your request logs to see which AI agents read your pages, how sure we are who each one was, and which AI assistants send people back.
Every other surface measures what the engines say. AI traffic measures what they do: the crawlers and agents that fetch your pages and the assistants that send real people back, from your own request logs. No JavaScript tracker, because bots do not run JavaScript.
What you get
| Number | What it is |
|---|---|
| Agent visits by purpose | Requests from an AI agent, split by what it came for: AI training (GPTBot, ClaudeBot), AI search (OAI-SearchBot, PerplexityBot, Googlebot), a user fetch (ChatGPT-User, Claude-User, Perplexity-User, Google-GeminiNotebook), an agentic browser (ChatGPT's agent, Google-Agent) or a coding agent. Agentic browser and coding agent are always inferred. OpenAI's ads check (OAI-AdsBot) is counted as a visit, never as interest in your page. |
| Verification level | Every visit is filed by the strongest check it passed. Signed: a Web Bot Auth signature checked against the agent's published keys. IP: an address the vendor publishes, or its reverse DNS. User agent only: it mentioned itself and nothing confirmed it. Inferred: it mentioned nothing, and its behaviour points to an agent. A visit from outside the vendor's addresses is user agent only: still counted in the crawler total, never as verified. Only signed and IP visits count as verified. |
| Agent files | How often agents fetched llms.txt, Markdown pages, your OpenAPI document, sitemaps and an MCP server card. A count of reads, never a score: publishing llms.txt is hygiene. |
| Docs and the rest of the site | Agent visits to your documentation (a docs subdomain, or a first folder such as /docs or /reference) against every other page. |
| AI referral sessions | People who arrived from chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com, claude.ai and the rest, attributed from the referrer without GA4. |
| Read, never cited | Pages crawlers fetch that no measured answer has ever cited. The engines are reading it and not trusting it: the most specific finding this product makes. |
Counts, never estimates. A zero means nothing happened, not that a model guessed low. Only AI classified requests are stored; ordinary visitors and other bots are counted in the delivery receipt and dropped. Visits stored before levels were recorded show as not recorded, neither verified nor unverified.
Setup in two steps
- 1Open AI traffic in the console and create an ingest key. It is an ingest only credential: it can write traffic rows and nothing else. Creating a key revokes the one it replaces.
- 2Pick your hosting platform on the same page. The guide, the URLs and the code already carry your key, so setup is a paste. The Connected sources strip turns green on the first delivery.
Platforms
| Platform | How | Plan |
|---|---|---|
| Vercel | A log drain to a custom endpoint, NDJSON, with an Authorization header. No code changes. | Vercel Pro or Enterprise |
| Cloudflare | A Worker on your zone that forwards after the response, or a Logpush job with the http_requests dataset. | Worker on any plan; Logpush on Enterprise |
| Netlify | An Edge Function on any plan, or a General HTTP log drain for traffic logs. | Edge Function any plan; log drain Enterprise |
| AWS CloudFront | Standard logging v2 to an Amazon Data Firehose stream with an HTTP endpoint destination, JSON or w3c output. | Any distribution |
| Fastly | Real time log streaming to an HTTPS endpoint with the JSON format string we provide. | Any service |
| Akamai | DataStream 2 to a custom HTTPS endpoint, JSON format, X-Ansengine-Key header. | DataStream 2 entitled |
| Google Cloud | A Cloud Logging sink for load balancer request logs into Pub/Sub, pushed to us with the key on the URL. | Any project |
| WordPress, Kinsta, WP Engine | A must use plugin that forwards AI traffic from the origin, non blocking. Full page caches that skip PHP miss some hits. | Any host |
| Next.js, Node | A middleware that forwards only AI traffic, off the request path. | Any host |
| Anything else | POST batches of up to 500 events to the REST endpoint with the ingest key as a Bearer token. | Any |
What is collected
Per request: path, host, user agent, referrer, the Accept header, client IP and, where the source records it, the status your site answered with. A request signed with Web Bot Auth also sends its method, its full URL and its three signature headers; only the code collectors (the Worker, the Edge Function, the plugin and the middleware) can see them, and log drains cannot carry them. Every log drain carries the status; the Next.js middleware runs before the answer exists, so it sends none. Not collected: cookies, request bodies, or any other identifier. The path is stored without its query string. The IP, the Accept header and the signature material are used in memory and dropped; what lands in the table is the classified agent, what it came for, its verification level, whether it asked for Markdown, the status, the path, the host and the referrer host.
Platform log drains deliver every request, humans included. The api keeps only AI classified lines; the rest are counted and discarded. Over the daily row cap the api answers 429 and drops the batch rather than truncating it silently, and the Connected sources strip shows the cap as an error.
Verification, honestly
A user agent is a claim, not an identity. Published measurements put spoofed AI crawler traffic at about 6 percent of everything bearing an AI crawler user agent, and most sites accept it. Address lists are fetched live from the vendors' own files (OpenAI, Perplexity, Anthropic, Google, Bing), cached, and treated as valid when stale, so a vendor's slow republish never turns real bots into spoofs. Anthropic publishes one list for its three bots, so a match there proves Anthropic, not which bot; Google's and Bing's lists cover several of their agents the same way. Reverse DNS is a second check, never a veto: some of Google's agent addresses have no reverse DNS name at all.
Signatures follow the IETF Web Bot Auth draft. Two signers are checked today, ChatGPT's agent (chatgpt.com) and Google-Agent (agent.bot.goog), each against the keys it publishes, and a signature that cannot be checked is never counted as signed. Only signed and IP visits count as verified.