Documentation
AI traffic and crawler logs
Connect your hosting platform's request logs so Ansengine can show which AI crawlers read your pages, verified, and which AI assistants send people back.
Every other surface measures what the engines say. AI traffic measures what they do: the crawlers that fetch your pages and the assistants that send real people back, from your own request logs. No JavaScript tracker, because bots do not run JavaScript.
What you get
| Number | What it is |
|---|---|
| Crawler visits | Requests from an AI crawler, split by purpose: AI training (GPTBot, ClaudeBot), AI search (OAI-SearchBot, PerplexityBot, Googlebot) and user triggered assistant fetches (ChatGPT-User, Claude-User, Perplexity-User). |
| Verified and failed verification | A request claiming to be a crawler is checked against the vendor's published IP ranges. Inside the ranges is verified; outside is reported as spoofed and never counted as a crawler visit. Vendors that publish no ranges get no verdict rather than an invented one. |
| AI referral sessions | People who arrived from chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com, claude.ai and the rest, attributed from the referrer without GA4. |
| Read, never cited | Pages crawlers fetch that no measured answer has ever cited. The engines are reading it and not trusting it: the most specific finding this product makes. |
Counts, never estimates. A zero means nothing happened, not that a model guessed low. Only AI classified requests are stored; ordinary visitors and other bots are counted in the delivery receipt and dropped.
Setup in two steps
- 1Open AI traffic in the console and create an ingest key. It is an ingest only credential: it can write traffic rows and nothing else. Creating a key revokes every previous one.
- 2Pick your hosting platform on the same page. The guide, the URLs and the code already carry your key, so setup is a paste. The Connected sources strip turns green on the first delivery.
Platforms
| Platform | How | Plan |
|---|---|---|
| Vercel | A log drain to a custom endpoint, NDJSON, with an Authorization header. No code changes. | Vercel Pro or Enterprise |
| Cloudflare | A Worker on your zone that forwards after the response, or a Logpush job with the http_requests dataset. | Worker on any plan; Logpush on Enterprise |
| Netlify | An Edge Function on any plan, or a General HTTP log drain for traffic logs. | Edge Function any plan; log drain Enterprise |
| AWS CloudFront | Standard logging v2 to an Amazon Data Firehose stream with an HTTP endpoint destination, JSON or w3c output. | Any distribution |
| Fastly | Real time log streaming to an HTTPS endpoint with the JSON format string we provide. | Any service |
| Akamai | DataStream 2 to a custom HTTPS endpoint, JSON format, X-Ansengine-Key header. | DataStream 2 entitled |
| Google Cloud | A Cloud Logging sink for load balancer request logs into Pub/Sub, pushed to us with the key on the URL. | Any project |
| WordPress, Kinsta, WP Engine | A must use plugin that forwards AI traffic from the origin, non blocking. Full page caches that skip PHP miss some hits. | Any host |
| Next.js, Node | A middleware that forwards only AI traffic, off the request path. | Any host |
| Anything else | POST batches of up to 500 events to the REST endpoint with the ingest key as a Bearer token. | Any |
What is collected
Four fields per request: path, user agent, referrer, client IP. Not collected: query strings, cookies, request bodies, other headers, or any other identifier. The path is taken without its query string on purpose. The IP is used in memory for range verification and dropped; what lands in the table is the classified bot, the verification verdict, the path and the referrer host.
Platform log drains deliver every request, humans included. The api keeps only AI classified lines; the rest are counted and discarded. Over the daily row cap the api answers 429 and drops the batch rather than truncating it silently, and the Connected sources strip shows the cap as an error.
Verification, honestly
A user agent is a claim, not an identity. Published measurements put spoofed AI crawler traffic at about 6 percent of everything bearing an AI crawler user agent, and most sites accept it. Ranges are fetched live from the vendors' own JSON files, cached, and treated as valid when stale, so a vendor's slow republish never turns real bots into spoofs. Anthropic publishes one flat list, so a Claude crawler verifies as Anthropic without separating ClaudeBot from Claude-User; that limit is stated on the page rather than papered over.