AI crawler
Also called AI bot, LLM crawler
An AI crawler is an automated fetcher operated by an AI company to collect web pages for training corpora, retrieval indexes, or live answer grounding. They identify themselves with their own user agents, and blocking one can remove your site from that engine's answers.
Three jobs, not one
The user agents fall into three groups, and conflating them causes real mistakes. Training and corpus crawlers collect pages to build or refresh a model. Index crawlers build the retrieval index an assistant searches while answering. Live fetchers retrieve a specific page at the moment a user asks about it.
Blocking a training crawler and blocking an index crawler have very different consequences. The first affects what a model absorbed over time. The second can remove you from answers immediately.
The block that costs the most
A blanket disallow that catches index and live-fetch agents is the most expensive mistake in this area, and it is usually accidental: a robots.txt rule written years ago for scrapers, a security product rejecting unfamiliar user agents, or a CDN returning a challenge page to anything without a browser fingerprint.
It is worth checking rather than assuming, because a site can rank perfectly well in classic search and still be invisible to an assistant that shares the same index.
Robots.txt matching is prefix based
User-agent tokens in robots.txt are matched case-insensitively as a prefix of the declared agent. A rule naming a shorter token can therefore catch agents you did not intend, and a rule with a trailing typo can silently match nothing at all. Both failures stay invisible until someone parses the file against the real agent list.
Related terms
- RetrievabilityRetrievability is whether an AI engine can fetch your page and extract your claims as text.
- Cited rateCited rate is the share of sampled AI answers that link to one of your own domains as a source.
- Supported rateSupported rate is the share of sampled answers that both name your brand and cite one of your own domains in the same answer.
Want this measured on your own domain rather than defined? Run a free report and see the rates, with their sample sizes and ranges, for your own buyer questions.