AI crawlers

Bots like GPTBot, ClaudeBot and Google-Extended that collect web content for AI training and retrieval.

AI crawlers are the user agents AI companies use to gather web content — for model training (e.g. GPTBot, ClaudeBot, Google-Extended) and for live retrieval when a user's question triggers web search (e.g. ChatGPT-User, Perplexity-User). Your robots.txt decides which of them may read your site.

Blocking all AI crawlers is a real strategic decision, not a default: it can keep your content out of both training data and live answers, which for most brands means invisibility exactly where buyers are asking. A deliberate policy usually allows retrieval agents on marketing content while controlling training access case by case.

Related terms

llms.txt AI web search mode Training data presence

If you want to know where your own brand stands, run a free audit — five engines, about two minutes, no credit card.

10,907 AI reports analyzed · 734 audits run · 78% of brands are invisible in AI

30-day money-back guarantee