AI crawlers and robots.txt: allow AI search, block training

Which AI bots read your site, what each one is for, and copy-paste robots.txt rules for the three usual choices. Then check your own page for free.

The one distinction that matters

Some AI crawlers collect pages to train models. Others fetch pages to answer a user right now and cite the source (AI search). Blocking the second kind means ChatGPT search, Claude and Perplexity can't quote or link you. Blocking training while allowing search is one option; it's a business choice, not a technical one.

The common robots.txt user-agent tokens

GPTBot (OpenAI): collects public pages for model training.
OAI-SearchBot (OpenAI): builds the index used by ChatGPT search results.
ChatGPT-User (OpenAI): fetches a page when a ChatGPT user or GPT asks for it.
ClaudeBot (Anthropic): collects public pages for model training.
Claude-SearchBot (Anthropic): indexes pages to improve Claude's search results.
Claude-User (Anthropic): fetches a page when a Claude user asks for it.
PerplexityBot (Perplexity): indexes pages for Perplexity's answers and citations.
Perplexity-User (Perplexity): fetches a page when a Perplexity user's question needs it. Because the fetch is user-triggered, robots.txt may not stop it; use server-side rules if you must block it.
Google-Extended (Google): not a separate crawler but a token that controls whether Googlebot-crawled content is used for Gemini models. Blocking it does not affect Google Search ranking.
Applebot-Extended (Apple): token that controls use of Applebot-crawled pages for Apple's AI model training.
Meta-ExternalAgent (Meta), Amazonbot (Amazon), Bytespider (ByteDance), CCBot (Common Crawl, a dataset many models train on).

Vendors rename and add bots; check each vendor's crawler documentation before relying on a token. robots.txt is a request, not an access control: well-behaved bots follow it, others may not.

1. Allow AI search, block model training

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

2. Block all of the AI bots above

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
Disallow: /

3. Allow everything

Make sure no rule above names these bots and that there is no User-agent: * with Disallow: /. An empty or missing robots.txt allows all crawlers.

Several User-agent lines in a row share the rules under them. Bots use the most specific group that names them, so a group naming GPTBot overrides your User-agent: * group for GPTBot.

robots.txt is only half of it

Even when AI bots are allowed, many don't run JavaScript. If your page is a JS app shell, they see an empty page. Our checker reads your page the way these bots do, then shows your robots.txt rules per bot, the text in the raw HTML, llms.txt and JSON-LD, with fixes ready to paste.

Check my page free →

Also: llms.txt format and template