GPTBot, ClaudeBot, PerplexityBot: should you block AI crawlers?

Your robots.txt file decides who can read your site. Many hosts and themes have recently added rules that block “AI” in one go, sometimes without telling you. Result: businesses that want to be recommended by ChatGPT make themselves invisible without knowing it.

Two families of crawlers: search and training

AI companies use several crawlers with different roles. That's the key to setting up your robots.txt correctly:

• Search crawlers: they read your site to answer users' questions with cited sources. Blocking them means giving up on being cited.
• Training crawlers: they collect pages to train models. Blocking them protects your content, with no effect on your Google presence.
• User-triggered agents: they visit a page at a user's specific request (for example when someone pastes your link into ChatGPT).

The main crawlers to know

• OAI-SearchBot (OpenAI): search in ChatGPT. Allow it if you want to be cited.
• GPTBot (OpenAI): model training.
• ChatGPT-User (OpenAI): visits on a user's request.
• ClaudeBot (Anthropic): training; Claude-SearchBot and Claude-User for search and user-triggered visits.
• PerplexityBot (Perplexity): indexing for Perplexity's answers.
• Google-Extended (Google): controls the use of your content by Gemini models. It's not a separate crawler and changes nothing about your presence in Google Search.
• Applebot-Extended (Apple): use for training Apple's models.
• CCBot (Common Crawl): a public crawl reused by many models.

Companies update these names; check each one's official documentation from time to time.

Three setups depending on your goal

You want to be cited everywhere (the most common case)

Don't block any AI crawler. Your robots.txt can stay simple: allow everything except your private pages (account area, cart).

You want to be cited, but not used for training

Allow search crawlers and block training crawlers. Example:

• User-agent: GPTBot → Disallow: /
• User-agent: ClaudeBot → Disallow: /
• User-agent: Google-Extended → Disallow: /
• User-agent: CCBot → Disallow: /
• (OAI-SearchBot, Claude-SearchBot and PerplexityBot stay allowed)

You want to stay out entirely

Block all AI crawlers, knowing you'll disappear from their answers. It's a defensible choice for a publisher that sells its content, rarely for a business looking for customers.

Check and fix in two minutes

The AI crawler checker reads your file and tells you, crawler by crawler, who's allowed or blocked, with the fix to copy. To start from a clean base, the robots.txt generator creates a file with the AI settings of your choice.

Common mistakes

• A “block AI” option turned on by the host or firewall, which also blocks search crawlers.
• A leftover “Disallow: /” after launching a site from staging.
• Blocking Google-Extended thinking it removes you from Google's AI Overviews (it doesn't: see our Gemini guide).
• Believing that allowing a crawler is enough to get cited: you also need content that answers (see how to get recommended by ChatGPT).