Free tools · Free tool · no sign-up
AI robots.txt checker (GPTBot, ClaudeBot, PerplexityBot)
Many sites block AI crawlers without knowing it, often because of a hosting setting or a security plugin. As a result, ChatGPT or Perplexity can't read their pages and never recommend them. This tool reads your robots.txt the way each crawler would.
How it works
- 1Enter your site's address (or a specific page).
- 2We read your robots.txt and apply each AI crawler's rules, one by one.
- 3If a crawler is blocked, we give you the exact lines to add to unblock it.
Frequently asked questions
What's the difference between search and training crawlers?
Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) read your site when someone asks a question: blocking them stops you from being cited. Training crawlers (GPTBot, ClaudeBot, Google-Extended) are used to teach models: blocking them is a valid choice, but the AI will know your brand less well.
Does blocking Google-Extended remove me from Google?
No. Google-Extended only covers Gemini using your pages for training. Your regular Google rankings depend on Googlebot, which isn't affected.
Is llms.txt required?
No, it isn't an official standard. It's a simple bonus: a summary of your site that some AIs can read.
Want to see everything holding your site back?
The free audit checks the 7 pillars of SEO, on Google and in AI answers. Results in a minute.
Other free tools
- robots.txt generator AICreate a clean robots.txt in 1 minute: folders to hide, sitemap, and a clear choice for AI crawlers (allow all, block training, block all).
- Google preview (SERP simulator)See how your title and meta description show up in Google, on desktop and mobile, with the ideal character count.
- Indexability checkerCheck in seconds whether a page can appear on Google: response code, redirects, noindex, canonical tag and robots.txt.
- Open Graph checkerSee your page's preview when it's shared (Facebook, LinkedIn, WhatsApp, X) and which Open Graph tags are missing.
- SEO tag checkerFree analysis of your homepage: title tag, meta description, H1, HTTPS, sitemap, structured data and llms.txt.
- Structured data generatorCreate for free the JSON-LD code that helps Google and AI understand who you are: company, local business, restaurant, service provider.
- llms.txt generator AICreate your llms.txt file in 2 minutes: a clear summary of your site and key pages that AI can read.