Free tools · Free tool · no sign-up
Free robots.txt generator (with AI crawler settings)
robots.txt tells crawlers what they may read. A single wrong line can hide your whole site from Google or AI. This generator produces a simple, safe file with an explicit choice for AI.
User-agent: * Disallow: /admin Disallow: /panier Sitemap: https://your-site.com/sitemap.xml
Save this file as robots.txt at the root of your site (your-site.com/robots.txt).
How it works
- 1Enter your sitemap URL and your private folders (admin, cart…).
- 2Choose what you allow AI crawlers to do.
- 3Copy the file and save it at the root of your site as robots.txt.
Frequently asked questions
Does robots.txt protect my private pages?
No. It politely asks crawlers not to read certain URLs, but doesn't stop anyone from accessing them. Protect private pages with a password.
Where does the file go on WordPress, Wix or Shopify?
WordPress: with an SEO plugin (Yoast, Rank Math) or at the root through your host. Wix: SEO settings, robots.txt editor. Shopify: the robots.txt.liquid file in your theme.
What should I choose for AI?
To be recommended by ChatGPT, Perplexity or Claude, keep at least the search crawlers allowed. That's the “block training, keep search” option if you don't want to feed the models.
Want to see everything holding your site back?
The free audit checks the 7 pillars of SEO, on Google and in AI answers. Results in a minute.
Other free tools
- AI crawler checker AICheck for free whether ChatGPT, Claude, Perplexity and Gemini are allowed to read your site: robots.txt, GPTBot, OAI-SearchBot, PerplexityBot, llms.txt.
- Google preview (SERP simulator)See how your title and meta description show up in Google, on desktop and mobile, with the ideal character count.
- Indexability checkerCheck in seconds whether a page can appear on Google: response code, redirects, noindex, canonical tag and robots.txt.
- Open Graph checkerSee your page's preview when it's shared (Facebook, LinkedIn, WhatsApp, X) and which Open Graph tags are missing.
- SEO tag checkerFree analysis of your homepage: title tag, meta description, H1, HTTPS, sitemap, structured data and llms.txt.
- Structured data generatorCreate for free the JSON-LD code that helps Google and AI understand who you are: company, local business, restaurant, service provider.
- llms.txt generator AICreate your llms.txt file in 2 minutes: a clear summary of your site and key pages that AI can read.