robots.txt generator for AI crawlers
Choose which AI crawlers may read your site, for answers and for training, and copy a robots.txt that does exactly that. It runs in your browser: nothing you enter is sent to us.
Your robots.txt
User-agent: *
Disallow: /admin/
Disallow: /api/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
Save it as robots.txt at the root of your site, so it opens at yoursite.com/robots.txt.
What each choice costs
The search bots, OAI-SearchBot, Claude-SearchBot and PerplexityBot, fetch pages so ChatGPT, Claude and Perplexity can find and quote them. Blocking one can keep your pages out of that assistant’s answers.
The training bots, GPTBot, ClaudeBot and Google-Extended, collect pages for future models. Blocking them keeps your new pages out of training and leaves the answers alone. Google says Google-Extended does not affect Google Search.
Googlebot is not on the list on purpose. It is how pages reach Google Search, AI Overviews included, and blocking it takes the site out of Google altogether.
Why allowed bots get no group of their own
Under the robots.txt standard, RFC 9309, a crawler follows only the group that names it and falls back to the * group when none does. Add User-agent: OAI-SearchBot with Allow: / to welcome it, and it stops reading the * rules, so the paths you closed there are open to it.
This generator avoids that: bots you allow follow *, with your private paths, and only blocked bots get a group, closing the whole site to them.
What robots.txt does not control
Fetches a person triggers can work differently: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. A CDN can also block bots before they reach the file, and a page that builds its text with JavaScript can look empty to crawlers that do not run it.
Our guide covers each of these with the vendors’ documentation: Is your site blocking ChatGPT?