Free robots.txt generator for AI crawlers
robots.txt is a small file that tells web crawlers which parts of your site they may visit. AI companies run several different crawlers, and some are used to build answers while others collect content to train models.
Pick a starting point, adjust individual bots if you like, then copy or download the file. Nothing leaves your browser.
Preview: robots.txt
# AI crawlers: allowed User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot User-agent: Claude-User Allow: / # AI crawlers: blocked User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: Amazonbot User-agent: Bytespider Disallow: / # Everyone else User-agent: * Allow: /
Where does this go?
Save the file as robots.txt and place it at the top level of your domain, so it opens at https://www.example.com/robots.txt. Each subdomain needs its own file.
If your site already has a robots.txt, compare before you replace it, so you do not lose rules you rely on. Firewalls and bot protection services can also block crawlers regardless of this file.
Check how your page scores
Once the file is live, run a free scan to see how readable and citable a page is to AI answer engines, with a plain-language list of fixes.
Scan a page