The robots.txt Guide for AI Crawlers: Allow the Right Bots
A copy-paste robots.txt that welcomes the AI crawlers you want citing you — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more.
Why robots.txt matters for AI visibility
robots.txt is the first gate every crawler checks. If an AI user-agent is disallowed — or just ambiguous — that engine may never read or cite your content.
Being explicit signals intent and maximizes coverage across AI surfaces.
A copy-paste robots.txt for AI
Allow the major AI user-agents and point to your sitemap:
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlDecide what you actually want
If you want AI citations, allow these bots. If you want to keep content out of AI training, disallow specific agents — but know that blocking them removes you from those engines' answers entirely.
Frequently asked questions
Which AI bots should I allow in robots.txt?⌄
To be cited, allow GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, and link your sitemap.
Does blocking GPTBot remove me from ChatGPT?⌄
Blocking GPTBot and OAI-SearchBot prevents OpenAI from indexing or citing your content in ChatGPT's answers. Only block if you intend to opt out.
Where does robots.txt go?⌄
At your domain root: https://yourdomain.com/robots.txt. Add explicit per-user-agent Allow rules for the AI crawlers you want.
