The Complete List of AI Crawlers in 2026 (and How to Handle Each)
Every major AI crawler in 2026 — who runs it, what it does, and whether to allow it. A reference table for your robots.txt strategy.
Why keep a crawler list?
New AI user-agents appear regularly, and each one is a potential source of citations — or, if mishandled, a gap. Knowing who's crawling lets you make deliberate robots.txt decisions.
The major AI crawlers
The table below lists the crawlers worth knowing in 2026. To be cited across AI surfaces, allow the ones whose engines you want answering with your content.
Keep it current
Re-check your robots.txt quarterly against the latest crawler list — GeoPageScan tracks new AI bots in its audit config and flags any you're blocking.
AI crawlers worth knowing in 2026
| Crawler | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | ChatGPT training & retrieval |
| OAI-SearchBot | OpenAI | ChatGPT Search index |
| ClaudeBot | Anthropic | Claude training & retrieval |
| PerplexityBot | Perplexity | Perplexity answer index |
| Google-Extended | Gemini & AI Overviews | |
| Applebot-Extended | Apple | Apple Intelligence |
| Amazonbot | Amazon | Alexa / Rufus answers |
Frequently asked questions
How many AI crawlers are there in 2026?⌄
More than a dozen matter, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and Amazonbot. Allow the ones whose engines you want citing you.
Which AI crawlers should I block?⌄
Only block crawlers whose engines you deliberately want to opt out of — blocking removes you from those AI answers. Most sites seeking visibility allow them all.
How do I keep my crawler list current?⌄
Re-audit quarterly. GeoPageScan tracks new AI bots in its config and flags any your robots.txt is blocking.
