AI crawler blocking by category
Share of the 69 well-known sites we track whose robots.txt disallows each AI crawler for the homepage path /, split by category.
Source: latest CrawlDiffBot robots.txt scan per site, scanned 2026-10-19 to 2026-10-20. Unreadable robots.txt excluded. robots.txt is advisory: well-behaved crawlers follow it, others may ignore it, and it is not proof of live access.
Blocked in robots.txt, by crawler and category
| Crawler | News & media | Docs & reference | SaaS & developer tools | Ecommerce & retail | AI companies | All |
|---|---|---|---|---|---|---|
| GPTBot | 47% 9/19 | 0% 0/12 | 11% 2/18 | 23% 3/13 | 0% 0/7 | 20% 14/69 |
| OAI-SearchBot | 26% 5/19 | 0% 0/12 | 0% 0/18 | 8% 1/13 | 0% 0/7 | 9% 6/69 |
| ChatGPT-User | 37% 7/19 | 0% 0/12 | 0% 0/18 | 8% 1/13 | 0% 0/7 | 12% 8/69 |
| ClaudeBot | 84% 16/19 | 0% 0/12 | 11% 2/18 | 23% 3/13 | 0% 0/7 | 30% 21/69 |
| Claude-Web | 74% 14/19 | 0% 0/12 | 6% 1/18 | 8% 1/13 | 0% 0/7 | 23% 16/69 |
| PerplexityBot | 79% 15/19 | 0% 0/12 | 0% 0/18 | 15% 2/13 | 0% 0/7 | 25% 17/69 |
| Google-Extended | 63% 12/19 | 0% 0/12 | 6% 1/18 | 15% 2/13 | 0% 0/7 | 22% 15/69 |
| Bytespider | 84% 16/19 | 0% 0/12 | 17% 3/18 | 23% 3/13 | 0% 0/7 | 32% 22/69 |
| CCBot | 89% 17/19 | 0% 0/12 | 11% 2/18 | 23% 3/13 | 0% 0/7 | 32% 22/69 |
| anthropic-ai | 79% 15/19 | 0% 0/12 | 6% 1/18 | 8% 1/13 | 0% 0/7 | 25% 17/69 |
| meta-externalagent | 68% 13/19 | 0% 0/12 | 6% 1/18 | 15% 2/13 | 0% 0/7 | 23% 16/69 |
| Applebot-Extended | 84% 16/19 | 0% 0/12 | 6% 1/18 | 8% 1/13 | 0% 0/7 | 26% 18/69 |
Summary by category
| Category | Block at least one AI crawler | Valid /llms.txt |
|---|---|---|
| News & media | 17 of 19 | 1 of 19 |
| Docs & reference | 0 of 12 | 5 of 12 |
| SaaS & developer tools | 3 of 18 | 12 of 18 |
| Ecommerce & retail | 3 of 13 | 5 of 13 |
| AI companies | 0 of 7 | 3 of 7 |