Which sites block ClaudeBot?

21 of 69 (30%) well-known sites we track disallow ClaudeBot (Anthropic's crawler) for the homepage path / in robots.txt.

Source: latest CrawlDiffBot scan of each tracked site's robots.txt, scanned 2026-10-19 to 2026-10-20. Sites whose robots.txt could not be read are excluded. robots.txt is advisory: well-behaved crawlers follow it, others may ignore it, and it is not proof of live access.

By category

CategoryBlock ClaudeBotShare
News & media16 of 1984%
Docs & reference0 of 120%
SaaS & developer tools2 of 1811%
Ecommerce & retail3 of 1323%
AI companies0 of 70%

Block ClaudeBot (21)

Allow ClaudeBot at the homepage (48)

More data

Check your own domain Watch from $12/mo