AI crawler reports
Which AI crawlers do well-known sites allow? Each report shows the robots.txt decision for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more, plus whether the site publishes /llms.txt. Refreshed about daily by CrawlDiffBot (robots.txt and llms.txt only).
robots.txt is advisory: well-behaved crawlers follow it, others may ignore it, and it is not proof of live access. Counts cover the homepage path /; many sites still block deeper sections.
News & media
- nytimes.com 12 of 12 blocked in robots.txt · llms.txt missing
- bbc.com 12 of 12 blocked in robots.txt · llms.txt missing
- theguardian.com 7 of 12 blocked in robots.txt · llms.txt missing
- cnn.com 10 of 12 blocked in robots.txt · llms.txt missing
- reuters.com 10 of 12 blocked in robots.txt · llms.txt missing
- washingtonpost.com 8 of 12 blocked in robots.txt · llms.txt missing
- wsj.com 9 of 12 blocked in robots.txt · llms.txt missing
- bloomberg.com 10 of 12 blocked in robots.txt · llms.txt missing
- apnews.com 7 of 12 blocked in robots.txt · llms.txt missing
- npr.org
- theverge.com 10 of 12 blocked in robots.txt · llms.txt missing
- techcrunch.com 9 of 12 blocked in robots.txt · llms.txt missing
Docs & reference
- developer.mozilla.org 0 of 12 blocked in robots.txt · llms.txt missing
- docs.python.org 0 of 12 blocked in robots.txt · llms.txt missing
- nodejs.org 0 of 12 blocked in robots.txt · llms.txt present
- react.dev 0 of 12 blocked in robots.txt · llms.txt present
- kubernetes.io 0 of 12 blocked in robots.txt · llms.txt missing
- docs.github.com 0 of 12 blocked in robots.txt · llms.txt present
- learn.microsoft.com 0 of 12 blocked in robots.txt · llms.txt missing
- developer.apple.com 0 of 12 blocked in robots.txt · llms.txt missing
- docs.aws.amazon.com 0 of 12 blocked in robots.txt · llms.txt present
- cloud.google.com 0 of 12 blocked in robots.txt · llms.txt present
- vercel.com 0 of 12 blocked in robots.txt · llms.txt present
- wikipedia.org 0 of 12 blocked in robots.txt · llms.txt missing
SaaS & developer tools
- stripe.com 0 of 12 blocked in robots.txt · llms.txt present
- github.com 1 of 12 blocked in robots.txt · llms.txt present
- notion.so 0 of 12 blocked in robots.txt · llms.txt present
- slack.com 0 of 12 blocked in robots.txt · llms.txt present
- atlassian.com 0 of 12 blocked in robots.txt · llms.txt present
- salesforce.com 0 of 12 blocked in robots.txt · llms.txt present
- hubspot.com 0 of 12 blocked in robots.txt · llms.txt present
- zoom.us 0 of 12 blocked in robots.txt · llms.txt missing
- figma.com 5 of 12 blocked in robots.txt · llms.txt missing
- dropbox.com 0 of 12 blocked in robots.txt · llms.txt present
- openai.com 0 of 12 blocked in robots.txt · llms.txt missing
- anthropic.com 0 of 12 blocked in robots.txt · llms.txt missing
- cloudflare.com 0 of 12 blocked in robots.txt · llms.txt present
Ecommerce & retail
- shopify.com 0 of 12 blocked in robots.txt · llms.txt present
- amazon.com 10 of 12 blocked in robots.txt · llms.txt missing
- ebay.com 8 of 12 blocked in robots.txt · llms.txt missing
- etsy.com 0 of 12 blocked in robots.txt · llms.txt present
- walmart.com 0 of 12 blocked in robots.txt · llms.txt present
- target.com 0 of 12 blocked in robots.txt · llms.txt present
- bestbuy.com 0 of 12 blocked in robots.txt · llms.txt missing
- ikea.com 0 of 12 blocked in robots.txt · llms.txt missing
- aliexpress.com 5 of 12 blocked in robots.txt · llms.txt present
- zalando.com 0 of 12 blocked in robots.txt · llms.txt present
- wayfair.com 0 of 12 blocked in robots.txt · llms.txt missing
- homedepot.com 0 of 12 blocked in robots.txt · llms.txt missing
- costco.com 0 of 12 blocked in robots.txt · llms.txt missing