AI crawler report: amazon.com
Scanned 2026-10-19T10:36:21.259Z (cached) · All reports · Run a fresh check
Summary
2 of 12 AI crawlers OK · 10 of 12 blocked in robots.txt · robots.txt present · llms.txt missing
robots.txt + llms.txt only (no live homepage fetches with AI crawler User-Agents); run a free check for the full test. CrawlDiff reports what our fetches saw. It does not guarantee AI citations or indexing. robots.txt is advisory: well-behaved crawlers follow it, others may ignore it, and it is not proof of live access.
Notes
Partial path blocks: 2 bots may fetch / but have Disallowed paths (up to 119 rules, e.g. /exec/obidos/account-access-login, /exec/obidos/change-style, /exec/obidos/flex-sign-in, /exec/obidos/handle-buy-box). "OK" means the homepage only, not the whole site.
Get alerted when this changes
CrawlDiff Watch re-checks amazon.com every 12 hours and emails you when an AI crawler gets newly blocked or llms.txt breaks.
Enter amazon.com in the Domain field at checkout.
Bot matrix
| Bot | robots.txt | Live HTTP | Status |
|---|---|---|---|
| GPTBot | deny | not checked | blocked |
| OAI-SearchBot | deny | not checked | blocked |
| ChatGPT-User | deny | not checked | blocked |
| ClaudeBot | deny | not checked | blocked |
| Claude-Web | deny | not checked | blocked |
| PerplexityBot | deny | not checked | blocked |
| Google-Extended | deny | not checked | blocked |
| Bytespider | deny | not checked | blocked |
| CCBot | deny | not checked | blocked |
| anthropic-ai | allow | not checked | ok |
| meta-externalagent | deny | not checked | blocked |
| Applebot-Extended | allow | not checked | ok |
llms.txt
Not found (HTTP 404, 2296 bytes)