AI crawler report: arstechnica.com

Scanned 2026-10-20T04:14:19.652Z (cached) · All reports · Run a fresh check

Summary

3 of 12 AI crawlers OK · 9 of 12 blocked in robots.txt · robots.txt present · llms.txt missing

robots.txt + llms.txt only (no live homepage fetches with AI crawler User-Agents); run a free check for the full test. CrawlDiff reports what our fetches saw. It does not guarantee AI citations or indexing. robots.txt is advisory: well-behaved crawlers follow it, others may ignore it, and it is not proof of live access.

Get alerted when this changes — $12/mo

CrawlDiff Watch re-checks arstechnica.com every 12 hours and emails you when an AI crawler gets newly blocked or llms.txt breaks.

Watch arstechnica.com — $12/mo See plans

Enter arstechnica.com in the Domain field at checkout.

Not ready to pay? Join the free weekly re-scan list for arstechnica.com (email only, launching soon). No spam; one-click unsubscribe.

Notes

Partial path blocks: 3 bots may fetch / but have Disallowed paths (up to 30 rules, e.g. /cgi-bin/, /wp/wp-admin/, /wp/wp-includes/, /wp/wp-content/). "OK" means the homepage only, not the whole site.

Bot matrix

Botrobots.txtLive HTTPStatus
GPTBotallownot checkedok
OAI-SearchBotallownot checkedok
ChatGPT-Userallownot checkedok
ClaudeBotdenynot checkedblocked
Claude-Webdenynot checkedblocked
PerplexityBotdenynot checkedblocked
Google-Extendeddenynot checkedblocked
Bytespiderdenynot checkedblocked
CCBotdenynot checkedblocked
anthropic-aidenynot checkedblocked
meta-externalagentdenynot checkedblocked
Applebot-Extendeddenynot checkedblocked

llms.txt

Not found (HTTP 404, 5565 bytes)