Know when AI crawlers get blocked
Paste a domain. See the robots.txt matrix and live fetches for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more. Watch alerts only on regressions (was OK → now blocked). Optional checkbox: also watch /llms.txt.
Your result shows which AI crawlers are blocked, with a $12/mo Watch option to get alerted when it changes. Pricing · Data: which sites block GPTBot · ClaudeBot · by category
robots.txt matrix
Allow/deny for the common AI bots that matter for GEO and training.
Live UA fetches
We request your homepage with each bot's User-Agent to catch CDN/WAF 403s robots.txt cannot show.
Regression alerts
Email only when something that was OK becomes blocked — not on deliberate steady-state blocks you mark intentional.
What we don't claim
CrawlDiff does not guarantee ChatGPT citations, rankings, or that any model will read your site. It reports crawl access and file health.
Frequently asked questions
What does CrawlDiff check?
It reads your robots.txt and decides allow or block for 12 AI crawler tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, Bytespider, CCBot, anthropic-ai, meta-externalagent, Applebot-Extended). It also fetches your homepage with each bot's User-Agent to catch CDN or WAF blocks that robots.txt cannot show, and can optionally check /llms.txt.
Is the check free?
Yes. One-shot checks are free and need no account, limited to 5 scans per IP per day.
How much does Watch cost?
Watch is $12 per month for 1 domain and Agency is $39 per month for up to 10 domains. Payments go through Stripe.
When does Watch send an alert?
Watch re-scans about twice a day and emails you only on regressions: when a crawler that was OK becomes blocked, or llms.txt breaks. Blocks you mark as intentional do not trigger alerts.
Does CrawlDiff guarantee AI citations or rankings?
No. CrawlDiff reports crawl access and file health. It does not guarantee that ChatGPT or any other AI system will read, cite or rank your site.