About CrawlDiff
CrawlDiff is a small independent project run by INTELLIFIN LIMITED. It checks whether AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach a website, and it can watch a site over time and email the owner when that access breaks.
Why it exists
Sites often block AI crawlers by accident: a new robots.txt rule, a CDN or firewall setting, or a deploy that removes /llms.txt. CrawlDiff is meant to catch those changes early, without guessing about rankings or citations.
How it works
- It reads
/robots.txtand decides allow or block for 12 AI crawler tokens. - It fetches the homepage with each crawler's User-Agent to see real HTTP responses.
- Optionally it checks that
/llms.txtis present and valid. - Our own requests identify as CrawlDiffBot, and site owners can opt out.
What we don't claim
CrawlDiff does not guarantee AI citations, rankings or indexing. It reports crawl access and file health only.
Contact
Email support@rendlet.com. See also Privacy and Terms.
Last updated: