AI Crawler Access Checker

Can ChatGPT, Claude, Perplexity and Google’s AI actually read your site? Check your robots.txt, llms.txt, sitemap, 404 behavior, firewall and brand entity in one pass - free, no sign-up.

We fetch a few fixed pages from your site (robots.txt, llms.txt, sitemap.xml, the homepage and one made-up URL that tests your 404s) with our published fetcher, RankedContentBot, and look your domain up on Wikidata. Results are cached for a day and the URL is shareable.

What this tool checks

robots.txt, per AI crawler

16 AI crawlers - from GPTBot, ClaudeBot and PerplexityBot to Applebot-Extended and meta-externalagent - evaluated using the longest-match rules real crawlers use (RFC 9309).

llms.txt

Determines if your site includes the emerging llms.txt convention that gives AI systems a curated overview of your site.

Firewall challenges

Detects if your host, CDN or firewall answers automated readers with a challenge instead of your content - the silent way sites disappear from AI answers.

404 hygiene

Fetches a made-up URL to test that nonexistent paths answer a real 404 - soft 404s let junk pollute what AI engines index and cite.

Sitemap

Checks /sitemap.xml (and robots.txt declarations) for a valid XML sitemap so crawlers can discover every page you want read.

Markdown for agents

Tests the emerging content-negotiation convention: does your homepage answer Accept: text/markdown with a markdown version for AI agents?

Brand entity

Looks your domain up on Wikidata - the machine-readable entity layer behind Wikipedia, one of the heaviest grounding sources for AI answers - and reports if a verified entity binds your brand to your site.