As more people get answers directly from AI chat assistants like ChatGPT, Claude, and Perplexity instead of clicking through search results, whether your website is accessible to these AI crawlers has become just as important as traditional search engine indexing.

This free tool fetches and parses your site's robots.txt file and checks it against the current list of known AI crawler user-agents, so you can see at a glance who's allowed in and who's blocked — and fix it if something's blocked by accident.

Which crawlers does this check?

• GPTBot & OAI-SearchBot — OpenAI (ChatGPT training and search)
• ClaudeBot & Claude-SearchBot — Anthropic (Claude)
• PerplexityBot — Perplexity
• Google-Extended — Google's AI training opt-out token
• Applebot-Extended — Apple Intelligence
• Bytespider — ByteDance / TikTok
• CCBot — Common Crawl, used to train many AI models
• Amazonbot — Amazon
• ...and several more

This tool also checks whether your site has an llms.txt file, a newer standard that helps AI systems understand your site's structure.