Free tool
Is your site blocking the AI assistants?
Enter your domain and see, bot by bot, which AI crawlers your robots.txt lets in — and which ones can never cite you.
Free, instant, no account. We only read public files on the site you enter.
How do I know if my site is blocking AI crawlers?
To find out whether your site blocks AI crawlers, open yourdomain.com/robots.txt and look for Disallow rules that apply to GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended or Bingbot. A crawler that is disallowed from your site root cannot read your pages, so that assistant can never cite your brand.
Common questions
- How do I know if my site is blocking AI crawlers?
- Check your robots.txt for Disallow rules covering the AI user agents — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bingbot. If any of those is disallowed from your site root, that assistant cannot read your pages and therefore cannot cite you. This tool reads your robots.txt and reports each bot separately.
- Which AI crawlers should I allow?
- Allow the search-and-cite crawlers if you want to be recommended: GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), Google-Extended (Gemini and AI Overviews) and Bingbot (Copilot). Some publishers block training-only crawlers such as CCBot while still allowing the ones that generate citations.
- Does blocking AI crawlers protect my content?
- It prevents those companies from crawling your pages, but it also removes you from their answers entirely. If buyers in your category ask ChatGPT for recommendations, a blocked site cannot appear — a competitor will. It is a trade-off between control and visibility, not a pure safety measure.
- I have no robots.txt at all. Is that a problem?
- Not for access: crawlers treat a missing robots.txt as permission to crawl. Publishing one is still worth it, because it states your intent explicitly and gives you a place to point crawlers at your sitemap.
- My crawlers are allowed but AI still does not mention me. Why?
- Crawler access is necessary, not sufficient. Being reachable gets you eligible to be read; being recommended depends on what the models have learned about you and which sources they trust in your category. That is what an AI visibility audit measures.