Check whether your website's robots.txt file blocks AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. If AI bots can't access your site, they can't recommend your brand. Enter your domain below to see which bots are allowed or blocked.
The checker fetches the live robots.txt at the root of the domain you enter, then tests it against seven AI user-agents: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, anthropic-ai, Google-Extended, and Bingbot. Each bot comes back marked Allowed or Blocked. A green Allowed means that crawler is free to read your pages. A red Blocked means a rule (either a bot-specific Disallow: / or a wildcard one that the bot does not override) is keeping it out.
If any bots show Blocked, the tool also prints a recommended robots.txt you can copy. It opens access to the wildcard agent and every AI user-agent above, and ends with a Sitemap line pointing at your domain so crawlers can find your URL list.
When someone asks ChatGPT, Gemini, Claude, or Perplexity for a recommendation, the assistant answers from content it has been allowed to crawl plus, for the browsing and search modes, pages it can fetch in the moment. A blanket User-agent: * / Disallow: /, or an AI-specific block, takes your site out of that pool. You are not penalized, you are simply absent, and the assistant fills the gap with whoever it can read instead, often a competitor.
Plenty of sites block these agents by accident. A privacy default in a CMS, a copied robots.txt from a template, or a well-meaning "keep AI off our content" decision can all quietly hide a brand from the surfaces where buyers now start their research. The fix is rarely all-or-nothing: allow the AI user-agents you want recommending you, and disallow only the specific paths that genuinely should stay private.
Disallow: /private-path/ rather than a site-wide block.It checks seven user-agents: GPTBot (ChatGPT training), ChatGPT-User (ChatGPT browsing), PerplexityBot (Perplexity), ClaudeBot and anthropic-ai (Claude), Google-Extended (Google AI and Gemini), and Bingbot (Copilot via Bing). Each is reported as Allowed or Blocked based on what your live robots.txt actually returns.
No. Blocking Google-Extended or GPTBot does not improve your Google Search rankings, and it does not stop your content from being scraped by parties that ignore robots.txt. What it does is remove you from the data these assistants draw on when they answer buyer questions, so you lose the chance to be cited.
At the root of your domain, reachable at yourdomain.com/robots.txt. Crawlers only read the file at that exact path. A robots.txt in a subfolder, or one that returns a 404, is treated as no rules at all.
Crawler traffic is lightweight relative to normal page views, and allowing AI user-agents does not change how human visitors or Google see your site. If a specific path (a staging area, a private export) should stay private, disallow that path only, not the whole site.
AI crawlers re-read robots.txt on their own schedule, usually within days. Re-run this checker after you deploy a new file to confirm the live version reflects your change.
Pair this with the other free AI visibility tools: generate an llms.txt so crawlers know what your site is about, then run the AI Search Simulator to see how the models answer about your category. When you want continuous tracking of where you stand across ChatGPT, Gemini, Claude, and Perplexity, run a free Scorecard or compare plans on pricing.