Skip to content
Devtoolry
Sign InJoin for Free

AI Crawler Checker

Check which AI crawlers a site allows: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 12 more, split into AI training, AI search and search engines.

Something not working, or missing a feature? Send feedback

About the AI Crawler Checker

The AI Crawler Checker shows which AI bots may crawl a site. It reads the site's robots.txt, applies Google's matching rules (the most specific user-agent group, longest path wins) for 14 AI crawlers and both major search engines, and sorts them by what blocking them does: AI training crawlers such as GPTBot, ClaudeBot and CCBot, AI search crawlers such as OAI-SearchBot and PerplexityBot that cite pages in answers, and user-triggered fetchers. It also reports Content-Signal lines, llms.txt and page-level robots directives.

Our server requests the URLs you enter, like a crawler: it reads the response without running the page's scripts, and the analysis runs in your browser. The URLs and results are not stored.

How to use the AI Crawler Checker

  1. Enter a site or page address; the check applies to that path.
  2. Read the verdict, then each group: Blocked shows the exact rule and user-agent group that decided.
  3. Decide what you want: many sites block training crawlers but keep AI search crawlers so they are still cited.
  4. Change the rules with the Robots.txt Generator and run the check again.

Works well with the AI Crawler Checker

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and pages a user asks for are fetched by ChatGPT-User. Block only GPTBot to stay out of training while remaining citable in ChatGPT search.

Does Google-Extended affect my Google rankings?

No. Google-Extended is a robots.txt token, not a separate crawler: Googlebot reads it to decide whether content may be used for Gemini models. Google says it does not affect inclusion or ranking in Google Search.

Is robots.txt enough to stop AI crawlers?

Only for crawlers that respect it. Reputable operators publish their user agents and follow robots.txt; others ignore it. To enforce a block, use your firewall or CDN bot settings, which can also block crawlers that robots.txt allows.

Guides and tutorials

Related tools