Who Blocks AI Crawlers? A robots.txt Study of the Top 1,000 Sites (2026)

Every site owner now has to decide what to do about AI crawlers. Some block all of them, some block none, and many block the bots that collect training data while letting in the ones that cite pages in AI answers. We wanted numbers instead of opinions, so on 4 November 2026 we downloaded the robots.txt files of the 1,000 most popular domains on the web and checked, bot by bot, which AI crawlers each one lets in.

The short version: about one in three top sites blocks at least one AI crawler, training crawlers are blocked almost twice as often as AI search crawlers, and llms.txt has already reached one in nine top domains.

How we ran the study

We took the top 1,000 domains of the Tranco list (list 746WX, generated on 3 November 2026), a ranking widely used in web measurement research because it combines several traffic sources and is hard to manipulate.

For each domain we requested https://<domain>/robots.txt with an identifying user agent and followed redirects. A file counted only if it answered 200 with plain text: login walls, captcha pages and HTML error pages served at /robots.txt were discarded. That left 496 domains with a readable robots.txt. The rest are mostly infrastructure domains without a website (CDNs, ad servers, API hosts), sites that refused automated requests, or sites that redirect robots.txt to their home page.

We then asked the same question for 14 AI crawlers and two search engines: may this bot fetch the home page (/)? The answer comes from the robots.txt matcher in our AI Crawler Checker, which follows Google's rules: a crawler obeys the most specific user-agent group that names it (or * if none does), and the longest matching path wins. We also recorded Content-Signal lines and whether /llms.txt exists.

Two limits are worth knowing. We only looked at the home page path, so a site that blocks AI bots from /articles/ but not from / counts as allowing them. And robots.txt is a public statement of intent: sites can also block bots in their firewall, and can answer different user agents with different files.

One in three top sites blocks at least one AI crawler

Of the 496 sites, 152 (30.6%) block at least one of the 14 AI crawlers from their home page. 37 of those block every crawler they do not name (User-agent: * with Disallow: /), usually because the domain is not meant to be crawled at all. Leaving those out, 116 of 459 sites (25.3%) single out AI crawlers by name.

Bigger sites block more often. Among the top 100 domains with a readable robots.txt, 37.5% block at least one AI crawler; for ranks 101 to 500 it is 31.8%, and for ranks 501 to 1,000 it is 28.6%.

Blanket blocks are rare. Only 19 sites block all eight training crawlers we checked, which shows how quickly the list of bots has grown: a robots.txt written in 2023 that blocks GPTBot and CCBot misses ClaudeBot, Google-Extended, Applebot-Extended and meta-externalagent.

Training crawlers are blocked far more than AI search crawlers

The two most blocked crawlers are not the famous ones. Bytespider (ByteDance) and CCBot (Common Crawl, whose archive feeds many training sets) top the list, ahead of GPTBot and ClaudeBot.

The clearest pattern is the split by purpose. GPTBot, which collects training data for OpenAI, is blocked by 21.6% of sites; OAI-SearchBot, which fetches pages for ChatGPT search and links back to them, by only 11.5%. Among the 77 sites that block GPTBot by name, 47 (61%) still allow OAI-SearchBot. Anthropic's crawlers show the same pattern: of the 75 sites that block ClaudeBot by name, 38 let Claude-SearchBot in.

On the day we checked, Yahoo, eBay, Medium, Forbes, Canva and Nature were among the sites that block GPTBot but allow OAI-SearchBot. That is the policy most publishers now describe: "do not train on our content, but do send us readers from AI answers." It only works if robots.txt names each crawler separately, because blocking GPTBot does nothing to OAI-SearchBot, and vice versa.

Google-Extended deserves a note. It is not a crawler but a token that Googlebot reads to decide whether content may be used for Gemini. Google says it does not affect Google Search, so the 18.3% of sites that block it have not hurt their rankings.

Search engines were almost never blocked by name

No site blocked Googlebot or Bingbot by name while allowing everything else. The 14 sites that block Googlebot are the same domains that block every crawler with User-agent: *, which fits infrastructure domains that should not appear in search at all. If your robots.txt has AI rules, it is worth checking that a broad rule has not caught search engines too: the AI Crawler Checker shows both side by side.

llms.txt and Content Signals are already out there

113 of the 1,000 domains (11.3%) serve an llms.txt file, the Markdown map of a site proposed for AI assistants in 2024. Stripe, GitHub, Cloudflare, Shopify, PayPal, Dropbox and Slack are among them, which suggests developer platforms see it as cheap documentation for coding assistants. Google has said it does not use llms.txt for Search, so treat it as a courtesy to assistants, not a ranking factor. Our llms.txt Generator builds one from your sitemap.

Content Signals, a robots.txt extension Cloudflare introduced in 2025 to state preferences such as ai-train=no or search=yes, appear on 17 sites (3.4%), 11 of which say ai-train=no. It is early: the signals express intent, and crawlers have to choose to honour them.

Finally, a basic one: 71.4% of the sites list a sitemap in robots.txt. The rest rely on search engines finding it some other way.

What to do with your own robots.txt

1. Decide per purpose, not per company. Blocking training crawlers keeps your content out of future models; blocking AI search crawlers removes you from the answers that send traffic. Most top sites that block anything block the first kind only.
2. Name every bot you mean. A rule for GPTBot does not apply to OAI-SearchBot, ChatGPT-User, ClaudeBot or CCBot. The Robots.txt Generator has a "Block AI training" preset with the current tokens.
3. Check the result. A typo in a user-agent name or an Allow rule in the wrong group silently changes the outcome. Run your site through the AI Crawler Checker after every change.
4. Remember robots.txt is a request. Reputable operators follow it; to enforce a block, use your firewall or CDN bot settings as well.

We plan to repeat this study every few months to see how these policies change. If you want your site's numbers today, the checker takes a few seconds, and the rest of our SEO tools cover sitemaps, redirects and on-page checks.