Robots.txt Checker
Validate a live or pasted robots.txt the way Google reads it: syntax errors, blocked pages and resources, sitemaps, 21 search and AI crawlers, and a URL tester.
Something not working, or missing a feature? Send feedback
About the Robots.txt Checker
The Robots.txt Checker validates a robots.txt file the way Google reads it. In Live site mode it fetches the file from any site and checks the HTTP status (a server error stops Google crawling the whole site), redirects, file type and the 500 KiB limit. In Paste mode it tests a draft in your browser before you upload it. Both modes find syntax mistakes, full URLs in rules, a site-wide block, blocked CSS and JavaScript, and missing sitemap lines. A table shows which of 21 search and AI crawlers may fetch the home page and which rule decides, and a robots.txt tester checks any list of URLs against the file. For a live site, Check indexability also fetches those pages and adds the status code, noindex (meta tag or X-Robots-Tag) and canonical, so you see whether each one can actually be indexed.
Our server requests the URLs you enter, like a crawler: it reads the response without running the page's scripts, and the analysis runs in your browser. The URLs and results are not stored, unless you are signed in and choose Monitor this page: then that page address is kept with your account until you remove it. Paste mode runs entirely in your browser.
How to use the Robots.txt Checker
- Pick Live site and enter any page of the site, or pick Paste robots.txt and paste (or drop) the file.
- Fix the errors first: a 5xx status, an HTML page served as robots.txt or "Disallow: /" for everyone.
- Check the crawler table: every search engine you want should say Allowed.
- Paste URLs or paths into the tester and pick a crawler to see which line allows or blocks each one; choose Check indexability to add status, noindex and canonical.
What a Disallow rule matches
| Rule | Blocks | Does not block |
|---|---|---|
| Disallow: / | Every URL on the host | Nothing (the whole site is blocked) |
| Disallow: (empty) | Nothing | Every URL: an empty value allows everything |
| Disallow: /admin | /admin, /admin/, /admin.html, /administrator | /Admin (paths are case-sensitive), /blog/admin |
| Disallow: /admin/ | /admin/, /admin/users | /admin (no trailing slash) |
| Disallow: /*.pdf$ | /files/a.pdf | /a.pdf?download=1, /a.PDF |
| Disallow: /*? | /shop?sort=price (any URL with a query string) | /shop |
| Disallow: /$ | / (the home page only) | /index.html and every other page |
A rule matches from the start of the path; * matches any characters and $ ends the match. When an Allow and a Disallow both match, the longer (more specific) rule wins, and Allow wins a tie. Rows checked with this tool's matcher on 2026-11-09.
How Google treats the robots.txt response
| Response | What Google does |
|---|---|
| 2xx | Reads the file and obeys the rules |
| 3xx | Follows up to 5 redirects; after that, acts as if there were no robots.txt |
| 4xx except 429 (404, 410, 401, 403) | Acts as if there were no robots.txt: crawls everything |
| 429 or 5xx | Treats the whole site as disallowed and stops crawling; after 30 days uses the last cached copy, or crawls without rules |
| Over 500 KiB | Reads the first 500 KiB and ignores the rest |
Checked against Google Search Central on 2026-11-08. Bing and other crawlers handle some cases differently.
Works well with the Robots.txt Checker
- Robots.txt Generator: write a corrected file.
- Googlebot Simulator: see what the bot gets once it is allowed.
- Sitemap Validator: validate the sitemaps robots.txt lists.
Frequently asked questions
Can I get an alert when robots.txt changes?
Yes. Sign in (free) and choose Monitor this page under a live result. We re-check robots.txt, the status code, noindex and the canonical of that page on a schedule (weekly on the free plan, daily on Pro) and email you when one changes, for example when a deploy adds "Disallow: /" or a noindex tag.
Google removed the robots.txt Tester. What replaces it?
Google retired the Search Console robots.txt Tester in late 2023. Its replacement, the robots.txt report (Settings, then robots.txt), shows the file Google last fetched and any parse errors, but it cannot test URLs or a file that is not live yet. Google points to third-party testers for that: use Paste robots.txt mode here to test a draft, then the URL tester to check single pages.
Does robots.txt keep a page out of Google?
No. It stops crawling, not indexing: a blocked URL can still appear in results (without a description) when other pages link to it. To keep a page out, allow crawling and add a noindex robots meta tag or X-Robots-Tag header.
How do I see the robots.txt of any website?
Open the site's root followed by /robots.txt, for example https://example.com/robots.txt. The file is always at the root of the host, and each subdomain has its own. Live site mode here fetches it for you and shows it with problem lines highlighted.
Where do I edit robots.txt in WordPress, Wix or Shopify?
WordPress serves a virtual file: edit it in your SEO plugin (Yoast and Rank Math both have a robots.txt editor) or upload a real robots.txt to the site root, which replaces the virtual one. Wix: SEO in the site dashboard, then Robots.txt Editor under Tools and settings. Shopify: Online Store, Themes, Edit code, then add the robots template (robots.txt.liquid).
Why does a missing robots.txt pass?
A 404 or 410 means "no rules", so crawlers fetch everything. That is fine for most small sites. A 5xx or 429 is different: Google treats it as "everything is blocked" and stops crawling until the file answers normally again.
Which group does a crawler follow?
Only one: the group whose User-agent matches the crawler most specifically. If there is a "User-agent: Googlebot" group, Googlebot ignores the "*" group entirely, so rules you want for everyone must be repeated there.
Is the checker the same as the generator's tester?
The Robots.txt Generator tests the file you are building with its form. This checker validates any file, live or pasted, including what the server returns (status, redirects, content type), which is what crawlers actually read.
Guides and tutorials
- Web Development
robots.txt Examples: Disallow Rules for Every Common Case
Copy-ready robots.txt examples: allow all, block a folder, WordPress, search pages, PDFs and AI bots, plus how Disallow matching works and the common mistakes.
· 5 min read
- Web Development
Who Blocks AI Crawlers? A robots.txt Study of the Top 1,000 Sites (2026)
We read the robots.txt of the top 1,000 sites: 31% block at least one AI crawler, GPTBot is blocked twice as often as ChatGPT search, and 11% have llms.txt.
· 5 min read
Related tools
Googlebot Simulator
See a page as Googlebot, Bingbot or GPTBot and as a visitor, side by side: status, redirects, title, canonical, noindex and text, to catch cloaking.
Robots.txt Generator
Create a robots.txt file with rules per crawler, sitemap links and presets for WordPress or blocking AI bots, then test any URL against it in your browser.
XML Sitemap Generator
Generate an XML sitemap from a list of URLs or paths: duplicates removed, special characters escaped, lastmod dates added and big lists split into an index.
Website SEO Checker
Check any page for on-page SEO problems: title, meta description, headings, canonical, robots.txt, noindex, structured data, alt text, HTTPS and speed, with a score.
Broken Link Checker
Find broken links on any web page: every internal and external link (and optionally image) is checked for 404s, server errors, timeouts and redirects. Export a CSV.
Redirect Checker
Check the HTTP status code and full redirect chain of up to 100 URLs: 301, 302, 404 and 5xx, response headers, loops and HTTPS downgrades, with fixes and CSV.