robots.txt Examples: Disallow Rules for Every Common Case

A robots.txt file is short, public and easy to get wrong. One misplaced slash can hide a whole section from Google, and a rule with a full URL quietly does nothing. This post collects the robots.txt examples people actually need, explains exactly what each Disallow line matches, and lists the mistakes we see most often.

Every example below was run through our Robots.txt Checker in Paste mode on 9 November 2026: no syntax errors, and the blocked and allowed paths are the ones the checker reports. Paste your own file there to test it the same way before you upload it.

How a robots.txt file is read

The file lives at the root of each host: https://example.com/robots.txt. A file in a subfolder is ignored, and blog.example.com needs its own. It is made of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Three rules decide what a crawler does:

1. A crawler follows one group only: the one whose User-agent matches it most specifically. If there is a User-agent: Googlebot group, Googlebot ignores the * group completely.
2. A rule matches from the start of the path. Disallow: /admin blocks /admin, /admin/ and also /administrator.
3. The longest matching rule wins. If an Allow and a Disallow both match, the longer (more specific) path decides. On a tie, Allow wins.

Sitemap lines sit outside the groups and apply to every crawler. Google's own description of these rules is in How Google interprets the robots.txt specification.

Copy-ready robots.txt examples

Replace example.com with your own domain in each one.

Allow everything

The right file for most public sites. An empty Disallow blocks nothing:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Having no robots.txt at all (a 404) has the same effect, but the file is still worth it for the Sitemap line.

Disallow all crawlers

For a staging copy or a site that is not ready yet:

User-agent: *
Disallow: /

This stops crawling, not indexing. If other sites link to a blocked page, Google can still list the URL without a description. To keep a staging site out of search, put it behind a password; to remove a public page, allow crawling and add a noindex robots meta tag, as our HTML meta tags guide explains.

Block one folder, with an exception

User-agent: *
Disallow: /admin/
Allow: /admin/help

Sitemap: https://example.com/sitemap.xml

/admin/users is blocked, while /admin/help is allowed because the Allow rule is longer than the Disallow rule. Note the trailing slash: Disallow: /admin/ does not block the page /admin itself.

WordPress

WordPress serves this virtual file when there is no real one (the Sitemap line points to the core sitemap added in WordPress 5.5):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

It is a good file as it is. Do not add Disallow: /wp-includes/ or /wp-content/: themes and plugins load their CSS and JavaScript from there, and Google needs those files to render the page. If an SEO plugin creates its own sitemap, point the Sitemap line at that one instead.

Block internal search and filter URLs

Search results and sort or filter parameters create endless near-duplicate URLs that waste crawl time:

User-agent: *
Disallow: /search
Disallow: /*?s=
Disallow: /*?*sort=

* matches any characters, so /*?s= blocks /?s=shoes and /blog/?s=shoes, and /*?*sort= blocks any URL whose query string contains sort=. Be careful: Disallow: /search also blocks /searching-for-a-job, because rules match prefixes.

Block a file type

User-agent: *
Disallow: /*.pdf$

$ anchors the end of the URL, so /files/report.pdf is blocked but /report.pdf?download=1 is not, and neither is /report.PDF: paths are case-sensitive.

Block AI training crawlers, keep AI search

AI companies run separate bots for training and for search. This file blocks the main training crawlers while search engines and AI search bots (OAI-SearchBot, PerplexityBot and others) stay allowed:

User-agent: *
Disallow:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row share one group. Our study of the top 1,000 sites found this "block training, allow search" policy is the most common among sites that block AI crawlers at all. The AI Crawler Checker lists what each token does.

Different rules for one crawler

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /private/
Disallow: /print/

Because Googlebot follows only its own group, /private/ has to be repeated there. Leave it out and Googlebot may crawl /private/ even though every other crawler is blocked.

What a Disallow rule matches

Disallow: /private* is the same as Disallow: /private: a trailing * adds nothing.

Common robots.txt mistakes

• Full URLs in rules. Disallow: https://example.com/admin/ matches nothing. Rules take a path that starts with /.
• Blocking CSS and JavaScript. Rules like Disallow: /assets/ or /_next/ can make pages look empty to Google, which renders them like a browser.
• noindex in robots.txt. Google stopped supporting it in September 2019. Use the robots meta tag or the X-Robots-Tag header.
• Expecting Crawl-delay to slow Google. Googlebot ignores it. Bing and some other crawlers read it.
• Hiding secrets. robots.txt is public. Listing /secret-admin/ advertises the path; protect it with a login.
• Serving an HTML page. Some servers answer /robots.txt with the home page or a 404 template and status 200. Crawlers find no rules in it.
• A server error. If robots.txt returns a 5xx status, Google treats the whole site as blocked until the file answers normally again.

Where to edit robots.txt on Wix and Shopify

On Wix, open SEO in the site dashboard, then the Robots.txt Editor under Tools and settings. On Shopify, go to Online Store, Themes, Edit code, and add the robots template (robots.txt.liquid); Shopify recommends adding rules with Liquid so its defaults keep updating. On other hosts, upload a plain text file named robots.txt to the web root.

Test before and after you upload

Google retired the Search Console robots.txt Tester in 2023. Its robots.txt report shows the file Google last fetched, but it cannot test a draft or a single URL. A quick routine that covers both:

1. Build the file by hand from the examples above, or with the Robots.txt Generator.
2. Paste it into the Robots.txt Checker and test your important URLs for Googlebot and Bingbot.
3. Upload it, then run the checker again in Live site mode to confirm the server returns status 200 and plain text.

A robots.txt that passes those checks rarely needs attention again until your site structure changes.