Robots.txt Generator
Create a robots.txt file with rules per crawler, sitemap links and presets for WordPress or blocking AI bots, then test any URL against it in your browser.
Runs in your browser · Nothing uploaded · No sign-up
Something not working, or missing a feature? Send feedback
About the Robots.txt Generator
The Robots.txt Generator builds the robots.txt file that tells crawlers which parts of your site they may fetch. Start from a preset (allow everything, WordPress, block AI training bots, or block a staging site), add rules per crawler and your sitemap, then test any URL against the result: the tester applies Google's matching rules and names the exact line that allows or blocks it.
Files and tags are generated in your browser; nothing you enter, including unpublished URLs and titles, is sent to a server.
How to use the Robots.txt Generator
- Pick a preset, or keep the default that blocks AI training crawlers but allows search engines.
- Edit the groups: add crawlers by name (Googlebot, GPTBot, *) and Allow or Disallow paths such as /admin/ or /*.pdf$.
- List your sitemap URLs, then copy or download the file and upload it as /robots.txt at the root of your domain.
- Type a URL and a crawler in the tester to see whether it is allowed and which rule decided.
Common mistakes
- Blocking CSS and JavaScript. Google renders pages like a browser. If robots.txt blocks /assets/ or /_next/, it cannot see the layout or content those files load, which can hurt how the page is understood.
- Expecting the * group to cover every crawler. A crawler that has its own group ignores the * group completely. If you add a Googlebot group, repeat the * rules in it that should still apply to Google.
- Using robots.txt to hide private pages. robots.txt is public and anyone can read it, so listing /secret-admin/ advertises it. Protect private areas with authentication instead.
Works well with the Robots.txt Generator
- XML Sitemap Generator: create the sitemap the Sitemap: line points to.
- Regex Tester: work out wildcard path patterns.
- Meta Tag Generator: add noindex to pages you want out of results.
Frequently asked questions
Does Disallow keep a page out of Google?
No. Disallow stops crawling, not indexing: a blocked URL can still appear in results if other pages link to it. To keep a page out, allow crawling and add a noindex robots meta tag, or protect it with a password.
How do I block ChatGPT and other AI crawlers?
Add a group for each AI crawler with Disallow: /. GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended collect training data; OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for AI search answers. The "Block AI training" preset blocks only the first kind, so your pages can still be cited in AI search.
Where do I put robots.txt?
At the root of each host: https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain (blog.example.com) needs its own.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay and adjusts its crawl rate automatically. Bing and Yandex respect it.
Related tools
XML Sitemap Generator
Generate an XML sitemap from a list of URLs or paths: duplicates removed, special characters escaped, lastmod dates added and big lists split into an index.
Schema Markup Generator
Generate JSON-LD schema markup for FAQ, Article, Product, Organization, Local Business, Breadcrumb and Event pages, with checks for the fields Google requires.
Open Graph Generator
Generate Open Graph and Twitter Card meta tags and preview how your link looks on Facebook, X, LinkedIn, WhatsApp and Slack before you share it. Copy the HTML.
SERP Preview Tool
Preview your page title and meta description in Google desktop and mobile results. Pixel-width checks show where Google cuts the text off before you publish.
UTM Builder
Build campaign URLs with UTM parameters for Google Analytics: presets for ads, social and email, bulk links for many pages, and a parser for existing links.