More
More
Robots.txt & Sitemap Generator
A robots.txt and an XML sitemap for your site.
Free tools that run in your browser. Nothing you type, open or create is uploaded to NerdBible or anyone else.
Test a URL against these rules
Optionally add a date after an address, separated by a space, comma or tab: https://example.com/about 2026-09-30
The tester follows the rules Google publishes: the crawler uses the most specific group that names it, otherwise the * group; the longest matching rule wins, and Allow wins a tie. * matches any characters and $ marks the end of the address. Other crawlers may read edge cases differently. robots.txt asks crawlers to stay out; it does not hide or protect a page.
How to use it
- On the robots.txt tab, start from a preset or add Allow and Disallow rules for all crawlers or for named ones, and list your sitemap.
- Test a few addresses against the rules, then copy or download the file and upload it to the root of your site, for example example.com/robots.txt.
- On the XML sitemap tab, paste your page addresses, add dates if you have them, and download the sitemap. Large lists are split into files of 50,000 addresses with an index.
Questions
Does Disallow stop a page appearing in Google?
Not reliably. Disallow stops crawling, but Google can still list a blocked address it finds through links, without a description. To keep a page out of results, allow crawling and add a noindex robots meta tag instead.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay. Bing and Yandex read it. If Googlebot is putting too much load on your server, reduce its crawl rate by returning 503 or 429 responses for a short time, as Google's own documentation suggests.
How big can a sitemap be?
Each file may hold up to 50,000 addresses and be up to 50 MB uncompressed. For more, use several sitemap files and a sitemap index that lists them, which this tool creates for you.
Do I need changefreq and priority?
No. Google ignores both. An accurate lastmod date, changed only when the content really changes, is the one optional field Google uses.
About this tool
About the robots.txt and sitemap generator
Two small files help search engines understand a website. robots.txt tells crawlers which parts of the site they may visit, and an XML sitemap lists the pages you want found. This tool writes both.
robots.txt
- Start from a preset or add Allow and Disallow rules for all crawlers or for named ones.
- List your sitemap address so crawlers find it.
- Test a few addresses to see which rule applies, following the matching rules Google publishes: the most specific group wins, the longest matching rule wins, and Allow wins a tie.
- Upload the file to the root of your site, for example example.com/robots.txt.
XML sitemap
Paste your page addresses, add last-modified dates if you have them, and download. Lists over 50,000 addresses are split into several files with a sitemap index, as the format requires. Google ignores changefreq and priority; an accurate lastmod is the optional field it uses.
Common misunderstandings
Disallow stops crawling, not indexing: Google can still list a blocked address it finds through links. To keep a page out of results, allow crawling and add a noindex tag instead. robots.txt is public and does not protect anything. Google also ignores Crawl-delay, though some other search engines read it.
Both files are built in your browser.
Related tools
- Meta Tag Generator & Social PreviewTitle, description and social tags for a page, with Google, Facebook, LinkedIn and X previews.
- Server Config Generator: Redirects & Security HeadersRedirect rules and security headers for Apache, Nginx, Netlify and Vercel.
- HTTP Status CodesWhat every HTTP status code means.
- Favicon GeneratorEvery favicon size from one image.
- URL ParserSplit a URL into its parts and query parameters.