Technical SEO
Robots.txt Generator
Generate a valid robots.txt in seconds — with the critical "block everything" warning built in.
Build your robots.txt
Defaults are already filled in — edit what you need, then copy or download the file.
- Opens full-screen in a new tab — no distractions
- This page stays open, so you can keep the guide handy
- The tool runs entirely in your browser
One wrong line in robots.txt can hide your whole site from Google — the classic mistake is a stray "Disallow: /" that no one notices for weeks. This generator writes clean rules for allow/disallow paths, crawl-delay and your sitemap, and warns you loudly before you copy something dangerous. Built by the studio that audits sites for a living, and it shows the same checks our free website audit runs on your live file.
How it works
- Pick the bot (or all bots), add allow and disallow paths.
- Paste your sitemap URL so crawlers find it.
- Copy or download the file and upload it to your site root.
Questions people ask
Where do I upload robots.txt?
At the very root of your domain — https://yourdomain.com/robots.txt must return it. In WordPress you can also use an SEO plugin to edit it, or ask your host; if we build your site, we handle this for you.
What is the most dangerous mistake in robots.txt?
"Disallow: /" under "User-agent: *" — it tells every search engine to skip your entire site. The generator detects this and shows a critical warning before you copy it.
Does robots.txt remove pages from Google?
No — it blocks crawling, not indexing. Pages can still appear without content. To truly remove a page you need a noindex meta tag or a removal request.
More free tools
Tools fix one thing. The audit checks everything.
Run the free website audit and get a 16-point report on your site's speed, SEO and security — the same checks we run before every rebuild.
Run the Free Audit