What this tool does
Robots.txt Generator builds multi-bot crawl rules with presets, path libraries, crawl delay, and multiple sitemaps.
← All Tools
🤖 Robots.txt Generator
Build robots.txt with presets, multi-bot rules, path libraries, sitemaps, and crawl delay — all in your browser.
Why it is useful
It helps site owners publish clear crawler guidance for SEO, staging, e-commerce, and AI-bot policies.
How it works
The browser assembles User-agent blocks from selected bots, allow/disallow paths, optional crawl delay, and sitemap lines.
Best input
Start from a preset, then refine paths for your admin, cart, search, and private areas.
Security tip
robots.txt is public. Use authentication to protect private content — not Disallow alone.
Privacy note
Robots.txt generation happens locally in your browser.
Important limitation
Crawlers may ignore robots.txt. It is guidance, not enforcement.
Quick answers
Does Disallow hide pages from Google?
No. It asks well-behaved crawlers not to fetch paths; pages can still appear if linked elsewhere.
What does block AI bots do?
It adds explicit Disallow: / groups for common AI training crawlers like GPTBot and CCBot.
Common Use Cases
Ship a public marketing site robots.txt
Use the open preset, add your sitemap, and disallow /admin and /api private routes.
Protect staging from indexing
Use the staging preset to disallow all crawlers while the environment is private.
Limit AI training crawlers
Enable Force-block known AI bots for sites that want to opt out of common scrapers.