Build a custom robots.txt file by selecting bots from our directory and defining rules for your website.
Choose bots to create targeted rules. Selected bots will use their official robots.txt identifiers.
Loading bots...
No rules yet. Click "Add Rule" to get started.
Select the AI bots you want to block from our comprehensive list (including GPTBot, ClaudeBot, Google-Extended, CCBot, and more). The builder automatically uses the correct user-agent identifiers for each bot. You can block all AI bots or select specific ones.
Disallow blocks crawlers from accessing specified paths, while Allow creates exceptions within broader Disallow rules. For example, you might disallow /private/ but allow /private/public/ to make a specific subfolder accessible.
Use 'User-agent: *' for rules that apply to all bots. For AI bots or specific crawlers you want to treat differently, use their specific user-agent names. Our builder handles both approaches - select bots from the list for specific targeting.
Use the Sitemaps section in the builder to add your sitemap URLs. You can add multiple sitemaps (e.g., sitemap.xml, sitemap-news.xml, sitemap-images.xml). Format: https://yourdomain.com/sitemap.xml
Crawl-delay sets the minimum time (in seconds) between requests from a crawler. Use it to prevent aggressive crawlers from overloading your server. Note: Major search engines like Google ignore this directive, but many other crawlers respect it.
Yes! Use our Robots.txt Validator to check your existing file, then click 'Enrich with Builder' to import it. You can then modify, enhance, or add more rules using the visual builder interface.
Upload robots.txt to your website's root directory so it's accessible at https://yourdomain.com/robots.txt (not in subfolders). Most web servers and hosting platforms allow you to upload files to the root via FTP, cPanel, or their file manager.
Blocking AI training bots (like GPTBot, CCBot, Google-Extended) does NOT affect your search engine rankings. These are separate from search engine crawlers like Googlebot and Bingbot. You can block AI bots while keeping search engines active.
Each page lists the bot's robots.txt user agent, ready-made rules, and how to verify it is genuine. Links open in a new tab so you do not lose your work.
Test whether a URL is allowed or blocked for any bot, and see exactly which rule applies.
Validate robots.txt syntax with our robots.txt validator. Check for errors, warnings, and best practices.
Instantly view and analyze any website's robots.txt right from your browser.
Verify if a bot is legitimate by checking its IP, user-agent, and DNS records.
Browse every known crawler by operator — user agents, robots.txt configuration, and verification methods.