Test whether a URL is allowed or blocked by robots.txt for any bot. Paste robots.txt content or fetch it from a domain and see exactly which rule applies.
Enter a hostname (e.g., google.com) — we'll fetch its robots.txt automatically.
Full URL or path (e.g. /admin/settings)
The robots.txt file is part of the Robots Exclusion Protocol (REP), a standard that allows website owners to communicate with web crawlers and other bots about which areas of their site should or should not be accessed.
Located at the root of your domain (e.g., https://example.com/robots.txt), this plain-text file contains directives that control crawler behavior:
All major search engines (Google, Bing, Yandex) and well-behaved crawlers respect robots.txt rules. However, it is advisory — not enforceable — so malicious bots may ignore it entirely.
The tool provides two ways to supply a robots.txt file: Live Fetch (enter a domain and we fetch its robots.txt for you) and Editor Mode (paste or type the content directly). Once you have the content, enter a URL, pick a bot from the dropdown, and click Test. Parsing happens automatically — there is no separate parse step.
Under the hood, the matching engine works as follows:
*) and end-of-URL anchors ($) are fully supported.If no rule matches the URL, it is allowed by default. After the test, you will see a clear Allowed / Blocked verdict, the exact matching rule, a diagnostics panel, and an expandable view of all parsed rules with the matched group highlighted.
A robots.txt file controls which web crawlers and bots can access specific parts of your website. It uses directives like User-agent, Disallow, and Allow to define rules per crawler. While most well-behaved crawlers respect these rules, robots.txt is advisory — it does not enforce access restrictions at the server level.
Robots.txt can prevent Googlebot from crawling pages, but it does not prevent indexing. If other pages link to a blocked URL, Google may still index the URL with limited information. To fully prevent indexing, use a "noindex" meta tag or X-Robots-Tag HTTP header instead.
If a website does not have a robots.txt file (returns a 404), crawlers assume they are allowed to access all pages. A missing robots.txt is not an error — it simply means there are no crawling restrictions in place for that domain.
The Disallow directive tells a crawler not to access a specific URL path. For example, "Disallow: /private/" blocks crawlers from accessing any URL starting with /private/. An empty Disallow (Disallow:) means nothing is blocked for that user-agent group.
Robots.txt rules only affect bots that choose to respect them. Major search engines like Google, Bing, and Yandex follow robots.txt directives. However, malicious bots, scrapers, and some AI crawlers may ignore robots.txt entirely. For strict access control, use server-side authentication or firewall rules.
Yes. You can target specific crawlers using their User-agent identifier. For example, "User-agent: GPTBot" followed by "Disallow: /" blocks OpenAI's GPTBot from your entire site while leaving other crawlers unaffected. Use our tester to verify which rules apply to which bots.
When both Allow and Disallow rules match a URL, the most specific (longest path) rule wins. For example, "Disallow: /docs/" blocks /docs/ but "Allow: /docs/public/" would permit crawling of that subfolder. If the paths are the same length, Allow takes precedence.
Google and some crawlers support wildcard patterns: * matches any sequence of characters, and $ marks the end of a URL. For example, "Disallow: /*.pdf$" blocks all URLs ending in .pdf. Not all crawlers support wildcards — they are an extension beyond the original robots.txt specification.
There are three simple steps: (1) provide a robots.txt — either switch to Live Fetch and enter a domain, or use Editor Mode to paste the content directly; (2) enter the URL you want to check and select a bot from the dropdown (or choose "Custom User-Agent" to type any user-agent string); (3) click Test. The tool automatically parses the robots.txt and shows whether the URL is allowed or blocked, which rule matched, and a breakdown of all parsed directives.
Yes. In the bot dropdown, the second option is "Custom User-Agent." Select it and a text field will appear where you can type any user-agent string. This is useful for testing rules for bots that are not in our built-in list or for verifying how your robots.txt handles an arbitrary crawler name.
Validate robots.txt syntax with our robots.txt validator. Check for errors, warnings, and best practices.
Build a custom robots.txt file with our visual builder. Select bots, define rules, and download.
Instantly view and analyze any website's robots.txt right from your browser.
Verify if a bot is legitimate by checking its IP, user-agent, and DNS records.