Robots.txt Tester - Check Which URLs Are Blocked for Googlebot and AI Bots
Paste your robots.txt, list the URLs you want to check, and pick the crawlers. The tester applies Google's matching rules (longest match, Allow wins a tie, * and $) and shows the rule and line that decided each result. Your file stays in your browser.
- Free
- Runs in your browser
- No file upload
UTF-8 text, up to 500 KiB. Google ignores content after 500 KiB, so only that part is tested and a warning is shown. The file is read locally and never uploaded.
One per line, up to 500. A full URL is reduced to its path and query. The host is not compared, so use the robots.txt of the same site.
Ready. Nothing leaves your browser.
Results
| URL or path | Path tested | User agent | Result | Deciding rule | Rule line | Group lines | Reason |
|---|
Warnings
Parsed groups and sitemaps
Sitemap lines
Results follow Google's documented matching. Review the rules against real crawl behavior before relying on them.
How to test robots.txt rules
Add robots.txt
Paste the file from your site, or choose a robots.txt or .txt file. The text is read locally in your browser.
List URLs and paths
Enter full URLs or paths, one per line. Query strings are compared, fragments are ignored.
Choose user agents
Tick the crawlers to test, or type a custom product token. Each URL is checked against each agent.
Click Run test. Read the Result and Deciding rule columns: the rule line tells you where the decision comes from in your file. Use Copy as TSV or Download CSV to keep the results.
How Google matches robots.txt rules
The tester follows the rules Google documents for its crawlers and the open-source Google parser they are based on. The decision for one URL and one crawler comes from these steps:
Groups. Consecutive user-agent lines form one group. Groups that name the same product token are merged, so their rules apply together. Rules before the first user-agent line are ignored.
Agent choice. A group that names the crawler's product token is used. If none does, the * group is used. If neither exists, nothing is blocked. A group for a related name, such as googlebot-news for Googlebot, does not apply to Googlebot.
Path matching. A rule matches when its path is a prefix of the URL path and query, and matching is case-sensitive. * matches any run of characters. A $ at the end of a rule anchors it to the end of the URL.
Longest match. The matching rule with the most characters decides. When an Allow and a Disallow tie, Allow wins. /robots.txt itself is always allowed.
| URL path | Decision | Why |
|---|---|---|
| /folder/other.html | Blocked | Only the Disallow matches. |
| /folder/page.html | Allowed | The Allow rule is longer than the Disallow rule. |
| /folder | Blocked | The Disallow matches the whole path. The Allow path is longer than this URL, so it does not match. |
| /Folder/page.html | Allowed | Paths are case-sensitive, so no rule matches. |
Blocking AI crawlers with robots.txt
Give each AI crawler its own group and disallow the paths you want to keep out. A group with Disallow: / blocks the whole site for that token:
User-agent: GPTBotDisallow: /
| Token | Used for |
|---|---|
GPTBot | OpenAI crawling for model training. |
OAI-SearchBot | OpenAI crawling for ChatGPT search results. |
ChatGPT-User | Fetches triggered by a ChatGPT user. |
ClaudeBot | Anthropic crawling. |
PerplexityBot | Perplexity crawling for search. |
CCBot | Common Crawl, a dataset that AI training often uses. |
Google-Extended | A control token. Blocking it stops use of your content for Google's AI models, and does not block Googlebot. |
Applebot-Extended | A control token for Apple's AI features. It does not block Applebot search crawling. |
robots.txt is voluntary. Operators that document support for it say they follow it, but the file cannot enforce anything, and a crawler that ignores it is not stopped by the file.
When to use this robots.txt tester
Check a change before you deploy it: paste the new robots.txt and confirm that key URLs stay allowed and that private paths are blocked for Googlebot.
Find the rule behind a surprising result. The line number points to the directive in your file, and the group lines show which user-agent block applied.
Confirm an AI crawler block. List a few article and product URLs, add GPTBot and ClaudeBot, and check that the expected ones are blocked.
Review a robots.txt you inherited from a template or a plugin, including warnings about directives that Google ignores.
Limitations
The tester checks rules against paths you give it. It does not replace a live crawl or a search console report. Keep these limits in mind:
No fetching. The page cannot download a robots.txt, by design. Copy the file from the live site, and make sure you are copying the version served for the host you care about.
Host and scheme. Rules apply only to the host, scheme and port that serve the file. Full URLs are reduced to path and query, so a URL on another host is compared as if it were on the same host.
Crawling, not indexing. robots.txt controls crawling. A blocked URL can still appear in search results without a snippet if other pages link to it.
Other crawlers. The rules follow Google. Other search engines and AI crawlers may read the same file differently or support different tokens.
Percent-encoding. Encoded unreserved characters such as %7E and %41 are decoded before matching, as RFC 9309 asks, so /%7Euser and /~user match the same rules. Non-ASCII paths are compared in UTF-8 percent-encoded form. Google's open-source parser compares an encoded URL literally, so a rule written with %7E can behave differently there.
robots.txt Tester FAQ
Is my robots.txt uploaded?
No. The file, the URL list and the results are handled in your browser. The page does not send them to a server, and nothing is stored in local storage.
Does the tool fetch my site's robots.txt?
No. Paste the file you see at https://yoursite.example/robots.txt. The tester does not check the host, so the file must be the one that applies to the URLs you enter.
Why is a URL allowed when I expected it blocked?
Look at the group lines column. If a group names the crawler, that group is used and the * group is ignored. Within the group, the longest matching rule decides. The rule line column points to the rule that decided the result.
Does Googlebot follow Crawl-delay?
No. Google ignores Crawl-delay. The tester lists it as a warning, and the line is not used for any decision.
Does robots.txt keep a page out of search results?
No. robots.txt controls crawling. To keep a page out of the index, leave it crawlable and add a noindex robots meta tag or an X-Robots-Tag header.
Can I block GPTBot and other AI crawlers with robots.txt?
You can add a group for each token with Disallow: /. Operators that honor robots.txt follow it, but the file is voluntary and cannot stop a crawler that ignores it.
Related tools
- .htaccess to Nginx Converter translates Apache .htaccess rules into Nginx configuration.
- Regex Tester checks a regular expression against sample text before you use it in a rule or script.
- Cron Expression Explainer describes a cron schedule in plain language, which helps when you plan when a crawl or job runs.