Skip to content
Free Indian Tools

SEO · Free tool

robots.txt Tester

Test whether a URL is crawlable under your robots.txt — for any user-agent, with the actual matching rule shown.

Result

ALLOW

Matched group: googlebot

Matched rule: no rule matched — default ALLOW

Mirrors Google's matching rules

The tester implements Google's robots.txt spec: longest-match wins, ties go to Allow, paths are case-sensitive, user-agents are not. It also handles wildcards (*) and end-of-line anchors ($) the way Googlebot does.

When to use this

Whenever you ship a robots.txt change, smoke-test the URLs that matter most. Search Console's robots tester was deprecated; this fills that gap. If you're writing the file from scratch, start with the robots.txt generator templates, then validate edge URLs here. The most common failure modes are documented in robots.txt mistakes that hide your site.

Notes on the result

  • "Allow" means crawlable — Google may still index based on other signals.
  • "Disallow" means Google won't crawl the URL; it may still appear in search if linked externally.
  • Set noindex via meta tag for guaranteed removal (after Google re-crawls it).
  • Bingbot uses the same spec as Googlebot for matching.

FAQ

Why does Google sometimes index pages I disallowed?

Disallow blocks crawling, not indexing. Google can still index a blocked URL based on external links and show it with a placeholder snippet. Combine Disallow with noindex (after temporarily allowing the crawl) to fully remove.

Are robots.txt rules case-sensitive?

Path matching is case-sensitive - /Page and /page are different rules. User-agent names are matched case-insensitively. Test both your actual URL casing and Googlebot casing assumptions.

What does Allow do when both Allow and Disallow match?

The more specific rule (longer path) wins. If lengths tie, Allow wins. This lets you Disallow a folder but Allow a specific file inside it.