Does this fetch my live robots.txt file?
No. Paste the content you want to test; all evaluation stays in your browser.
Desenvolvimento
robots.txt Tester
Check a crawler and path against pasted robots.txt rules without uploading site configuration.
Uses specific user-agent groups and longest matching Allow or Disallow rule precedence.
✓ As entradas das ferramentas são processadas no seu navegador.✓ Sem conta
Test crawler access against robots.txt rules locally.
When to use this robots.txt tester
Use it before deploying crawl rules or while investigating a blocked path.
- Check crawler access to a path.
- Verify an Allow exception beneath a blocked folder.
- Inspect wildcard and sitemap directives privately.
Ferramentas relacionadas
Ferramentas relacionadas
FAQ
Perguntas
Perguntas
How are conflicting rules resolved?
The most specific matching path wins, with Allow winning when matching rules have equal length.
Does this guarantee indexing?
No. robots.txt controls crawling; indexing also depends on other page and crawler signals.