robots.txt Tools

Forget the limited tool provided by Google Webmaster Tool (which, by the way, since May 2012 dropped support for the robots.txt editor):
Visual SEO Studio sports a powerful viewer, tester and editor for robots.txt files.

• A full-featured viewer/editor with syntax highlighting
• Drag'n'Drop path editing from a folder tree view
• Cross search engine syntax checker and diagnostics
• Multi-agent tester
• Looks for Crawl-Delay presence (Google ignores it, but others don't and it is important to have it in place!)
• BOM issues Diagnostics (your directives might be ignored even if the syntax is correct!)
• and more...
• Checks for Sitemaps list.
• Correctly supports multi robots.txt per domain (every used sub-domain is supposed to have one!)
• Reports robots.txt HTTP response code issues that might affect your indexing or crawl performance.

This feature is not ready for prime time and only partially published in the current version.

See also:

• Help page Manual: View stored robots.txt.