View stored robots.txt

This viewer permits to keep under control all robots.txt files downloaded within the current crawl session.

List of robots.txt files

The table lists all robots.txt files stored for the current crawl session.
Bots access for each root domain and each subdomain (i.e. each "Authority") involved is regulated by its own robots.txt file; missing robots.txt files are stored as HTTP 404 "Not Found" requests, which according to the protocol mean there are not restriction for bots crawling the domain/subdomain.
Other HTTP status codes are considered as errors and thus prevent bots crawling the domain/subdomain resources.
Clicking on a table row the related robots.txt file will be selected and its content shown in the viewer.

Icon

The icon column gives an indication on the state of the robots.txt file.
Used icons are:

• The robots.txt file was successfully downloaded.
  Crawlers are supposed to obey to its directives when accessing the site resources.
• The robots.txt file is not present for the current domain/subdomain ("Authority").
  A missing robots.txt file is a valid state, it means there are no access restrictions set for spiders.
• A HTTP status code not expected in the robots.txt protocol occurred when attempting to download the file.
  According to the protocol, this should prevent crawlers to access the domain/subdomain ("Authority") resources.

Authority

The root domain or the subdomain where the robots.txt file was found.
Bot access to each authority is regulated by the robots.txt file it contains.

Authority

The selected root domain or subdomain, where the robots.txt file was found.
Bot access to the authority resources is regulated by the robots.txt file it contains.

Download Date

The date when the robots.txt file was downloaded.

Status Code

The textual description of the HTTP response code received from the web server upon requesting the robots.txt file.

Viewer

Displays the text extracted from selected robots.txt file.