Manual: View stored robots.txt
The feature "View stored robots.txt" of Visual SEO Studio, documented in detail.
View stored robots.txt
This viewer permits to keep under control all robots.txt files downloaded within the current crawl session.
List of robots.txt files
The table lists all robots.txt files stored for the current crawl session.
Bots access for each root domain and each subdomain (i.e. each "Authority") involved is regulated by its own robots.txt file; missing robots.txt files are stored as HTTP 404 "Not Found" requests, which according to the protocol mean there are not restriction for bots crawling the domain/subdomain.
Other HTTP status codes are considered as errors and thus prevent bots crawling the domain/subdomain resources.
Clicking on a table row the related robots.txt file will be selected and its content shown in the viewer.
The icon column gives an indication on the state of the robots.txt file.
Used icons are:
The robots.txt file was successfully downloaded.
Crawlers are supposed to obey to its directives when accessing the site resources.
The robots.txt file is not present for the current domain/subdomain ("Authority").
A missing robots.txt file is a valid state, it means there are no access restrictions set for spiders.
- A HTTP status code not expected in the robots.txt protocol occurred when attempting to download the file.
According to the protocol, this should prevent crawlers to access the domain/subdomain ("Authority") resources.
The root domain or the subdomain where the robots.txt file was found.
Bot access to each authority is regulated by the robots.txt file it contains.
The selected root domain or subdomain, where the robots.txt file was found.
Bot access to the authority resources is regulated by the robots.txt file it contains.
The date when the robots.txt file was downloaded.
The textual description of the HTTP response code received from the web server upon requesting the robots.txt file.
Displays the text extracted from selected robots.txt file.