FAQs: Sitemap Issues

This usually happen when dealing with a pre-production website where the crawled URL is canonicalized with the URL of the production website.
The visual sitemap generator by default only exports pages fit for a XML Sitemap, and non-canonical pages are normally skipped.
You can force the generator to include non-canonical URLs by unchecking the two options "Skip non-canonical URLs" and "Use canonical URL if exists".

Canonical URL options for XML Sitemap Generator

Albeit this method lets you can generate a correct sitemap for the case described, it cannot distinguish URLs which are meant to be canonicalized also on the production website.
We recommend thus to generate the XML Sitemap directly on crawl sessions from a production website.

This as well could happen when dealing with a pre-production website where site admin to prevent accidental indexing has forced all pages to be "noindex".
The visual sitemap generator by default only exports pages fit for a XML Sitemap, and "noindex" pages are normally skipped.

You can force the generator to include noindex pages, by unchecking option "Skip 'noindex' pages".

"noindex" pages option for XML Sitemap Generator

The options is only visible when clicking the "Export to XML Sitemap" button from the Tabular View. We decided not to show it in the visual generator in order to keep it simpler for the most common cases.
Here a short video showing you how to invoke the options dialog:

Video: how to Export to XML Sitemap from Tabular View

Albeit this method lets you can generate a correct sitemap for the case described, it cannot distinguish pages which are meant to be noindex also on the production website.
We recommend thus to generate the XML Sitemap directly on crawl sessions from a production website.

Note: if your pre-production website is exposed to the public, using noindex is not the recommended way to prevent its indexing by a search engine.
We recommend using HTTP Authentication (and configure Visual SEO Studio to be able to crawl it), or if you are really lazy and don't care to protect it to disable its indexing via robots.txt (in this case to crawl it with Visual SEO Studio you should leave an Allow directive for it, or add the site to the Verified Sites list and crawl it instructing the spider to ignore the robots.txt file).

In case of site migration where all of part of the URLs change and are redirected, it is good practice to create a temporary XML Sitemap listing all old redirected URLs to submit to the search engine. This way the search engine will understand faster that the new URLs are not duplicates and transfer them the old URLs PageRank.

The first step is filtering the site pages list to only show in Tabular View the redirected URLs. You can achieve this by using the Custom Filters feature.

Then similarly to the previous point, you can force the generator to include non-"200 OK" URLs, by unchecking option "Skip non-'200 OK' URLs".

non-"200 OK" URL option for XML Sitemap Generator

The options is only visible when clicking the "Export to XML Sitemap" button from the Tabular View. We decided not to show it in the visual generator in order to keep it simpler for the most common cases.
Here a short video showing you how to invoke the options dialog:

Video: how to Export to XML Sitemap from Tabular View