You can upload a list of URLs to be crawled instead of the default spider crawl. Simply choose List mode in Crawler Settings. Read on for details.

There are two modes available in Crawler Settings: Spider and List.

Spider mode is the default method, which will start crawling the home page of the site and recursively crawl all links found on this each page. (learn more about spider mode)

List mode allows you upload a list of URLs to be crawled. Only pages included in this list will be crawled.

Changing to List mode

To change the crawling mode to List, navigate to Crawler Settings, located in the bottom of the navigation menu on the left side of every page.

On the Crawler Settings page, set the Mode option to List

From here, you will need to provide the list of URLs you'd like crawled.

If crawling 3,000 URLs or less, you may paste them into the text box:

If you need to crawl more than 3,000 URLs or simply prefer to use a file, select Upload from File, download the template, enter your URLs and then click Upload File.

Please note that the required format is a list of URLs (one-per-line), so an unmodified XML sitemap will not work. We recommend extracting the URLs from the sitemap first before submitting.

Finally, select whether or not you'd like to crawl redirects. When set to "No", only the URLs in the list will be crawled. If set to "Yes", any URLs in the list that redirect to another URL will have the redirect target URL included in the crawl.

Finally, save the changes.

By default, the new settings will take effect with the next scheduled crawl. If you wish to crawl immediately, you can initiate a manual crawl.

Did this answer your question?