There are two options for how Dragon Metrics will crawl your site: spider and list.

Spider Mode

Spider mode is the default method for crawling your site. When selected, our crawler will begin from the website URL for this campaign and any alternate seed URLs (if entered) and recursively crawl all links found on this and all other pages included in the crawl scope. This is similar to how search engines typically crawl your site.

Crawls in spider mode will stop when there are no remaining URLs left to crawl or the crawl limit has been reached.

By default, all URLs within in the site scope will be included in the crawl. To limit which areas of the site should not be included in the crawl (such as a subdomain, subfolder, or parameter), exclusions can be used.

Learn more about spider mode

List Mode

List mode allows you upload a list of URLs to be crawled. Only pages included in this list will be crawled.

This is useful if you already have a specific list of URLs that you know you want crawled, especially if you have a very large site where a Spider crawl may include a large number of less-useful pages. You can even extract URLs from an existing XML Sitemap to crawl only these URLs.

You'll also have the option of following redirects if any of the URLs included in the list return a 3xx HTTP status.

Learn more about List crawls

How to change crawl mode

To change the crawl mode, navigate to Crawler Settings under Campaign Settings in the left navigation.

Choose your desired crawl mode

Update any other relevant crawl options

Click Save changes when you are finished.

By default, the new settings will take effect with the next scheduled crawl. If you wish to crawl immediately, you can initiate a manual crawl.

Did this answer your question?