What is crawl scope?

The scope of a crawl determines which URLs are considered part of the site. For example, the scope can be set to

The scope is determined by the site field in General Campaign Settings the first time a campaign is created. By default, rank tracking, site audits, and backlink reports all share this same scope.

However, there may be times when you may want to use a different scope for site audits than for rank tracking or backlink reports. For example, you may want to include URLs from all subdomains in rank tracking (root domain scope) but only crawl the www subdomain (subdomain scope) for site audits.

To do this, you can set a separate scope for crawls.

How to update crawl scope

Navigate to Crawler Settings under Campaign Settings in the left navigation

By default, you'll see the crawl scope is set to Same as main site, and the value will match what is set under General Campaign Settings.

To change the scope without affecting rank tracking or backlink reports, change Scope to Use other scope

This will unlock the text and dropdown field. Now you can select the scope to instruct how our crawler should crawl your site.

There are 4 match types of Site Scope you can set up for a crawl:

Match Type

Description

Example

Root Domain

Includes all URLs on this root domain

*.example.com/*

Subdomain / subfolder

Only include URLs on the entered subdomain and/or subfolder

*.subdomain.example.com/subfolder/*

Exact URL

Only include the entered URL

example.com/products/blue-widgets

Root domain only

Only include URLs on this root domain that have no subdomain

example.com/*

As a reminder, you may exclude the scheme when entering as site, as both http and https will be included.

  • Track all URLs on a root domain: Include only the root domain (example.com)

  • Track all URLs on a single subdomain: Include the subdomain + root domain (www.example.com)

  • Track all URLs in a subfolder on any subdomain: Include the root domain + path (example.com/en/uk/)

  • Track only URLs in a single subdomain on a single subfolder: Include the subdomain + root domain + path (www.example.com/products)

  • Track a single URL: Include only the URL you want to track (www.example.com/products/19312981.html)

When you're finished, save your changes.

The changes will take effect next crawl, but you can always initiate a manual crawl immediately.

Did this answer your question?