What is crawl scope?
The scope of a crawl determines which URLs are considered part of the site. For example, the scope can be set to
- Root domain (example.com)
- Subdomain (www.example.com)
- Subfolder (example.com/en/uk/)
- Subdomain and Subfolder (www.example.com/en/uk)
- URL (https://www.example.com/products/green-widgets)
The scope is determined by the site field in General Campaign Settings the first time a campaign is created. By default, rank tracking, site audits, and backlink reports all share this same scope.
However, there may be times when you may want to use a different scope for site audits than for rank tracking or backlink reports. For example, you may want to include URLs from all subdomains in rank tracking (root domain scope) but only crawl the www subdomain (subdomain scope) for site audits.
To do this, you can set a separate scope for crawls.
How to update crawl scope
Navigate to Crawler Settings under Campaign Settings in the left navigation
By default, you'll see the crawl scope is set to Same as main site, and the value will match what is set under General Campaign Settings.
To change the scope without affecting rank tracking or backlink reports, change Scope to Use other scope
This will unlock the text field. Enter your new desired scope into the box.
As a reminder, you may exclude the scheme when entering as site, as both http and https will be included.
- Track all URLs on a root domain: Include only the root domain (example.com)
- Track all URLs on a single subdomain: Include the subdomain + root domain (www.example.com)
- Track all URLs in a subfolder on any subdomain: Include the root domain + path (example.com/en/uk/)
- Track only URLs in a single subdomain on a single subfolder: Include the subdomain + root domain + path (www.example.com/products)
- Track a single URL: Include only the URL you want to track (www.example.com/products/19312981.html)
When you're finished, save your changes.
The changes will take effect next crawl, but you can always initiate a manual crawl immediately.