What are crawl limits?
Crawl limits determine the maximum number of URLs crawled for a campaign.
Every campaign that has crawling enabled must set a crawl limit greater than 0. The maximum value allowed for crawl limits is defined by the lower of either:
- The number of remaining crawl credits in your account
- The maximum URLs crawled per campaign.
Both of these numbers are determined by your subscription plan.
Setting a crawl limit for a campaign reserves these crawl credits for this campaign. The crawl credits consumed by a campaign are determined by the crawl limits allocated to the campaign, not the number of URLs actually crawled. Therefore, if the number of URLs crawled are less than the URLs actually crawled, you can free up crawl credits by lowering the crawl limits for this campaign.
For example, if the crawl limits of Campaign A is set to 5,000, it will consume 5,000 crawl credits. If only 100 URLs are crawled, there are 4,900 crawl credits that could be freed up. If you expect future crawls to not exceed 100 URLs, it may be wise to set the crawl limits to 100 to free up these credits.
How to update crawl limits
There are two places where Crawler Settings may be modified:
- Crawler Settings: Best for changing 1 campaign at a time
- All Campaigns Settings: Best for updating multiple campaigns at once
Navigate to Crawler Settings under Campaign Settings in the left navigation
Update the number under Limits
Save your changes
The changes will take effect next crawl, but you can always initiate a manual crawl immediately.
All Campaigns Settings
Navigate to All Campaigns Settings under Account Settings in the left navigation.
Click on the crawl limits column to reveal a popover. Enter the new limits and click the check button to save.