Exclude URLs is a crawl option in Dragon Metrics created to give user ability to instruct our crawlers to not crawl certain URLs on their sites.
This feature can be useful to user for 2 main reasons:
- Faster data retrieval - Because our crawler don't need to crawl as much URLs, with URL exclusions you will be able to get crawl data faster every time we crawl your sites
- More efficient limit usage - Every account in Dragon Metrics has a limit in how many URLs they can crawl, by excluding URLs you can save up crawl limits for future usage
URL exclusions is set at campaign level and it can be found in the following pages:
In the URL Exclusions menu will be a table showing you all the currently excluded URLs, to add excluded URLs, click on the + icon button:
After clicking the + button, there will be another popup window allowing you to exclude URLs by subdirectory, subdomain or individual URLs, click on Show Advanced Options you can exclude by URL parameters or Regular Expression: