There are three primary ways to restrict Dragonbot from crawling parts of your site.
- Crawl in "List mode": You can set the crawler to only include a list of URLs you submit. To make sure Dragonbot doesn't crawl anything you don't want it to, simply leave off any of these URLs when uploading your list. learn more about crawling in list mode
- Add URL Exclusions: You can choose to crawl in "Spider mode", where Dragonbot will crawl similar to a search engine, but add rules that prevent certain areas of the site from being crawled. Learn more about URL exclusions
- Block Dragonbot with the robots.txt file: Dragonbot respects the robots.txt directive. You can prevent Dragonbot from crawling specific areas of your site by writing directives for the user agent "dragonbot". Learn more about how to block Dragonbot with the robots.txt