There are three primary ways to restrict Dragonbot from crawling parts of your site.

  • Crawl in "List mode": You can set the crawler to only include a list of URLs you submit. To make sure Dragonbot doesn't crawl anything you don't want it to, simply leave off any of these URLs when uploading your list. learn more about crawling in list mode
  • Add URL Exclusions: You can choose to crawl in "Spider mode", where Dragonbot will crawl similar to a search engine, but add rules that prevent certain areas of the site from being crawled. Learn more about URL exclusions

Did this answer your question?