Dragonbot is the name of the crawler Dragon Metrics uses to collect onsite data to be used in features such as Site Explorer, Site Audit, and URL X-Ray.
How to Identify Dragonbot
Dragonbot identifies itself as "dragonbot" in the user agent field of the HTTP request header when it crawls a site. You can identify it as such in your server's logs.
Dragonbot respects webmasters' and server resources by adhering to crawler politeness rules, including but not limited to:
- Respecting and following the robots.txt directive
- Limiting the interval and number of server requests per minute
- Self-identifying itself in the user agent string of the HTTP request header
If crawling is enabled for a campaign, Dragonbot will begin the initial crawl immediately after the campaign is created. After the initial crawl, Dragonbot will return to your site approximately 14 days later. As soon as the crawl has completed, the data will immediately be made available in the platform.