Dragonbot is the name of the crawler Dragon Metrics uses to collect onsite data to be used in features such as Site Explorer, Site Audit, and URL X-Ray.

The basics

How to Identify Dragonbot

Dragonbot identifies itself as "dragonbot" in the user agent field of the HTTP request header when it crawls a site. You can identify it as such in your server's logs.

Here is one example of a user agent string Dragonbot uses:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36;Dragonbot; http://www.dragonmetrics.com"

Please note that several variations of this user agent string are in use, and may change from time to time. To identify the user agent as Dragonbot, please do not match the entire string. Instead, use the final part of the user agent string to identify it, which will remain constant:

Dragonbot; http://www.dragonmetrics.com

Politeness

Dragonbot respects webmasters' and server resources by adhering to crawler politeness rules, including but not limited to:

  • Respecting and following the robots.txt directive

  • Limiting the interval and number of server requests per minute

  • Self-identifying itself in the user agent string of the HTTP request header

Crawl Frequency

If crawling is enabled for a campaign, Dragonbot will begin the initial crawl immediately after the campaign is created. After the initial crawl, Dragonbot will return to your site approximately 14 days later. As soon as the crawl has completed, the data will immediately be made available in the platform.

Learn more about Dragonbot

Did this answer your question?