Dragonbot is the name of the crawler Dragon Metrics uses to collect onsite data to be used in features such as Site Explorer, Site Audit, and URL X-Ray.

The basics

How to Identify Dragonbot

Dragonbot identifies itself as "dragonbot" in the user agent field of the HTTP request header when it crawls a site. You can identify it as such in your server's logs.

Politeness

Dragonbot respects webmasters' and server resources by adhering to crawler politeness rules, including but not limited to:

  • Respecting and following the robots.txt directive
  • Limiting the interval and number of server requests per minute
  • Self-identifying itself in the user agent string of the HTTP request header

Crawl Frequency

If crawling is enabled for a campaign, Dragonbot will begin the initial crawl immediately after the campaign is created. After the initial crawl, Dragonbot will return to your site approximately 14 days later. As soon as the crawl has completed, the data will immediately be made available in the platform.

Learn more about Dragonbot

Did this answer your question?