There may be a number of reasons why Dragonbot does not find all of the pages in your site:

Page rendered in JavaScript

By default, Dragonbot does not execute Javascript, so if the site relies on JavaScript for rendering content or links, these will not be found or included by the crawler.

If this is the case for your site, we recommend enabling Javascript crawling in Dragon Metrics. After enabling Javascript crawling, these links will be found in the next crawl.

Dragonbot works the same way other web crawlers do - Dragonbot crawls all of the links on your home page and visits them. Then it crawls all the links on this page, and so on. If there are no links to the pages you want crawled, Dragonbot will not be able to find them.

Page Size Too Large

Dragonbot only downloads the first 500KB of each page. If you have a very large page with links at that the bottom, these links are likely to not be crawled or discovered by Dragonbot.

Dragonbot follows a maximum of 500 links per page. Links above this limit will not be followed or crawled.

Outside of Website Scope

The website address you add when creating the campaign determines the scope of the crawl. Dragonbot will not crawl outside this scope. The table below shows some examples:

Website: or
Scope: All subdomains and all subdirectories

Website: or
Scope: All subdirectories on the www subdomain and its subdomains (e.g.

Website: or
Scope: The products subdirectory on the www subdomain and its subdomains

Scope: All subdomains and all subdirectories under the domain ( will not be crawled)

Redirecting Based on IP Address

Some multinational websites will sniff the users' IP address, and redirect them to a different domain based on their location. (For example, users from the UK may be redirected to when they try to visit, while users from China may be redirected to when they visit

When this happens, in many cases Dragonbot will not be able to crawl the site, depending on how the redirects are done.

Outside Crawl Limits

Dragonbot is limited by the number of crawl credits assigned to each campaign. Therefore, if the crawl limits are set to 10,000, Dragonbot will only crawl 10,000 URLs on your site. URLs blocked by the robots.txt file do not count towards this limit.

Some sites may be larger than the crawl limits, so for these sites, Dragonbot will crawl URLs prioritized by depth (link distance from the home page). Page that may be several links away from the home page may not be crawled.

Disallowed by Robots.txt

Dragonbot follows the robots.txt directive. If Dragonbot is blocked in this file, it will not crawl the site or parts of the site that is blocked.

Important note: If you verify ownership of your site, it is possible override the default behavior of Dragonbot to ignore the robots.txt or use a custom robots.txt for your site. If it doesn't look like Dragonbot should be blocked by the robots.txt file on your site, please ensure this advanced setting has not been enabled.


URLs may have been excluded from the crawl by selecting Manage exclusions in Crawler Settings.

Content Behind Login

Any content that users must log in for will not be accessible to Dragonbot.


Some sites that use cookies to redirect users may confuse Dragonbot, and it will not be able to crawl the site effectively.

Blocked by server

It may be possible your website server is blocking Dragonbot by user agent, IP address, throttling requests, or other methods.

Server errors

The website server hosting your site may have a configuration error, could be down at the time of the crawl, or another issue preventing Dragonbot from requesting data from the site.

Links in iframes, Java, or Flash

Dragonbot only crawls content and links in plain text. Therefore, if there are links in iframes, Java, Flash, or some other format other than plain text, Dragonbot will not be able to discover or follow these links.

Other issues

There could be other issues preventing the crawl from completing. If you are unable to find your issue from this list, please contact Dragon Metrics support for help in troubleshooting.

Did this answer your question?