By default, our crawler does not execute any Javascript, so any content rendered with Javascript will be not be included in site crawls.

If your site renders content using Javascript, you will need to enable Javascript crawling in Dragon Metrics to ensure all content on your site is found. Otherwise, you may see incorrect crawl data in Dragon Metrics onsite tools such as Site Auditor and Site Explorer.

If your site does not use Javascript to render content on your site, we recommend leaving this option disabled, which will speed up crawls.

How Javascript crawling works

If Javascript crawling has been enabled, our crawler Dragonbot will use a browser to crawl your site and will render Javascript and CSS just like a normal visitor would. The Javascript crawler uses Chromium (the open source version of Google Chrome), which is the same as Googlebot uses when it crawls your site. This crawler tries to emulate how Google crawls your site as closely as possible.

Please note that each URL crawled with JavaScript will use 5 crawl credits.

Dragon Metrics Javascript crawler tries to follow Google's Web Rendering Service as closely as possible. This means our crawler may still have a slightly different experience than a typical user.

Here are some details on how our Javascript crawler works:

  • Javascript redirects are followed. We label this with a custom HTTP code 399 in Dragon Metrics reports such as Site Auditor and Site Explorer so they can be identified.

  • Common analytics, advertisement, and other tracking scripts are blocked from running to ensure Dragonbot doesn't inflate analytics data.

  • Coookies are not accepted

  • Local storage is cleared on each page

  • User permission requests (such as camera or notifications) are declined

In addition, our crawler sets a maximum time-out for server response times, Javascript execution, and total time. Servers with a slow response time or heavy scripts may result in crawl errors or incomplete data.

  • From the time our crawler makes the first GET request for this page until the domcontentloaded event is fired, we will wait a maximum of 15 seconds.

  • From the domcontentloaded event to when we consider the HTML fully rendered, we will wait a maximum of 10 seconds. If Javascript has not fully rendered within this time, our crawler will download the HTML as-is, assuming it is fully rendered.

  • A maximum time-out of 20 seconds for the entire process is in place.

(These maximum wait times are in line with SEO best practices. If content is not being loaded within these times, it's likely Google may have issues rendering and indexing your content as well.)

How to enable Javascript crawling in Dragon Metrics

To enable Javascript crawling, navigate to Crawl Settings, which can be found under Campaign Settings > Crawler in the bottom left navigation.

On this page, set Render Javascript to Yes

Save changes

Javascript rendering has now been enabled, but existing data isn't updated immediately. The changes will not take effect until the next scheduled crawl. Click Recrawl site now at the top of the page to start a new crawl with the new settings.

Did this answer your question?