Some advanced crawler settings require verifying that you own or manage the site you'd like to crawl.
These advanced crawler settings include:
How to verify your site
To verify your site, navigate to Campaign Settings > Crawler in the bottom left navigation.
Scroll down the page to Ownership Status and click the Verify ownership link
The Verify Site Ownership modal will be shown
There are 4 methods to verify your site:
Google Search Console (recommended)
Google Search Console (recommended)
Verifying with Google Search Console is the easiest and recommended method. If you've already connected a Google account that has access to a Google Search Console site that matches this campaign, verification will be completed automatically.
If you haven't done so already, click Add a new Google Account and sign into a Google account that access to a Google Search Console site that matches this campaign.
Please note that the site in Google Search Console must be exactly the same as it's entered for this campaign, or have a larger scope.
For example, if the site is entered as https://example.com/products in Google Search Console, but example.com in Dragon Metrics, the site will not be verified since the scope of the Google Search Console site (the /products subfolder) is smaller than the scope of the Dragon Metrics campaign (the entire example.com root domain).
On the other hand, if the site is entered as sc-domain:example.com in Google Search Console, but https://example.com/products in the Dragon Metrics campaign, the site will be verified since the scope of the Google Search Console site is larger than the Dragon Metrics campaign.
To verify using a DNS record, you'll need to complete the following steps:
Sign in to your domain name provider
A string of text will be provided. Copy this text and create a new TXT record with this string into the DNS configuration for this site.
After this has been completed, click re-check status. Please note that it may take 24 hours for DNS records to propagate, so if it doesn't work immediately, try again tomorrow.
Do not delete this DNS record after initial verification. Every time the site is crawled, we will check for verification.
To verify with an HTML tag, copy and paste the text provided and place in the home page of your site, then click re-check status. If we find this tag in the home page, the site will be verified.
Do not delete this HTML tag after initial verification. Every time the site is crawled, we will check for verification.
To verify using an HTML file, simply download the file provided, then upload it to the root for the domain or subdomain.
You can click the link provided to check whether the file is in the right place. When it's ready, re-check the status to complete verification.
Do not delete this HTML file after initial verification. Every time the site is crawled, we will check for verification.
Failed verification at crawl time
Every time a site is crawled, we will re-check verification status. Therefore it is important you do not remove the method that you used to verify your site. Don't remove the Google Search Console site, DNS record, HTML tag, or HTML file — leave them in place even after initial verification.
If the verification fails at the time a scheduled crawl takes place, the crawl will fail and no data will be collected. You will receive an email notification that this has happened so you can help take action. You'll also be provided with an additional recrawl credit, so that you can start a recrawl after fixing the issue without using additional credits.
Before crawls can resume, you must either re-verify your site or disable advanced crawl options.
Re-verify your site
To re-verify your site, simply follow the instructions on this page above just as you did before.
Once the site is verified, you can manually start a new crawl of the site immediately so you don't have to wait until the next scheduled crawl date.
Disable advanced options
If you are unable to re-verify your site again, but wish to resume crawling, you can disable any selected advanced crawl options:
Once these settings have been disabled, you can manually start a new crawl of the site immediately so you don't have to wait until the next scheduled crawl date.