What is the Meta Robots Tag?
The meta robots tag is a line of code found in the head element of the page that specifies how search engines and other crawlers should treat the page content.
A typical meta robots tag:
<meta name="robots" content="noindex"/>
There are several values that could be included in the content attribute, but the most common are index, noindex, follow, and nofollow. Multiple values can be used, comma-separated.
- index: Instructs search engines to index the page content. This is the default value, so it is not actually needed to be written explicitly.
- noindex: Instructs search engines not to include the page in the search engine's index.
- follow: Instructs search engines to follow all links on the page as normal. Again, this is the default behavior, so is not really needed.
- nofollow: Instructs search engines not to follow the links on the page.
What Does "Meta Robots NoFollow" Mean?
If a URL has the value of "nofollow" in the meta robots tag, search engines will not crawl any of the links on the page. This means that if these URLs are not linked to from other places, they will not be crawled/indexed by search engines, and will not be accessible to users via organic searches.
What Should I Do With This Data?
Look through the URLs to see if there are any pages that whose links you want to be followed. If you find any pages that are unintentionally in this list, you will want to remove this tag (or the "nofollow" value) from this URL.
You may also want to look through the list to see if there are any URLs that you expected would be marked as nofollow that you don't see in this list. It may be possible that the tag could be missing or is improperly formatted. (However, it's possible that Dragon Metrics simply did not include this page in the crawl.)