What is the Meta Robots Tag?
The meta robots tag is a line of code found in the head element of the page that specifies how search engines and other crawlers should treat the page content.
A typical meta robots tag:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
There are several values that could be included in the content attribute, but the most common are index, noindex, follow, and nofollow. Multiple values can be used, comma-separated.
- index: Instructs search engines to index the page content. This is the default value, so it is not actually needed to be written explicitly.
- noindex: Instructs search engines not to include the page in the search engine's index.
- follow: Instructs search engines to follow all links on the page as normal. Again, this is the default behavior, so is not really needed.
- nofollow: Instructs search engines not to follow the links on the page.
What does it mean if a URL is blocked by meta robots?
Any URL that has the value of noindex in the meta robots tag will not be indexed by search engines. This means that any URL listed in the table below will not be accessible to users via organic searches.
What Should I Do With This Data?
Look through the URLs to see if there are any pages that you want accessible via organic search. If you find any pages that are unintentionally in this list, you will want to remove this tag (or the noindex value) from this URL.
You may also want to look through the list to see if there are any URLs that you expected would be blocked via the meta robots tag that you don't see in this list. It may be possible that the tag could be missing or is improperly formatted. (However, it's possible that Dragon Metrics simply did not include this page in the crawl.)