What is the Meta Robots Tag?

The meta robots tag is a line of code found in the head element of the page that specifies how search engines and other crawlers should treat the page content.

A typical meta robots tag looks like this:

<meta name="robots" content="noindex"/>

There are a number of values that could be included in the content attribute, but the most common include:

  • all: No restrictions for robots. This is the default value, so it is not actually needed to be written explicitly.

  • index: Instructs search engines to index the page content. This is the default behavior, so is not really needed.

  • noindex: Instructs search engines not to include the page in the search engine's index.

  • follow: Instructs search engines to follow all links on the page as normal. Again, this is the default behavior, so is not really needed.

  • nofollow: Instructs search engines not to follow the links on the page.

  • none: Equivalent to “noindex, nofollow”.

Multiple values can be used, comma-separated.

What does it mean if a URL is blocked by meta robots?

Any URL that has the value of noindex in the meta robots tag will not be indexed by search engines. These URLs will not be accessible to users via organic searches.

What Should I Do With This Data?

If the URLs should be excluded from search engines' index, nothing needs to be done. However, if this page should be included in search engines' index, remove the meta robots tag or remove the noindex or none directive.

You may also want to look through the list to see if there are any URLs that you expected would be blocked via the meta robots tag that you don't see in this list. It may be possible that the tag could be missing or is improperly formatted. (However, it's possible that Dragon Metrics simply did not include this page in the crawl.)

Did this answer your question?