What is the X-Robots Tag?

The x-robots tag is an area in the HTTP response header that allows sites to tell search engines and other crawlers whether or not they are not allowed to access the content found on the URL. In this way, it is very similar to the meta robots tag or the robots.txt file. The difference is that this information is found in the HTTP response header instead of the page source or the robots.txt file on the root of the domain.

A typical x-robots tag:

X-Robots-Tag: noindex

Similar to the meta robots tag, the x-robots tag can contain multiple values, comma-separated.

Below are the most important values:

  • all: The default value. No restrictions for robots.

  • noindex: Exclude this URL from being indexed

  • nofollow: Do not follow links on this page

  • none: Do nothing with this page (same as "noindex, nofollow")

The x-robots tag is relatively new and not as widely used or accepted as robots.txt or the meta robots tag. Google, Bing, and Yahoo support it, but other search engines may not.

What Does "Blocked by X-Robots Tag" Mean?

Any URL that has the value of "noindex" in the x-robots tag will not be indexed by search engines. These URLs will not be accessible to users via organic searches.

What Should I Do With This Data?

Look through the URLs to see if there are any pages that you want accessible via organic search. If you find any pages that are unintentionally in this list, you will want to remove this tag (or the "noindex" value) from this URL.

You may also want to look through the list to see if there are any URLs that you expected would be blocked via the x-robots tag that you don't see in this list. It may be possible that the tag could be missing or is improperly formatted. (However, it's possible that Dragon Metrics simply did not include this page in the crawl.)

Did this answer your question?