The x-robots tag is a field in the HTTP response header that allows sites to tell search engines and other crawlers whether or not they are not allowed to access the content found on the URL. In this way, it is very similar to the meta robots tag or the robots.txt file. The difference is that this information is found in the HTTP response header instead of the page source or the robots.txt file on the root of the domain.
A typical x-robots tag:
Similar to the meta robots tag, the x-robots tag can contain multiple values, comma-separated.
Below are the most important values:
- all: The default value. No restrictions for robots.
- noindex: Exclude this URL from being indexed
- nofollow: Do not follow links on this page
- none: Do nothing with this page (same as "noindex, nofollow")