The meta robots tag is a line of code found in the head element of the page that specifies how search engines and other crawlers should treat the page content.
A typical meta robots tag:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">There are several values that could be included in the content attribute, but the most common are index, noindex, follow, and nofollow. Multiple values can be used, comma-separated.
- index: Instructs search engines to index the page content. This is the default value, so it is not actually needed to be written explicitly.
- noindex: Instructs search engines not to include the page in the search engine's index.
- follow: Instructs search engines to follow all links on the page as normal. Again, this is the default behavior, so is not really needed.
- nofollow: Instructs search engines not to follow the links on the page.