The Robots view in Site Explorer will give you information about your pages' robots.txt, meta robots and x-robots data.

Date points available in Robots view:

  • URL / Title - This column shows the crawled URL and its page title
  • Status - The HTTP status code of requested URL
  • Keywords - Mapped keywords associated with this URL
  • User Agents Blocked by Robots.txt - Shows all the user agents blocked by robots.txt for this URL
  • Meta Robots - If Meta Robots tag exists, its value
  • Meta Robots NoFollow - Whether the URL contain NoFollow meta robots tag
  • All User Agents Blocked by Robots.txt - If all user agents are being blocked by robots.txt, the URL will be flagged here
  • X-Robots Tag - If X-Robots tag exists for this URL, its value
  • All User Agents Block by X-Robots tag - If all user agents are being blocked by x-robots, the URL will be flagged here
Did this answer your question?