The Robots view in Site Explorer will give you information about your pages' robots.txt, meta robots and x-robots data.
Date points available in Robots view:
- URL / Title - This column shows the crawled URL and its page title
- Status - The HTTP status code of requested URL
- Keywords - Mapped keywords associated with this URL
- User Agents Blocked by Robots.txt - Shows all the user agents blocked by robots.txt for this URL
- Meta Robots - If Meta Robots tag exists, its value
- Meta Robots NoFollow - Whether the URL contain NoFollow meta robots tag
- All User Agents Blocked by Robots.txt - If all user agents are being blocked by robots.txt, the URL will be flagged here
- X-Robots Tag - If X-Robots tag exists for this URL, its value
- All User Agents Block by X-Robots tag - If all user agents are being blocked by x-robots, the URL will be flagged here
Did this answer your question?