A web crawler (also known as spider, or robot) is an automated computer program that browses the Internet, typically for the purpose of indexing the content.

How do crawlers work

Crawlers work by starting with a URL or group of URLs which they will download and view the content of, which they may or may not decide to add to their index. They will then visit each URL that is linked to on these pages and do the same with them. They will continue recursively visiting all of the URLs linked to on these pages as well, exponentially increasing the number of pages they need to visit.

The word crawl describes the metaphor of spiders (computer programs) crawling (browsing) the World Wide Web.

Types of crawlers

Search engines are one of the most common types of crawlers. Search engines crawl your site to index the content so it can be returned as query results on a SERP.

Other types of spiders may crawl your site for other reasons, such as validating site code or site structure, scraping content (downloading data or content on your site to analyze, mine, or steal and re-purpose on another's site), or even for hacking or other nefarious purposes.

Dragon Metrics uses a web crawler (named Dragonbot) to gather data about sites for the Site Audit, Site Explorer and URL X-ray features.

Did this answer your question?