What is a web crawler?

A web crawler, sometimes called a spider or bot, is software that automatically visits URLs, downloads each page, extracts the links it finds, and queues those links to visit next. The process repeats until the crawler exhausts the link graph or hits a limit on depth, page count, or time.

Search engines run crawlers continuously to keep their indexes fresh, and AI companies run their own: Googlebot and Bingbot feed search indexes, while GPTBot, ClaudeBot, and PerplexityBot collect pages for model training and retrieval. Outside search, crawlers power price monitoring, SEO audits, archival, brand monitoring, and the data pipelines that feed retrieval-augmented LLMs.

A production-grade crawler has to handle a lot more than the loop above. It needs polite request scheduling, robots.txt compliance, sitemap parsing, JavaScript rendering for SPAs, retries with backoff, deduplication of canonicalized URLs, and a strategy for when the target site starts returning 429s or blocking the IP. Most teams underestimate how much of the engineering goes into the second list, not the first.