What is Scrapy?
A Python framework for building large-scale web crawlers, with batteries included for scheduling, retries, deduplication, and data pipelines.
Scrapy is what you reach for when "fetch a few pages with requests" stops scaling. You define a Spider class with start URLs and a parse method, yield more requests or extracted items from each response, and Scrapy handles the queue, concurrency, throttling, retries, redirects, cookies, and output serialization. A single Spider can crawl millions of URLs from a laptop.
The architecture is async-first (built on Twisted historically, with asyncio integration in recent versions) so the framework keeps thousands of requests in flight without blocking. Middlewares plug into request and response paths for things like proxy rotation, captcha handling, or custom retry logic; pipelines transform extracted items before they hit the database, JSON Lines file, or whatever sink you configure.
Scrapy is not a fit for everything. If you only need a few pages, a script with httpx and selectolax is simpler. If the target requires JavaScript rendering, you bolt on scrapy-playwright or pre-render upstream. But for breadth-first crawls of static or server-rendered sites, Scrapy is still the most productive option in Python.
In the wild
- Running a nightly crawl of 50,000 supplier websites to refresh a product catalog
- Building a vertical search index over a list of news domains
- Scraping every page of a competitor's docs to feed an LLM's retrieval index