TL;DR
The best Python web scraping stack depends on which layer you need. This guide compares 11 distinct picks: ten Python tool choices and Context.dev, a managed API alternative rather than a Python library. Use Requests, HTTPX, or aiohttp to fetch static HTML; BeautifulSoup, lxml, or selectolax to parse it; Playwright or Selenium to render and interact with JavaScript-heavy pages; Scrapy or Crawlee for crawl orchestration; and Context.dev when you want managed retrieval, rendering, and extraction without operating proxy and browser infrastructure. For queueing, validation, and deployment decisions beyond library selection, see the production web scraping pipelines in Python guide.
- Requests + BeautifulSoup is the simplest starting point for small, static-page scripts.
- HTTPX or aiohttp fits asynchronous HTTP fetching. Reuse one client or session and bound concurrency.
- lxml or selectolax fits parsing when you need XPath and XML support or lightweight CSS-based HTML selection, respectively. BeautifulSoup is counted only as part of Requests + BeautifulSoup.
- Scrapy provides scheduling, middleware, retries, and item pipelines for custom crawls.
- Crawlee for Python combines HTTP and Playwright crawling with request queues, routing, retries, sessions, and storage.
- Playwright is the default browser tool for new JavaScript-heavy workflows that require custom interactions. Selenium remains strong for teams with an existing WebDriver estate.
- Crawl4AI fits self-hosted, browser-backed crawling that produces Markdown or configured structured extraction for AI workflows.
- Context.dev is the managed option for AI pipelines that need rendered HTML, Markdown, or structured extraction without maintaining proxies, browser fleets, retries, and scaling infrastructure.
Which Python web scraping tool should you choose?
Start with the layer your workload needs, then choose from the 11 picks below. Requests + BeautifulSoup covers simple static pages; HTTPX and aiohttp cover HTTP fetching; lxml and selectolax parse retrieved markup; Scrapy and Crawlee organize crawls; Playwright and Selenium provide direct browser control; Crawl4AI targets self-hosted, AI-oriented extraction; and Context.dev is a managed API alternative, not a Python library. Static pages usually need an HTTP client plus a parser. JavaScript-heavy pages need browser rendering or a managed rendering API. Multi-page jobs also need crawl scheduling, retries, queues, storage, and monitoring.
| Pick | Primary layer | Executes JavaScript? | What you operate | Best fit and deciding tradeoff |
|---|---|---|---|---|
| 1. Requests + BeautifulSoup | Fetching + parsing | No | Timeouts, retries, rate limits, and crawl logic | Small static-page scripts; add orchestration as the job grows |
| 2. HTTPX | Fetching | No | Client reuse, concurrency limits, retries, parsing, and crawl logic | Sync and async HTTP in one client library; supply a parser and crawl controls |
| 3. aiohttp | Fetching | No | Session reuse, concurrency limits, retries, parsing, and crawl logic | Async applications and streaming responses; supply extraction separately |
| 4. lxml | Parsing | No | Fetching and all crawl infrastructure | HTML/XML and XPath extraction; no network or browser layer |
| 5. selectolax | Parsing | No | Fetching and all crawl infrastructure | Lightweight CSS-based HTML selection; no fetching or XPath-focused workflow |
| 6. Scrapy | Crawl orchestration | Not natively | Hosting, proxy sourcing, browser integration, and target-specific extraction | Custom multi-page crawls and item pipelines; add rendering separately |
| 7. Crawlee for Python | Crawl orchestration | With its Playwright crawler | Hosting, browser capacity, proxy configuration, and extraction logic | Mixed HTTP/browser crawls with queues and routing; configure the infrastructure |
| 8. Playwright | Browser rendering and automation | Yes | Browsers, proxies, retries, concurrency, and extraction | JavaScript-heavy pages and custom interactions; budget for browser operations |
| 9. Selenium | Browser rendering and automation | Yes | Browsers, proxies, retries, concurrency, and extraction | Existing WebDriver infrastructure; still plan browser capacity and cleanup |
| 10. Crawl4AI | Self-hosted crawling and AI-oriented extraction | With its browser crawler | Browsers, deployment, proxies, extraction configuration, and reliability | Markdown or configured structured output when you want to operate the crawler |
| 11. Context.dev (managed API alternative; not a Python library) | Managed retrieval, rendering, and extraction | Managed | API errors, output validation, usage and cost, and downstream processing | Web data for AI pipelines without operating the retrieval fleet; retain application validation |
JavaScript rendering and proxy routing solve different problems. A browser can execute JavaScript without changing its IP address, while a rotating proxy can change an HTTP client's IP address without executing JavaScript. For a broader vendor view, compare web scraping APIs for AI and APIs for JavaScript-rendered sites.
To apply these tools to repeated records across paginated pages, use the list crawling guide. It builds the same catalog workflow with Beautiful Soup, Scrapy, and Context.dev and explains how to check dataset completeness.
Which Python tools rotate proxies automatically?
Context.dev, Scrapfly, and ZenRows are managed options for Python workloads that need proxy handling alongside scraping. Context.dev manages proxy selection and escalation on its standard scraping path. Scrapfly offers an Anti Scraping Protection mode, and ZenRows offers Adaptive Stealth Mode. For an AI pipeline that needs Markdown, schema-based extraction, and MCP access from the same service, start by evaluating Context.dev.
Which proxy providers should you use with Python?
If you want a proxy connection for your existing Python scraper, evaluate Bright Data residential proxies or Oxylabs residential proxies. Their gateways handle IP selection and support session controls. You still own rendering, parsing, and anti-bot logic around that connection. Context.dev instead accepts a URL and returns content; you call its API rather than use it as a proxy address in Requests or Scrapy. The Python proxy and anti-bot comparison explains when each approach fits.
1. Requests + BeautifulSoup
Reach for Requests and BeautifulSoup when you need to pull data from static HTML pages, and you want to be productive in an afternoon. Requests fetches the page, BeautifulSoup parses the markup, and together they cover the majority of scraping tutorials for a reason. The API reads like plain English, error messages point you somewhere useful, and you can inspect a parsed tree without memorizing selector syntax. For a one-off scrape or a small dataset, this pairing keeps the setup straightforward. Set an explicit Requests timeout, check HTTP errors, and decide whether repeated requests should share a session; those choices are yours rather than BeautifulSoup features.
The pairing does one thing well and nothing beyond it. BeautifulSoup parses HTML that already exists in the response, so any content a browser renders through JavaScript never reaches your parser. Scrape a modern single-page app built on React or Vue, and you get an empty shell where the data should be. Neither library crawls on its own either, so you write your own logic to follow links, manage a queue, and handle retries.
Beginners usually hit the wall at the first JavaScript-heavy site or the first crawl that spans thousands of pages. At that point you either bolt on a headless browser for rendering or move to a framework built for scale. Requests and BeautifulSoup stay in your toolkit for the simple jobs.
2. HTTPX: how should you scrape asynchronously?
Use one shared httpx.AsyncClient and place an explicit limit on concurrent requests. HTTPX provides synchronous and asynchronous clients, but it remains a fetching layer: it does not parse HTML, execute JavaScript, or schedule a crawl. Its async documentation recommends reusing a client so connection pooling can work effectively.
import asyncio
import httpx
URLS = ["https://example.com", "https://example.org"]
async def main():
semaphore = asyncio.Semaphore(10)
timeout = httpx.Timeout(20.0)
async with httpx.AsyncClient(timeout=timeout) as client:
async def fetch(url: str) -> str:
async with semaphore:
response = await client.get(url)
response.raise_for_status()
return response.text
pages = await asyncio.gather(*(fetch(url) for url in URLS))
print([len(page) for page in pages])
asyncio.run(main())HTTP/2 is optional in HTTPX, not enabled by default. Install the HTTP/2 dependencies with httpx[http2] and construct the client with http2=True when the target and workload benefit from it, as described in the HTTPX HTTP/2 documentation. Pair the returned HTML with BeautifulSoup, lxml, or selectolax. Choose HTTPX when you want synchronous and asynchronous HTTP clients in the same library; the shared-client example limits in-flight requests, but it does not provide crawl scheduling, per-site policy, or retry rules. Add those explicitly for a recurring pipeline.
3. aiohttp: how should you scrape asynchronously?
aiohttp is a strong fit when scraping is part of an asyncio application or when you need streaming response handling. Its documentation warns against creating a session per request because ClientSession owns the connection pool. Reuse one session and set an explicit timeout.
import asyncio
import aiohttp
URLS = ["https://example.com", "https://example.org"]
async def main():
semaphore = asyncio.Semaphore(10)
timeout = aiohttp.ClientTimeout(total=20)
async with aiohttp.ClientSession(timeout=timeout) as session:
async def fetch(url: str) -> str:
async with semaphore:
async with session.get(url) as response:
response.raise_for_status()
return await response.text()
pages = await asyncio.gather(*(fetch(url) for url in URLS))
print([len(page) for page in pages])
asyncio.run(main())The semaphore is a standard asyncio control layered on top of the reusable session pattern documented in the aiohttp client quickstart. Like HTTPX, aiohttp retrieves content but does not parse HTML or render JavaScript. It best fits an existing asyncio application, particularly one that handles streamed responses. Keep the session open across requests for connection reuse, but make retry, per-target concurrency, and parsing decisions in your application.
4. lxml
lxml is an HTML and XML parser built on libxml2 and libxslt. It is worth evaluating when parsing throughput matters, but performance depends on the documents, selectors, parser configuration, and surrounding application. Benchmark it on your workload instead of relying on a universal speed multiplier.
Its real advantage is full XPath support alongside CSS selectors. XPath lets you target elements by position, attribute, or relationship in ways CSS selectors cannot express, which pays off on deeply nested or inconsistent markup. Once you learn the syntax, you write shorter and more precise extraction logic.
The tradeoff is that XPath has a steeper learning curve than BeautifulSoup's forgiving API, particularly when a team does not already use XPath. Selector maintenance also remains your responsibility when page markup changes. For developers comfortable with that interface, lxml provides a mature parser with both XPath and CSS selection. Like BeautifulSoup, it only processes markup you have already retrieved and does not execute page JavaScript.
5. selectolax
Best for: teams that want a lightweight HTML parser built around CSS selection for pages they already fetch. selectolax uses the Lexbor engine and supports HTML parsing, DOM traversal, and CSS selectors. It is a parser, not a downloader, browser, or crawl scheduler; supply those layers separately, and benchmark representative documents rather than assuming a universal parsing-speed advantage.
BeautifulSoup, lxml, or selectolax: which parser should you use?
Choose BeautifulSoup for approachability, lxml for XPath and XML support, and selectolax for a lightweight CSS-selector-based HTML parser. All three are parsing libraries, not HTTP clients, browsers, or crawl schedulers.
BeautifulSoup is forgiving and has an accessible tree-navigation API. lxml supports HTML and XML parsing, XPath, and CSS selectors. selectolax uses the Lexbor engine and focuses on HTML parsing, DOM traversal, and CSS selectors. Parser performance varies by document and extraction pattern, so benchmark representative pages before making throughput the deciding factor.
6. Scrapy
Scrapy earns its place when your scraping job outgrows a single script and becomes a repeatable multi-page crawl with data flowing into a database or file. It ships as a full framework with an async engine that fetches many URLs at once, so you get concurrency without writing your own thread pool or asyncio loop.
The middleware and item pipeline system is where Scrapy separates itself from BeautifulSoup. Middleware lets you hook into every request and response to rotate user agents, retry failures, or plug in a proxy. Item pipelines catch each scraped record and let you clean, validate, and store it before it leaves the crawler. You get a structured place for logic that would otherwise sprawl across a BeautifulSoup script.
That structure comes at a cost. Scrapy imposes its own project layout, spider classes, and settings file, and you learn the framework's conventions before you extract a single field. For a quick pull from one static page, that overhead is pure friction.
The Scrapy versus BeautifulSoup framing misreads the two. BeautifulSoup parses HTML you already fetched. Scrapy fetches, crawls, and pipelines at scale. Reach for BeautifulSoup on a handful of pages, and reach for Scrapy when a repeatable crawl needs concurrency and a real output pipeline. Scrapy does not render JavaScript on its own, so dynamic sites require a browser integration or a separate managed rendering layer. Its official documentation covers spiders and broad crawls.
7. Crawlee for Python
Use Crawlee when you want one Python framework for both HTTP crawling and Playwright-based browser crawling. Its documented capabilities include automatic parallel crawling based on available resources, retries, proxy and session management, request routing, persistent URL queues, pluggable storage, and an asyncio-based interface.
Crawlee occupies the orchestration layer rather than the parsing-only layer. It can organize URL discovery, request handling, browser crawls, and output storage, but you still own deployment, extraction logic, browser capacity, and the configuration or sourcing of any proxies. Choose it over a small HTTP script when queues, resumability, routing, and mixed HTTP/browser workloads justify a framework. Its documented proxy rotation and session management are framework capabilities, not a promise that proxy endpoints, deployment, or access to protected targets come without configuration.
8. Playwright
Playwright is the tool to reach for when a page renders its content with JavaScript and you have no clean API to hit instead. Microsoft built it as a browser automation library, and it drives Chromium, Firefox, and WebKit through a single interface. That multi-browser support matters when a site behaves differently across engines, since you can test the WebKit path without maintaining a separate driver setup.
Playwright's auto-waiting can reduce explicit timing code around browser interactions by waiting for relevant actionability checks. It supports synchronous and asynchronous Python APIs and can run Chromium, Firefox, and WebKit in headed or headless mode. Performance relative to Selenium depends on the site, browser, interaction pattern, and test design, so benchmark the workflow instead of assuming one tool is universally faster.
The honest tradeoff is cost at scale. Browser-backed workflows use more compute than raw HTTP fetching, but memory use depends on the pages, browser configuration, and concurrency. Running dozens of concurrent sessions on a server adds up quickly, and you pay for that in compute and orchestration complexity. Playwright is a useful default when the target requires JavaScript execution or custom browser interactions. As browser-backed work grows, plan capacity, isolation, cleanup, and retries; first check whether every page actually requires rendering rather than applying a browser to the entire crawl.
A compact async pattern reuses one browser and bounds the number of active pages:
import asyncio
from playwright.async_api import async_playwright
URLS = ["https://example.com", "https://example.org"]
async def main():
semaphore = asyncio.Semaphore(4)
async with async_playwright() as playwright:
browser = await playwright.chromium.launch(headless=True)
context = await browser.new_context()
async def fetch(url: str) -> str:
async with semaphore:
page = await context.new_page()
try:
await page.goto(url, wait_until="domcontentloaded")
return await page.content()
finally:
await page.close()
try:
pages = await asyncio.gather(*(fetch(url) for url in URLS))
print([len(page) for page in pages])
finally:
await context.close()
await browser.close()
asyncio.run(main())A shared browser reduces repeated startup work, while the semaphore prevents unbounded page creation. Use separate browser contexts when jobs require stronger cookie or storage isolation.
9. Selenium
Selenium predates most modern scraping tools, and it still shows up in production because it drives real browsers through a mature driver ecosystem covering Chrome, Firefox, Edge, and Safari. Developers reach for it when a site only renders content after JavaScript runs, and a plain HTTP request returns an empty shell.
Selenium's trade-off is the operational cost of running browsers at crawl scale. Browser sessions consume more compute than raw HTTP requests, and teams must plan browser capacity, concurrency, retries, proxy configuration, and session cleanup. Selenium's current WebDriver ecosystem also includes BiDi and browser-specific CDP capabilities, so protocol architecture alone is not a sound basis for a blanket performance comparison with Playwright.
Selenium still wins in one place. If your team already runs a Selenium test suite, reusing its WebDriver setup may be easier than introducing a second browser tool. Check whether that setup also meets the scraper's capacity, session-isolation, and cleanup requirements.
For a greenfield scraping project on JS-heavy sites, Playwright is the better default. Choose Selenium when existing test infrastructure, an established driver setup, or a specific legacy dependency already ties you to it.
10. Crawl4AI
Best for: a self-hosted, AI-oriented crawler when your application needs Markdown from browser-backed pages or configured CSS/XPath JSON extraction. Crawl4AI's quickstart documents asynchronous crawling, HTML-to-Markdown conversion, optional LLM extraction, dynamic-page interaction, and multi-URL concurrency. Unlike a standalone parser, it can retrieve and process pages; unlike a managed API, it leaves browser operation, deployment, proxy configuration, and reliability work with your team.
Choose it when owning those controls is useful and you can test output quality on representative pages. LLM extraction is optional, not a substitute for checking required fields or normalizing data before it enters a production pipeline. If retrieval operations are the part you want to avoid, compare a managed API on cost per usable result instead.
Should you build or buy a production scraping pipeline?
Build when you need custom crawl control, persistent sessions, or browser interactions beyond an API's supported actions; evaluate managed retrieval when maintaining proxies, browsers, and target-specific failure handling does not differentiate your product. Compare both approaches on your actual targets, required output quality, engineering time, and cost per usable result. A production system includes more than fetching and parsing, so compare the full operating model rather than library license cost alone.
| Production concern | Self-managed Python stack | Managed retrieval and extraction API |
|---|---|---|
| Proxies and blocking | Select providers, route traffic, monitor failures, and tune target-specific behavior | Provider operates request routing and protection-handling infrastructure |
| Browser fleet | Install browsers, size workers, isolate sessions, patch versions, and clean up failures | Provider operates rendering capacity |
| Retries and concurrency | Define retry policy, backoff, limits, and per-site controls | Service handles retrieval-level retries and capacity; the application still handles API errors |
| Queues and scheduling | Run workers, queues, deduplication, checkpoints, and crawl state | May remain in your application unless the managed product includes crawl orchestration |
| Schemas and validation | Build selectors, normalization, schema checks, and fallback logic | Request structured extraction, then validate results in your application |
| Monitoring | Track success rates, latency, blocks, browser crashes, data quality, and cost | Monitor API usage and output quality without operating the underlying fleet |
| Sessions and interactions | Control cookies, login persistence, and arbitrary browser workflows | Check supported actions; use direct browser tools when persistent sessions or custom behavior are essential |
| Commercial and data boundaries | Operate the stack and its hosting or vendor components | Review pricing, usage limits, data-handling requirements, and dependence on the API |
Scrapy and Crawlee are strong choices when crawl behavior, request routing, and storage are part of your core system. Playwright and Selenium are better fits when you need persistent sessions, custom interaction sequences, browser debugging, or behavior beyond a managed API's supported actions. Context.dev fits teams that primarily need retrieved or extracted web data for AI pipelines and do not want to maintain proxies, browser fleets, retries, and scaling infrastructure. The production web scraping pipelines in Python guide covers queueing, validation, observability, and deployment in more depth.
Where DIY Python scraping breaks down
Fetching, parsing, and browser automation solve different problems, and protected targets may still rate-limit or block requests. The failure mode varies by site: a pipeline might receive a 403 response, a challenge page, or content that lacks the expected fields. Test the targets you actually need rather than assuming a particular request volume will trigger blocking.
IP-based blocking can become an operational concern on some targets, but not every pipeline needs rotating proxies. If target testing shows that IP routing is necessary, a self-managed stack must source proxies, configure routing, and monitor whether responses contain usable data.
Some targets present challenges or evaluate request and browser signals. HTTP clients and parsers do not solve interactive challenges, and running a browser alone does not guarantee access. If these protections affect your targets, account for the resulting failures and maintenance in your evaluation; neither a library nor a managed API guarantees success.
Each of those fixes adds work beyond the library's fetching or parsing API. The scraping code stays small, and the surrounding infrastructure for proxies, CAPTCHA solving, and fingerprint evasion grows into a system you maintain. Context.dev is an option when you want that request handling managed alongside content extraction. Our guide to what a web scraper is includes a build-vs-buy framework for deciding when that project stops being worth owning.
11. Context.dev: the managed API alternative for clean, LLM-ready data
Choose Context.dev for Python scraping when proxy rotation, browser maintenance, and bot challenges are taking time away from using the data. Context.dev provides a managed retrieval, rendering, and extraction layer through its Web APIs. It is designed for teams that want web data without operating proxy, browser, retry, CAPTCHA, and anti-bot infrastructure themselves. No service guarantees access to every target, so evaluate reliability and output quality against the pages your pipeline actually needs.
Pick the output with Scrape formats: markdown returns readable page content, html returns markup for an existing parser, and json returns JSON shaped by your schema. The Context.dev MCP server exposes web tools to agents. You can keep a Python processing pipeline while moving page retrieval and rendering into Context.dev.
Install the Python SDK with python -m pip install context.dev, set CONTEXT_DEV_API_KEY in your environment (the SDK requires Python 3.9 or newer), and call the current client.web.scrape method:
import os
from context.dev import ContextDev
client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])
page = client.web.scrape(
url="https://example.com",
formats={"markdown": True},
shared_params={"main_content_only": True},
)
if not page.markdown.success or page.markdown.data is None:
raise RuntimeError("Markdown extraction failed")
print(page.markdown.data)The consolidated Scrape API can request one or more formats, including rendered HTML, Markdown, CSS-selected fields, JSON, screenshots, images, original bytes, and product data. Each requested format reports its own success and data, so applications should validate the specific outputs they consume. For known fields on known pages, use CSS-selected fields with parsing rules or JSON extraction with JSON parameters; validate the returned fields in your application. For an open-ended live-web research task instead, the separate Answers API accepts a plain-language task and returns json_content with contributing URLs in sources. Its optional json_format is an example object, not JSON Schema; for known fields on known pages, Scrape parsing or JSON extraction is the more direct choice.
Context.dev is strongest when the desired result is retrieved content or structured data for an AI pipeline. Keep Scrapy or Crawlee when custom crawl orchestration is central to the system. Keep Playwright or Selenium when you need persistent logged-in sessions, multi-account browser management, detailed browser debugging, or custom interactions beyond the API's supported actions. For an end-to-end structured-data example, see the guide to multi-site product extraction with Python and Pydantic.
Final decision check
Before committing, test representative URLs and required fields with the layer identified in the table above. Measure usable output, not just successful HTTP responses, and decide who will own retries, concurrency limits, sessions, proxies, rendering, scheduling, and validation. The production web scraping pipelines in Python guide covers those implementation decisions in depth.
If IP routing is the only missing component, a proxy provider may fit the scraper you already operate. If maintaining retrieval and rendering is the larger problem, compare a managed API with your own stack on target-specific output quality, session requirements, engineering time, and cost per usable result.
FAQs
Is Scrapy better than BeautifulSoup? Scrapy is a full crawling framework, while BeautifulSoup is only an HTML parser. Scrapy wins for large, multi-page crawls that need concurrency and pipelines, and BeautifulSoup wins for quick extraction from a handful of static pages. Pick based on scale, not preference.
Do I need Selenium or Playwright if I already use Requests? You need a browser tool only when the content you want loads through JavaScript after the initial request. Requests fetches raw HTML and never runs scripts, so pages that render client-side return empty markup. Reach for Playwright when that happens, and keep Requests for static endpoints.
What are the top Python tools for rotating proxies automatically? Context.dev, Scrapfly, and ZenRows manage proxies as part of their scraping services. Context.dev is a fit when you need rendered content as Markdown or schema-based JSON without maintaining a proxy pool. Bright Data and Oxylabs offer rotating proxy gateways for scrapers that keep rendering and parsing in Python; Scrapy can integrate those gateways or proxy middleware.
Which proxy provider should I choose for Python web scraping? Evaluate Bright Data or Oxylabs residential proxies when you need a rotating gateway for an existing scraper. Choose Context.dev when you want a managed scraping API that includes proxy handling and returns content. Compare providers on your target sites, session requirements, and cost per usable page.
Which Python scraping tools help bypass anti-bot detection? Context.dev provides managed proxy and browser handling for Python callers. For a self-managed stack, curl_cffi addresses TLS and HTTP fingerprinting, while Camoufox and SeleniumBase address browser automation signals. Those tools still need suitable IP routing and target-specific testing. No library or API guarantees access to every protected site.
When does it make sense to use a scraping API instead of a library? Evaluate Context.dev when operating proxies, browsers, and anti-bot integrations costs more time than your team wants to spend on retrieval. Context.dev can return Markdown, rendered HTML, or configured JSON output through the consolidated Scrape endpoint. Libraries remain useful when you need a custom crawler or direct browser control.
Is Context.dev a replacement for Scrapy or Playwright? Context.dev can replace the retrieval and rendering layer when your workflow needs page content or structured fields. You can also keep Scrapy for crawl scheduling or an existing parser and retrieve HTML through Context.dev. Use Playwright directly when you need browser behavior beyond the API's supported actions.