TL;DR
- Playwright is the best default for scraping JavaScript-heavy sites. Its locator model and auto-waiting remove much of the timing code that makes browser automation flaky.
- Selenium remains the right call when you already run Selenium Grid, share cross-browser test infrastructure, or maintain a legacy codebase built around WebDriver.
- Scrapy wins for high-volume crawls of static or lightly dynamic pages, where its asynchronous pipeline outruns browser-driven tools.
- Scrapy plus Playwright is a legitimate production architecture. Scrapy can orchestrate the crawl while Playwright renders only the pages that need JavaScript.
- A managed API is often the better fit when an LLM pipeline, agent, or RAG system needs clean Markdown rather than raw HTML and browser infrastructure.
Playwright vs Selenium vs Scrapy at a glance
Popularity does not tell you which framework fits your target site. What separates these three is how they handle JavaScript rendering, how much browser infrastructure they require, and whether they scale through browser concurrency or raw HTTP throughput. A site that renders its content client-side rules out a pure HTTP crawler. A million-page static crawl makes a headless browser the wrong tool.
| Playwright | Selenium | Scrapy | |
|---|---|---|---|
| Setup complexity | Low, with browser installation handled by the Playwright CLI | Medium, with language bindings and browser setup | Low for static pages, higher once browser rendering is added |
| Speed | Fast for browser-driven work | More protocol and wait-management overhead | Fastest at volume when no browser is needed |
| JavaScript rendering | Native | Native | Requires an integration such as scrapy-playwright or Splash |
| Waiting model | Built-in auto-waiting for relevant locator actions | Implicit and explicit waits configured by the developer | Not applicable without a browser integration |
| Browser support | Chromium, Firefox, and WebKit | Chrome, Edge, Firefox, Safari, and more | None natively |
| Best-fit use case | JavaScript-heavy pages that need reliable rendering | Existing Grid infrastructure and cross-browser reuse | Large static multi-page crawls |
Read this table as a decision tree, not a scoreboard. First determine whether the target requires JavaScript rendering. Then consider crawl volume and the infrastructure your team already runs. In most cases, those answers identify the right framework before you write any code.
Playwright: browser automation for modern JavaScript-heavy sites
Playwright is the strongest default for JavaScript-heavy sites because it addresses two common browser-scraping problems: knowing when an element is ready and isolating concurrent sessions. Its auto-waiting checks the states relevant to each action, such as whether an element is visible, stable, enabled, or able to receive events. That removes many arbitrary sleep() calls and reduces timing failures as pages hydrate and re-render.
Its browser contexts give each job an isolated cookie jar and storage state inside a browser process. You can run independent sessions without launching a completely new browser for every job. Playwright also supports Chromium, Firefox, and WebKit through one API, which makes it practical to reproduce rendering differences across engines.
For scraping, the locator API is especially useful. A locator identifies an element at the moment an operation runs instead of storing a stale element reference. That behavior is more resilient to the DOM churn produced by single-page applications.
Here is a minimal render-and-extract job for a JavaScript-heavy product page.
from playwright.sync_api import sync_playwright
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=True)
context = browser.new_context()
page = context.new_page()
page.goto("https://example.com/products")
cards = page.locator(".product-card")
cards.first.wait_for(state="visible")
for index in range(cards.count()):
card = cards.nth(index)
name = card.locator(".title").inner_text()
price = card.locator(".price").inner_text()
print(name, price)
browser.close()The explicit wait targets a meaningful state instead of a fixed delay, and every field read resolves against the current DOM.
The cost lands in infrastructure. Playwright installs browser binaries tied to its release, and the official browser documentation shows that those downloads consume hundreds of megabytes. Linux containers also need the required system libraries and fonts, which makes images larger than a plain Python runtime.
Memory is the harder ceiling. Browser contexts are lighter than separate browser processes, but pages still consume browser resources. As concurrency rises, you need limits per worker, process recycling, health checks, and enough capacity to absorb pages that use far more memory than average. Playwright scales well for browser work, but it cannot match the density of an HTTP crawler that never launches a rendering engine.
Selenium: the mature standard with the largest ecosystem
Selenium remains the framework with the deepest browser-automation ecosystem. WebDriver is a W3C Recommendation, and Selenium supports the major desktop browsers through bindings for several popular languages. If your organization already runs browser automation across multiple languages, that reach can matter more than a newer API.
The practical difference from Playwright appears in timing. Selenium offers implicit waits and explicit waits, but developers still need to choose and configure the right readiness conditions for dynamic pages. Selenium's own guide to waiting strategies explains why a completed page load does not guarantee that JavaScript-rendered elements are ready. Playwright builds more of those checks into locator actions, so routine interaction code tends to be shorter.
Here is the explicit-wait pattern for a common scraping task.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome()
driver.get("https://example.com/products")
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".product-title"))
)
titles = driver.find_elements(By.CSS_SELECTOR, ".product-title")
data = [title.text for title in titles]
driver.quit()The wait makes the render condition visible in the code. That control is useful, but every custom condition becomes another choice to maintain as the target changes.
Keep Selenium deliberately in three cases. First, your team already runs Selenium Grid and knows how to operate it. Second, you want to reuse cross-browser test infrastructure for controlled data collection. Third, your codebase contains substantial WebDriver logic and a migration would not remove enough maintenance to justify the effort. In each case, the existing investment can outweigh Playwright's cleaner defaults.
Scrapy: the throughput-first framework for large static crawls
Scrapy solves a different problem. Playwright and Selenium drive browsers, while Scrapy runs an asynchronous crawling engine that fetches many pages concurrently, parses responses, and follows links according to rules you define. If the target serves the needed content in its initial HTML and you need to crawl thousands of pages, Scrapy is built for the job.
The framework separates crawl behavior from data processing. A spider defines where to start and which links to follow, a callback extracts records and requests, and an item pipeline cleans, validates, deduplicates, or stores each result. Scrapy also provides concurrency controls, request delays, retries, and automatic throttling, so you do not have to assemble the orchestration around a basic HTTP client.
The following spider crawls a paginated product listing and extracts each card.
import scrapy
class ProductSpider(scrapy.Spider):
name = "products"
start_urls = ["https://example.com/products?page=1"]
def parse(self, response):
for card in response.css("div.product"):
yield {
"name": card.css("h2::text").get(),
"price": card.css("span.price::text").get(),
"url": response.urljoin(card.css("a::attr(href)").get()),
}
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, callback=self.parse)Scrapy cannot execute JavaScript on its own. The common solutions are scrapy-playwright, which sends selected requests through Playwright while preserving the Scrapy workflow, or a separate rendering service such as Splash. Both add browser binaries and their resource cost to what was otherwise a lightweight HTTP crawler.
That does not make the hybrid a bad design. It means you should reserve rendering for pages that require it. Let static pages use Scrapy's regular downloader and send only client-rendered pages through Playwright.
Where each framework breaks down at scale
The first month with any of these frameworks can look clean because you test against a stable target under low load. Failures emerge as concurrency rises, browsers update, and target sites change their markup or defenses. Each framework breaks differently, but all three need monitoring once the data matters.
Timing failures
Playwright's auto-waiting handles common element states, but it cannot infer every application-specific definition of readiness. A page can expose an element, fire another request, and repaint the same region after your first read. Selenium makes more of the waiting strategy explicit, which gives you control but also leaves you tuning conditions and timeouts across the scraper.
The solution is not longer fixed sleeps. Wait for the actual state the extraction depends on, such as a known response, a stable item count, or the visibility of a terminal element. Then record timeouts and retry rates so slow failures do not disappear inside a general retry loop.
Browser lifecycle failures
Playwright ties browser revisions to library releases, while Selenium's modern tooling can manage drivers automatically. Even so, browser automation still has a lifecycle. Versions change, container dependencies drift, pages crash, and orphaned processes consume resources. At production concurrency, you need controlled upgrades, worker recycling, and metrics for page crashes and memory growth.
Scrapy avoids most of that overhead until you add a browser integration. Once scrapy-playwright or Splash enters the stack, the browser layer needs the same operational care it would require in a standalone service.
Blocking and anti-bot systems
Proxy rotation and anti-bot handling can become a standing maintenance burden. A target that served plain HTML during development can later add rate limits, a JavaScript challenge, or fingerprint checks. A browser alone does not guarantee access, and an HTTP crawler can fail completely when content moves behind client-side rendering.
Treat access failures as a separate concern from parsing failures. Monitor status codes, challenge pages, response sizes, and extraction completeness independently. Otherwise a pipeline can mistake a block page for valid content and keep running.
Selector drift
Selector drift affects all three tools and often fails silently. A frontend team renames a CSS class or restructures a component, and a locator or CSS selector matches nothing. The crawl still completes, but its records are empty or wrong.
Schema validation is the defense. Set minimum field requirements, track record counts and null rates, and save a small sample of failed pages for inspection. A scraper that fails loudly is easier to repair than one that quietly publishes bad data.
How to choose between Playwright, Selenium, and Scrapy
Three questions decide which framework fits. Answer them in order before writing code.
1. Does the target require JavaScript rendering?
Fetch the page with a plain HTTP client and inspect the response body. If the data is already present, you have a static target and Scrapy will usually handle it with less overhead. If the data appears only after JavaScript runs, use a browser tool or find the underlying data endpoint.
2. What crawl volume do you need?
Browser tools spend memory and CPU per page. A high-volume crawl of JavaScript-rendered URLs needs tighter concurrency limits and more infrastructure than the same crawl over static HTML. Scrapy's asynchronous downloader can process static pages at much greater density because it does not launch a rendering engine.
3. What infrastructure do you already operate?
A team with a working Selenium Grid, established test suites, and mature CI around WebDriver may save more by keeping Selenium than it gains by migrating. A Python data team with existing Scrapy pipelines may get the best result by adding Playwright only for the dynamic portion of a crawl.
The right tool is not always the one with the cleanest greenfield API. Existing operational knowledge is a real part of the decision.
Migration paths that make sense
The most common browser migration is Selenium to Playwright. Teams make it when custom waits and driver-era maintenance dominate their scraping code. Playwright can remove a large portion of that timing logic, but a migration still needs regression tests for authentication, downloads, popups, and any application-specific readiness conditions.
The Scrapy plus Playwright hybrid is often a better architecture than a full rewrite. Scrapy handles scheduling, deduplication, retries, and item pipelines, while Playwright renders only marked requests. Use one hybrid crawl when static and dynamic pages share a schedule and output schema.
Split the crawl into separate jobs when the two portions have different schedules, proxy requirements, scaling profiles, or reliability targets. Independent jobs make each half easier to tune and monitor. If both halves move together, a single hybrid pipeline avoids duplicated orchestration.
When none of the three is the right call
Self-hosting Playwright, Selenium, or Scrapy makes sense when browser control is core to your product, when the workload is small enough that operations remain simple, or when you already run the infrastructure. In those cases, the frameworks give you control that a managed service cannot match.
The comparison changes when the crawler is only an input to an LLM pipeline, agent, or RAG system. A browser tool gives you rendered HTML. Your model usually needs clean Markdown or structured JSON, so you still have to remove navigation, cookie banners, scripts, and other boilerplate before indexing or prompting.
That output gap is where self-hosting becomes expensive. You maintain the fetch layer, the rendering layer, and the parsing layer, even though the pipeline only consumes the final document.
Context.dev's URL-to-Markdown API turns a URL into clean Markdown in one request. It handles rendering and proxy escalation behind the endpoint, which removes both the browser fleet and the cleanup code between a page and an LLM-ready document. For agents, Context.dev's MCP integration exposes web data as a tool call instead of a crawler process you keep alive.
The decision comes down to what your team should own. If browser infrastructure is part of the product, keep the framework. If web data is merely an input and the infrastructure is overhead, a managed API lets your team spend its time on the pipeline that uses the data.
FAQs
Is Playwright replacing Selenium?
Playwright is a strong default for greenfield browser scraping because its locators and auto-waiting reduce routine timing code. Selenium still has a large installed base, mature Grid infrastructure, broad language bindings, and extensive cross-browser coverage. Existing infrastructure can make Selenium the better choice even when Playwright offers a cleaner API.
Can Scrapy render JavaScript on its own?
No. Scrapy fetches and parses HTTP responses without executing page scripts. Add scrapy-playwright, Splash, or another rendering service when a target requires JavaScript, and account for the additional browser infrastructure.
Which framework is fastest at scale?
Scrapy is usually fastest for large static crawls because its asynchronous engine handles many concurrent requests without launching a browser. For JavaScript-heavy targets, performance depends on the page, concurrency limits, waits, and browser lifecycle strategy. Benchmark the real target instead of assuming one browser framework always wins.
Can you mix frameworks in one pipeline?
Yes. Scrapy plus Playwright is a common and sensible pattern. Scrapy handles crawl orchestration and item processing, while Playwright renders only the requests that need a browser.
How does a managed API fit an existing Scrapy or Playwright codebase?
A managed API can replace the request and rendering layer one target at a time. Existing spiders, queues, schemas, and downstream pipelines can remain in place while selected URLs return clean Markdown or JSON instead of raw HTML. That makes an incremental migration possible without rewriting the whole system.