TL;DR
- Speed winner. selectolax with Lexbor parsed the benchmark document in 0.02 seconds. BeautifulSoup took 0.92 to 2.47 seconds depending on its backend, making it 37 to 98 times slower.
- XPath and Scrapy pick. Choose lxml for direct XPath support. Choose parsel when you want lxml performance with Scrapy-style CSS and XPath selectors.
- Easiest-to-learn pick. BeautifulSoup offers the most approachable API for prototypes and small scraping jobs, but its convenience adds substantial parsing overhead.
- Managed alternative. Use Context.dev when your AI pipeline needs clean JSON or Markdown without maintaining parsers, retries, concurrency, and site-specific extraction code.
Comparison at a glance
The independent Art of Web Scraping benchmark measured initial parsing on the same HTML. Network latency often dominates small scraping jobs, but parser choice affects CPU use at higher volumes.
| Library | Speed | HTML5 correctness | Selector support | Best for |
|---|---|---|---|---|
| selectolax | Fastest at 0.02 to 0.03 seconds | Full HTML5 parsing | CSS | Maximum throughput |
| lxml | Fast at 0.09 seconds | Forgiving, but not fully HTML5 compliant | XPath and CSS with cssselect | Fast extraction with XPath |
| parsel | Fast at 0.09 seconds | Inherits lxml behavior | XPath and CSS | Scrapy-compatible extraction |
| BeautifulSoup | Slowest at 0.92 to 2.47 seconds | Depends on backend. html5lib provides HTML5 behavior | CSS and tree traversal | Readable prototypes and messy input |
BeautifulSoup took 37 to 98 times longer than selectolax for initial parsing, depending on its backend. That independent range exceeds the commonly repeated vendor estimate of 30 times, although both comparisons point to the same performance order.
selectolax: built for throughput
selectolax is the best fit when HTML parsing consumes a meaningful share of CPU time. In an independent benchmark, its Lexbor backend parsed the test document in 0.02 seconds. lxml and parsel each took 0.09 seconds, while BeautifulSoup with lxml took 0.92 seconds on the same test (benchmark results).
selectolax gets its speed by exposing native C parsers through compiled Cython bindings. Lexbor and Modest perform HTML5 parsing outside Python, while the relatively thin Python API provides access to the resulting tree. That architecture reduces the interpreter work involved in building and querying large numbers of documents.
from selectolax.lexbor import LexborHTMLParser
tree = LexborHTMLParser(html)
title = tree.css_first("h1").text()
links = [
node.attributes.get("href")
for node in tree.css("a[href]")
]Lexbor should be the default backend for new code. selectolax still provides selectolax.parser.HTMLParser, which uses Modest, but the maintainers keep Modest for compatibility because its underlying C library is no longer maintained. The project now identifies Lexbor as the preferred backend (selectolax repository).
The smaller API creates real constraints. selectolax supports CSS selectors but not XPath, so you cannot directly reuse XPath-heavy extraction rules. Its node interface also offers fewer convenience methods than BeautifulSoup. The Lexbor and Modest classes expose similar APIs, but their feature sets can differ at the edges.
Choose selectolax for CPU-bound parsing across thousands or hundreds of thousands of pages. For small crawls dominated by network latency, its benchmark advantage may have little effect on total runtime.
lxml: the speed-and-power baseline everything else builds on
lxml remains the best general-purpose choice when extraction logic depends on XPath. Its C extension wraps libxml2 for parsing and exposes full XPath 1.0 support. Parsel wraps lxml with a smaller selector API, while BeautifulSoup can use lxml as an optional parsing backend.
import requests
from lxml import html
response = requests.get("https://books.toscrape.com/", timeout=10)
response.raise_for_status()
doc = html.fromstring(response.content)
books = doc.xpath('//article[@class="product_pod"]')
results = [
{
"title": book.xpath('string(.//h3/a)').strip(),
"price": book.xpath('string(.//p[@class="price_color"])')
}
for book in books
]XPath gives lxml an advantage for selectors that need text matching, positional rules, or relationships between elements. In one independent benchmark, lxml parsed the test document in 0.09 seconds. Selectolax took 0.02 seconds, while BeautifulSoup with the lxml backend took 0.92 seconds. The same benchmark shows that BeautifulSoup’s Python API adds substantial overhead even when libxml2 handles the initial parse.
lxml performs less well when code creates many independent elements or moves nodes between documents. libxml2 ties each element to a document context, which adds work during element creation and merging. Official lxml microbenchmarks measured independent element creation at roughly 12 times slower than cElementTree and cross-document moves at roughly 27 times slower in the tested configurations.
Installation can also cause friction when pip cannot find a compatible wheel. Source builds on Linux require libxml2, libxslt, and Python development headers. macOS source builds may require Xcode command-line tools or libraries installed through Homebrew. For extraction-heavy pipelines that need XPath, lxml usually earns that operational cost.
parsel: lxml's ergonomics layer for the Scrapy ecosystem
Parsel gives lxml a concise selector API built around chainable CSS and XPath queries. Scrapy uses Parsel under the hood, so its .css(), .xpath(), .get(), and .getall() methods feel familiar to anyone who has written a Scrapy spider. You can also use Parsel independently with requests, httpx, or stored HTML.
from parsel import Selector
html = """
<article class="card">
<h2>Parser guide</h2>
<a href="/guide">Read more</a>
</article>
"""
sel = Selector(text=html)
title = (
sel.css("article.card")
.xpath("./h2")
.css("::text")
.get()
)
url = sel.css("article.card a::attr(href)").get()The example starts with CSS, switches to XPath, and returns plain strings through .get(). Parsel also adds the nonstandard ::text and ::attr(name) pseudo-elements, which make common extraction tasks shorter than equivalent raw lxml expressions.
Parsel remains in lxml's performance class because lxml performs the parsing and selector evaluation. The wrapper mainly changes how you express queries rather than introducing a separate parsing engine. Scrapy users therefore get lxml's parsing behavior and XPath support without working directly with its lower-level tree API.
Parsel works best for selector-driven extraction, not general DOM manipulation. It lacks lxml-style native tree traversal and mutation methods. It also lacks a direct equivalent to lxml's text_content() for collecting all descendant text, so you usually select text nodes explicitly and join the returned values.
BeautifulSoup: the readable, forgiving default
BeautifulSoup prioritizes readable extraction code and forgiving input handling over throughput. In an independent benchmark, BeautifulSoup placed last for initial parsing and selector extraction. Initial parsing took 0.92 seconds with the lxml backend, compared with 0.02 seconds for selectolax using Lexbor.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
title = soup.find("h1").get_text(strip=True)
links = [a.get("href") for a in soup.select("article a[href]")]BeautifulSoup wraps another parser and adds its own tree-building API. The built-in html.parser backend avoids another dependency, while lxml parses faster. The html5lib backend follows HTML5 parsing rules more closely but produced the slowest result in the benchmark. Even with the lxml backend, BeautifulSoup cannot match raw lxml because its wrapper adds processing overhead.
BeautifulSoup handles poorly declared character encodings particularly well. Its UnicodeDammit component can recover text when incorrect or misplaced encoding declarations confuse other parsers, according to the lxml documentation. You can also use UnicodeDammit for decoding and pass the resulting Unicode to lxml when encoding recovery and higher throughput both matter.
Choose BeautifulSoup for prototypes, small scraping jobs, messy encodings, or codebases where readability carries more weight than parsing speed. Its familiar .find() and .select() methods keep extraction logic easy to review, but CPU-heavy pipelines should prefer selectolax or raw lxml.
Which library fits your use case
| Use case | Pick | Why |
|---|---|---|
| Maximum throughput at scale | selectolax with Lexbor | Its C-backed HTML5 engine delivered the fastest parsing and selector execution in the independent benchmark. |
| XPath-dependent extraction logic | lxml | Its libxml2 engine provides full XPath 1.0 support without another abstraction layer. |
| Existing Scrapy codebase | parsel | Scrapy selectors already use Parsel, so extraction code keeps the same chainable CSS and XPath API. |
| Beginners and readability | BeautifulSoup | Its approachable tree-navigation API makes small scripts and prototypes easier to write, provided parsing speed is secondary. |
When parsing HTML yourself stops being worth it
DIY parsing remains sensible for small, stable jobs. Selectolax, lxml, Parsel, and BeautifulSoup turn HTML into navigable trees, but you still write selectors, remove irrelevant content, normalize fields, and validate the output. A few predictable pages rarely justify another service.
Production pipelines create a larger maintenance burden because every source structures content differently. You may find a product name in a heading on one site and inside embedded JSON on another. Layout changes break selectors, while inconsistent fields can produce malformed records for downstream LLM consumers. You also own scheduling, retries, concurrency, rendering, site-specific adapters, and schema validation.
Context.dev provides a managed alternative for pipelines that need consistent web data across many sites. Our API handles scraping, crawling, and structured delivery, then returns clean JSON or Markdown through REST or MCP. Your application can send that output directly into an AI agent or LLM pipeline without choosing a parser or maintaining extraction infrastructure.
Context.dev fits teams whose parsing code has grown into an internal crawler. A local script or stable single-site integration may remain simpler with a Python library. Once site variation and output consistency consume regular engineering time, managed extraction can remove work that parser benchmarks do not measure.
Decision wrap-up
Choose the parser whose mechanism matches your constraint. Selectolax favors throughput, lxml supports XPath, Parsel fits Scrapy conventions, and BeautifulSoup prioritizes readable code.
Before committing, measure the work beyond parsing. If cleaning inconsistent markup, validating schemas, and maintaining extraction code consume more effort than parsing itself, use a managed API such as Context.dev to deliver structured JSON or Markdown directly to your LLM pipeline.
FAQs
Is selectolax faster than lxml?
Selectolax parsed benchmark input in 0.02 seconds versus 0.09 seconds for lxml. Its Lexbor engine runs parsing in C. Choose selectolax when parsing consumes meaningful CPU time.
How do BeautifulSoup and lxml differ?
BeautifulSoup wraps interchangeable parser backends and offers readable tree navigation. lxml directly wraps libxml2 and supports XPath. Choose BeautifulSoup for simplicity or lxml for faster extraction and richer queries.
Does BeautifulSoup support XPath?
BeautifulSoup does not expose XPath. Its lxml backend parses input but does not add an XPath API. Use CSS selectors or choose lxml or Parsel.
When should you use Parsel outside Scrapy?
Parsel works as a standalone selector library. Parsel wraps lxml with chainable CSS, XPath, and convenient extraction methods. Use it when you want Scrapy-style selectors without adopting Scrapy.
Which parser handles malformed HTML5 pages best?
selectolax Lexbor follows HTML5 parsing rules. BeautifulSoup with html5lib offers similar correction but runs much slower. Choose Lexbor when browser-like recovery and throughput both matter.
Can a managed API replace a parsing library?
A managed extraction API can replace local parsing when you need structured output. Context.dev returns JSON or Markdown instead of raw HTML. Structured delivery reduces parser maintenance and schema cleanup for LLM pipelines.