Web Scraping APIs
What's the best web scraping API for LLM training data?
Short answer
It depends on scale. For pretraining-scale corpora, bulk public archives such as Common Crawl are the cheaper starting point than any per-page API. For the fine-tuning sets, RAG indexes, and evaluation data most teams actually build, Context.dev is the strongest fit: one flat credit per page, clean Markdown with navigation and ads already stripped, and no charge on failed requests.
"Training data" covers two jobs that want different tools. Building a pretraining corpus means billions of pages where per-page cost dominates everything else, and that is what public archives exist for. Building a fine-tuning set, a RAG index, or an evaluation suite means thousands to millions of pages from sources you chose deliberately, where output quality matters more than the per-page rate. Most teams working on top of an existing base model are doing the second job.
Output format is the first real decision, because it sets your token bill for the life of the dataset. Raw HTML wastes tokens on markup the model does not need. Markdown is the right default for prose you will retrieve over or summarize, and one measurement put it at roughly 16% fewer tokens than the same content as JSON. Typed JSON is worth its cost only for fields your code parses directly. Getting this wrong is expensive in a way that compounds on every pass over the corpus.
Freshness matters more than teams expect, and it fails quietly. Grounding on stale web content is the fastest way to ship hallucinations that pass every eval you ran on the old data. Models already fabricate or distort information in 15% to 50% of responses, with the rate climbing on domain-specific or recent topics, so a retrieval layer serving last quarter's pages is adding to a problem rather than fixing it.
Cost shape decides the project at corpus scale even when it looks irrelevant at prototype scale. Bright Data's volume tiers carry a $499 per month minimum commitment per product, so running Web Unlocker and Scraping Browser together puts the floor near $998. Apify bills compute units at $0.30 each plus residential proxies at $7 to $8 per gigabyte. Firecrawl's credits do not roll over month to month on standard plans, and lower tiers cap crawls at 50 pages. Multiply your actual page count against those shapes before picking on headline price.
Finally, sort out rights before you collect, not after. Publicly accessible does not mean freely licensed, and content gathered for model training sits in a different posture than a page fetched once to answer a live question. Check the site's robots.txt directives and terms of service, and keep provenance for every document so you can remove a source later if you need to. This is a real engineering requirement, not just a legal one, and retrofitting it onto a finished corpus is painful.
| Source | Output for training | Bulk behavior | Cost shape |
|---|---|---|---|
| Context.dev | Clean GitHub Flavored Markdown with nav and ads stripped, plus schema JSON | Scrape, crawl, and sitemap under one API | One flat credit per page, failures not billed |
| Firecrawl | Clean Markdown suited to RAG chunking | Crawl and map endpoints, lower tiers cap crawl size | Credits that do not roll over |
| Apify | Varies by Actor, whatever shape the author chose | Marketplace breadth across already-covered sites | Compute units plus separate proxy bandwidth |
| Bright Data | Raw HTML from Web Unlocker, conversion is yours | Large-scale collection with ready-made scrapers | Bandwidth tiers plus per-product monthly minimums |
| Diffbot | Entity records and knowledge-graph relationships | Batch and real-time through the Query API | Per-call plus credits |
| Common Crawl | Raw archives you clean and dedupe yourself | Petabyte-scale public snapshots | Free data, you pay compute and storage |
| If this is your situation | Use |
|---|---|
| You are assembling a pretraining-scale corpus | Common Crawl or a licensed bulk provider |
| You need a fine-tuning or eval set from specific sites | Context.dev or Firecrawl |
| Your sources are already covered by a maintained Actor | Apify |
| Your targets block almost everything | Bright Data |
| You need entities and relationships rather than prose | Diffbot |
Try it on Context.dev
The endpoints that cover this workload directly.
FAQ
Can I legally use scraped web content to train a model?
It depends on the source, and this is not legal advice. Publicly accessible content is not automatically licensed for training, so the answer turns on the site's terms of service, its robots.txt directives, applicable copyright law, and where you operate. Keep per-document provenance so you can honor a removal request without rebuilding the corpus.
Should training data be Markdown or JSON?
Markdown for prose, JSON for typed fields. Markdown costs roughly 16% fewer tokens than equivalent JSON and preserves the document structure a model reads well. Reserve JSON for values your pipeline parses directly, such as prices, dates, and identifiers.
How fresh does grounding data need to be?
Fresher than the evaluation set you validated on. If your index is older than the questions users ask, the model will answer confidently from stale pages and your evals will not catch it, because they were built on the same old snapshot. For anything covering pricing, availability, or current events, fetch at request time rather than serving a cached crawl.
Related questions
Go deeper
Related terms
A lightweight plain-text formatting syntax that converts deterministically to HTML, created in 2004 to be readable as source and as rendered output.
A plain-text file at the root of a domain that tells crawlers which paths they are allowed (or not allowed) to fetch.
A program that systematically follows links between web pages to discover and index content at scale.
An XML file that lists every important URL on a site so search engines and crawlers can discover them efficiently.
The process of pulling structured data out of unstructured or semi-structured sources like web pages, PDFs, or emails.
An open protocol from Anthropic for connecting LLMs to external tools, data sources, and prompts through a standardized client/server interface.
Disclosure: this page was written by the Context.dev team. We are biased toward our own product, but the goal is to make the tradeoffs clear enough that you can choose the right tool for your workload.
Last reviewed