Developers building Retrieval-Augmented Generation (RAG) pipelines, LLM training corpora, and autonomous AI agents routinely hit a critical ingestion bottleneck. Client-side rendered Single-Page Applications (SPAs) built with React, Vue, and Angular return empty HTML shells when queried with standard HTTP fetchers. Getting the actual data requires executing JavaScript, managing network waits, and pruning messy DOM nodes into clean text.
There are two primary ways to solve this. The traditional route involves a complex do-it-yourself headless browser setup using Playwright. The modern alternative relies on a managed web scraping API designed for AI agents, which renders JavaScript in a stealth browser and converts dynamic pages into clean, LLM-ready Markdown in a single API call. This guide breaks down the mechanics of scraping SPAs, the hidden costs of scaling self-hosted browsers, and how to implement both approaches.
Why Do Standard HTTP Scrapers Return Empty Shells on React, Vue, and Angular Apps?
Traditional server-rendered websites construct the complete HTML document on the server and send populated markup over the wire. Modern SPAs rely on Client-Side Rendering (CSR) or deferred hydration instead.
According to W3Techs Usage Statistics (October 2026), React powers 6.0% of all websites globally, while Next.js has grown to 3.7%. Vue.js accounts for 0.8% of the JavaScript library market, and Angular covers 0.3% overall. When a developer targets these sites with a standard HTTP client such as Python's requests.get(), Node's fetch(), or cURL, the server returns an HTTP 200 OK status alongside a minimal HTML bootstrap shell:
<!DOCTYPE html>
<html>
<head>
<title>App Dashboard</title>
<script defer src="/static/js/main.8f92a.js"></script>
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
</body>
</html>Scraping a React, Vue, or Angular single-page application with a plain HTTP client like requests or fetch returns an HTTP 200 status with an empty <div id="root"></div> shell because the content only materializes after a browser executes the JavaScript bundle. Prices, reviews, product listings, and dashboards all render dynamically after the browser runs the code. Static HTTP fetchers request the page and read whatever the server returns immediately, meaning the scraper succeeds but the data is completely missing.
Why Should You Convert Rendered SPA HTML into Main-Content Markdown for LLMs?
Even after executing JavaScript to populate the DOM, feeding raw rendered HTML directly into an LLM or embedding model wastes tokens and degrades retrieval accuracy.
Feeding raw rendered HTML into an LLM wastes up to 98% of your context window on markup and navigation noise. Across 2026 benchmarks using the o200k_base tokenizer, raw HTML markup is a median 30.3 times larger than its visible text, with median pages ranging from 78,925 to 128,051 tokens. Converting the rendered DOM to Markdown and stripping page chrome with mainContentOnly reduces the median page to under 3,000 tokens.
Key studies highlight this massive bloat:
- 30x Markup Bloat: A 2026 empirical study of 193 web pages found that the median raw HTML page consumes 78,925 tokens, whereas extracted Markdown is just 1,189 tokens (UntilNextSession Benchmark, 2026). Format conversion combined with main-content extraction cuts 98.1% of tokens.
- Context Window Saturation: An October 2026 benchmark measuring 20 live public pages found that the median raw rendered HTML page was 128,051 tokens. Extracting main-content LLM Markdown reduced the median to 2,948 tokens (Leif Albrand, DEV Community, Oct 2026).
- Navigation Boilerplate Leak: Naive DOM extraction leaks hidden CSS elements like collapsed menus and cookie banners. A benchmark by URLpipe evaluating 33 complex pages showed that with standard Mozilla Readability, 21.3% of the extracted Markdown consisted of boilerplate text a human reader never sees.
How Do You Scrape a Single-Page App Yourself with Playwright?
When developers build an in-house SPA scraper, the standard open-source choice is Microsoft's Playwright driving headless Chromium. Getting complete, clean Markdown requires handling four distinct engineering hurdles.
How Do You Wait for SPA Hydration Without Hanging on Network Idle?
In an SPA, page.goto(url) resolves by default on the load event, which fires before the asynchronous data requests finish populating the DOM. Developers often fall into the networkidle trap, waiting until there are no network connections for 500 ms. On modern SPAs with websocket heartbeats or streaming Server-Sent Events (SSE), networkidle never triggers and causes timeouts. The reliable pattern is navigating with waitUntil: 'domcontentloaded' and explicitly calling page.waitForSelector('.target-component') to confirm hydration.
How Do You Trigger Infinite Scroll and Lazy-Loaded Components?
SPAs defer rendering below-the-fold components using IntersectionObserver or virtualized lists. If the headless browser never scrolls the viewport, lazy-loaded text sections and documentation tables never attach to the DOM. A custom script must inject a scroll loop (window.scrollBy), pause for network mutations, and monitor the scroll height.
How Do You Strip Page Chrome and Convert to Markdown?
Playwright only returns a rendered DOM. You must run page.evaluate() inside the browser context to remove <nav>, <header>, <footer>, and <script> elements, then pass the sanitized HTML string into a converter like turndown or markdownify.
Here is a complete DIY Playwright implementation combining these steps:
// DIY Route: Self-hosted Playwright + Turndown
import { chromium } from "playwright";
import TurndownService from "turndown";
async function scrapeSpaWithPlaywright(url: string, readySelector: string) {
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const page = await context.newPage();
try {
// 1. Navigate and wait for SPA hydration
await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30000 });
await page.waitForSelector(readySelector, { timeout: 15000 });
// 2. Scroll to trigger IntersectionObserver / lazy-loaded components
for (let i = 0; i < 3; i++) {
await page.evaluate(() => window.scrollBy(0, window.innerHeight));
await page.waitForTimeout(800);
}
// 3. Prune navigation, footers, scripts, and chrome in the DOM
const cleanHtml = await page.evaluate(() => {
const selectorsToRemove = [
"nav", "header", "footer", "aside", "script", "style",
"[role='navigation']", ".cookie-banner", ".newsletter"
];
selectorsToRemove.forEach((sel) => {
document.querySelectorAll(sel).forEach((el) => el.remove());
});
const main = document.querySelector("main, article, [role='main']") || document.body;
return main.innerHTML;
});
// 4. Convert pruned HTML into Markdown
const turndown = new TurndownService({ headingStyle: "atx", codeBlockStyle: "fenced" });
return turndown.turndown(cleanHtml);
} finally {
await context.close();
await browser.close();
}
}What Does Self-Hosted Playwright Actually Cost in Memory and Proxy Bandwidth?
Scaling a local Playwright script exposes severe operational bottlenecks. A cold headless Chromium instance consumes 50 MB to 150 MB of RAM before loading a single pixel, and 300 MB to 500 MB of RAM per concurrent instance when rendering a modern JavaScript page. Exploring the memory and maintenance cost of self-hosted headless browsers reveals that parallel worker environments can experience severe memory leaks.
Residential proxies multiply this expense. An October 2026 benchmark by SOAX measured scraping a JS-heavy page: a headless browser consumed 4.0 MB per page. This is 22x more proxy bandwidth than an optimized extraction request. Combined with 70% to 85% success rates against modern bot protection platforms, self-hosted fleets become 3x to 6x more expensive per successful page.
How to Scrape Dynamic JavaScript Single Page Applications for LLM Training Data
To extract LLM-ready training data from an SPA, your pipeline must execute client-side JavaScript in a headless browser, wait for asynchronous state hydration, strip navigation chrome, and convert the rendered DOM into Markdown.
Instead of managing Chromium pools and proxy rotation manually, developers can offload the entire pipeline to Context.dev. Every POST /web/scrape request runs in a real stealth headless browser that executes client-side JavaScript, resolves dynamic network requests, and handles anti-bot challenges.
While self-hosted Playwright gives developers low-level control over Chromium, each active browser instance consumes 300 MB to 500 MB of RAM and pulls roughly 4.0 MB of residential proxy bandwidth per page, which is 22 times more traffic than request-based extraction. Context.dev (POST /web/scrape) replaces that browser fleet with a single API call: passing formats: { markdown: true } and sharedParams: { mainContentOnly: true } renders JavaScript in a stealth browser, handles proxy escalation, strips page boilerplate, and returns clean Markdown for 1 credit per page.
How Do You Scrape an SPA to Markdown in TypeScript and Python?
By leveraging the official SDKs, converting a URL to Markdown takes only a few lines of code. Setting sharedParams.mainContentOnly retains only the primary content region.
Here is the verified implementation using the Context.dev TypeScript SDK (npm install context.dev, requires Node.js 20+):
import ContextDev from "context.dev";
const client = new ContextDev({
apiKey: process.env.CONTEXT_DEV_API_KEY,
});
const page = await client.web.scrape({
url: "https://example.com",
formats: { markdown: true },
sharedParams: { mainContentOnly: true },
});
if (!page.markdown.success || page.markdown.data === null) {
throw new Error("Markdown extraction failed");
}
console.log(page.metadata.title);
console.log(page.markdown.data);For Python pipelines (pip install context.dev, requires Python 3.9+), the Markdown formatting options look nearly identical:
import os
from context.dev import ContextDev
client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])
page = client.web.scrape(
url="https://example.com",
formats={"markdown": True},
shared_params={"main_content_only": True},
)
if not page.markdown.success or page.markdown.data is None:
raise RuntimeError("Markdown extraction failed")
print(page.metadata.title)
print(page.markdown.data)How Do You Handle Infinite Scroll and Clicks with Browser Actions?
When scraping an SPA that requires scrolling or waiting for a specific component to hydrate before extracting data, you can pass up to 5 ordered instructions via sharedParams.actions.
import ContextDev from "context.dev";
const client = new ContextDev({
apiKey: process.env.CONTEXT_DEV_API_KEY,
});
const page = await client.web.scrape({
url: "https://example.com/products",
formats: { markdown: true },
sharedParams: {
mainContentOnly: true,
excludeSelectors: [".cookie-banner", ".newsletter"],
actions: [
{
type: "scroll",
direction: "down",
amount: "viewport",
maxScrolls: 3,
},
{
type: "waitFor",
selector: ".product-card:nth-child(20)",
},
],
},
});
if (page.markdown.success) {
console.log(page.markdown.data);
}How Does Self-Hosted Playwright Compare to a Managed Web Scraping API?
The table below contrasts the engineering overhead of DIY scraping against a managed architecture.
| Dimension | DIY Self-Hosted Playwright | Managed Scrape API (POST /web/scrape) |
|---|---|---|
| Setup & Maintenance | High engineering overhead. Requires packaging 1.5 GB to 3.0 GB Docker images with system libraries, managing browser context pools, and spending ~40% of engineering bandwidth fixing patches. | Zero browser infrastructure. Install the SDK via npm or pip, set an API key, and make a single client.web.scrape() call. |
| JavaScript Rendering | Manual orchestration. You write custom waitForSelector logic, avoid networkidle hangs, build window.scrollBy loops, and absorb 300 MB to 500 MB+ of RAM per active browser tab. | Always-on real browser rendering. Automatically executes React, Vue, and Angular bundles. Supports declarative actions (scroll, waitFor) for infinite scroll. |
| Proxies & Anti-Bot | You buy, configure, and rotate residential proxies separately. A default headless browser consumes ~4.0 MB of proxy bandwidth per page with 15% to 30% block rates. | Built-in stealth browsers, automatic residential proxy escalation, and bot-challenge handling included in every request at no extra bandwidth surcharge. |
| Output Format | Returns raw DOM HTML averaging ~78,900 to 128,000 tokens per page. Requires chaining custom DOM pruning scripts and external libraries like Turndown. | Returns LLM-ready GitHub Flavored Markdown with one-flag boilerplate stripping, cutting token usage by up to 98% vs. raw HTML. |
| Cost | Predictable TCO is poor. Combines always-on cloud compute, per-GB residential proxy bills, and retry overhead, costing 3x to 6x more per successful page at scale. | Predictable credit pricing. 1,000 free credits/month, then $19/month for 7,500 credits. A standard rendered Markdown scrape costs 1 credit. |
Frequently Asked Questions
Can Python requests or BeautifulSoup scrape a React or Vue single-page app?
No. requests and BeautifulSoup only download and parse the initial HTTP response from the server, without executing JavaScript. On a client-side rendered React, Vue, or Angular app, that initial response is an empty HTML container (<div id="root"></div>) and a <script> tag. You need either a headless browser like Playwright or a JavaScript-rendering scraping API.
Why does waitUntil: 'networkidle' time out on modern single-page applications?
Playwright's networkidle waits until there are zero network connections for at least 500 milliseconds. Modern SPAs frequently maintain open WebSocket connections, Server-Sent Events (SSE), analytics pings, or background polling, so the network never goes completely idle. Waiting for a specific DOM selector is much more reliable.
How many LLM tokens does converting HTML to main-content Markdown save?
Converting raw rendered HTML to main-content Markdown typically reduces token counts by 90% to 98%. In 2026 benchmarks using the o200k_base tokenizer, raw HTML pages averaged between 78,925 and 128,051 tokens per page, while cleaned main-content Markdown averaged between 1,189 and 2,948 tokens.
Does Context.dev render JavaScript automatically on every scrape request?
Yes. Every request to POST /web/scrape runs in a real stealth browser that executes JavaScript automatically. You do not need to toggle a rendering flag. A standard scrape with JavaScript rendering and managed proxies costs 1 credit, and the free plan includes 1,000 credits per month.
How can I scrape thousands of SPA pages for LLM training data or RAG indexing?
Instead of scaling a cluster of headless Chromium containers, you can submit up to 25,000 URLs in a single asynchronous job using the Batch API (POST /v1/batch/submit) or crawl an entire domain to receive every rendered page as clean Markdown.