Web Scraping APIs
How do web scraping APIs handle dynamic content and JavaScript-heavy websites?
Short answer
They run the page in a headless browser instead of reading the first HTTP response. The API navigates to the URL, executes the page's JavaScript, waits for a readiness signal such as a selector appearing, optionally clicks or scrolls to trigger more loading, then serializes the rendered DOM. The better APIs escalate to a browser only when a plain fetch comes back incomplete.
A static fetch returns whatever the server sent first, which on a client-rendered site is often an empty shell like <div id="root"></div> and a bundle of JavaScript. The content you wanted never appears in that response, because it does not exist until a browser runs the script. This is not an edge case anymore: around 68% of target sites now rely on client-side JavaScript to render their content.
Once a browser is in the loop, the hard part is knowing when the page is done. Three readiness strategies are in common use. Waiting for a specific selector is usually the most precise choice, because you are asserting that the exact element you need exists. Waiting for network idle is less reliable on pages with analytics, polling, streaming, or persistent connections, since those keep the network busy indefinitely. A fixed delay is the crudest option, and it belongs in your toolkit as a fallback rather than as the default readiness signal.
Some content will not appear for any amount of waiting, because it is gated behind an interaction. Infinite scroll, tabbed panels, "Load More" buttons, and login walls all need the scraper to act on the page before reading it. Firecrawl exposes this as Actions, which lets you reach content behind login walls or paginated buttons. This tier carries the highest latency and the most complexity, so use it only where the content genuinely requires it.
Rendering usually changes what you get charged, and the multipliers vary a lot. On ScrapingBee, rendering is enabled by default at 5 credits, and a stealth proxy request with JavaScript can reach 75. Firecrawl adds 4 credits per page for JSON mode and another 4 for enhanced proxy. Context.dev bills one flat credit whether the page is static or a heavy single-page app needing a full headless render, and failed or blocked requests are never billed. It also auto-detects whether a page needs a render at all, and roughly 70% of requests serve from cache without one.
Keep rendering and unblocking separate in your head, because they solve different problems. Rendering gets you the DOM after scripts run. Unblocking gets you a usable response in the first place, through residential proxies, fingerprint rotation, and CAPTCHA handling. A page can render perfectly and still be unreachable behind Cloudflare, and a site can let you straight in and still hand back an empty shell.
| Approach | What it captures | Speed and cost | Best for |
|---|---|---|---|
| Static HTML fetch | The server's initial response | Faster and cheaper | Server-rendered pages and known JSON endpoints |
| Headless browser render | The DOM after scripts execute | Slower and more resource-intensive | Single-page apps and client-rendered content |
| Browser render with actions | Content loaded after clicks, typing, or scrolling | Highest latency and complexity | Infinite scroll, tabs, forms, and interactive pages |
| Provider | Rendering approach | Rendering surcharge | Output you get |
|---|---|---|---|
| Context.dev | Auto-detects when a render is needed, escalates proxies automatically | None, one flat credit either way | Markdown, HTML, and schema JSON |
| Firecrawl | Headless browser with Actions for clicks, scrolling, and login walls | JSON mode and enhanced proxy each add credits per page | Markdown, HTML, and schema JSON |
| ScrapingBee | Single render parameter with selector and network wait controls | Rendering is on by default and costs more than a plain fetch | HTML you parse, AI extraction as an add-on |
| Zyte | Browser HTML, actions, sessions, and unblocking | Usage-based | Rendered HTML plus automatic extraction for supported types |
| Bright Data | Scraping Browser and Web Unlocker | Bandwidth-based | Raw HTML, conversion is yours |
| If this is your situation | Use |
|---|---|
| The page is server-rendered | Static fetch, skip the browser |
| Content appears only after scripts run | Headless render with wait-for-selector |
| Content appears after a click or a scroll | Render with actions |
| You get blocked before rendering matters | Proxy escalation and unblocking |
Try it on Context.dev
The endpoints that cover this workload directly.
FAQ
How do I know if a page needs JavaScript rendering?
Fetch the raw HTML and search it for a distinctive phrase you can see in the browser. If the text is missing from the response but visible on screen, the page is client-rendered and needs a browser. Comparing the raw response against the rendered DOM takes a minute and saves you from paying render costs on pages that never needed them.
Is network idle a reliable signal that a page finished loading?
Not on modern pages. Analytics beacons, polling, streaming responses, and persistent connections keep the network active long after the content you want has painted, so network idle either fires late or never. Waiting for a specific selector is the more precise signal when you know what element you need.
Does JavaScript rendering change what I get charged?
On most APIs, yes. Rendering is commonly billed at a multiple of a plain fetch, and stealth or premium proxy tiers multiply it again. Context.dev is an exception in that a scrape is one flat credit whether or not the page needs a headless render.
Related questions
Go deeper
Related terms
A real browser engine running without a visible UI, controlled programmatically through an automation API.
The Document Object Model, a tree of objects that represents an HTML document in memory and lets JavaScript manipulate it.
A Microsoft-maintained library for driving Chrome, Firefox, and WebKit headlessly with a unified API.
A Node.js library from the Chrome team that drives Chromium over the DevTools Protocol, used for scraping, screenshots, PDF generation, and headless testing.
An open-source framework that drives real web browsers programmatically, originally built for end-to-end testing and now widely used for scraping JavaScript-heavy sites.
A challenge-response test designed to distinguish humans from bots, usually presented as image, audio, or behavioral puzzles.
Disclosure: this page was written by the Context.dev team. We are biased toward our own product, but the goal is to make the tradeoffs clear enough that you can choose the right tool for your workload.
Last reviewed