How do web scraping APIs handle dynamic content and JavaScript-heavy websites?
Short answer
They run the page in a headless browser instead of reading the first HTTP response. The API navigates to the URL, executes the page's JavaScript, waits for a readiness signal such as a selector appearing, optionally clicks or scrolls to trigger more loading, then serializes the rendered DOM. The better APIs escalate to a browser only when a plain fetch comes back incomplete.
A static fetch returns whatever the server sent first, which on a client-rendered site is often an empty shell like <div id="root"></div> and a bundle of JavaScript. The content you wanted never appears in that response, because it does not exist until a browser runs the script. This is not an edge case anymore: around 68% of target sites now rely on client-side JavaScript to render their content.
Once a browser is in the loop, the hard part is knowing when the page is done. Three readiness strategies are in common use. Waiting for a specific selector is usually the most precise choice, because you are asserting that the exact element you need exists. Waiting for network idle is less reliable on pages with analytics, polling, streaming, or persistent connections, since those keep the network busy indefinitely. A fixed delay is the crudest option, and it belongs in your toolkit as a fallback rather than as the default readiness signal.
Some content will not appear for any amount of waiting, because it is gated behind an interaction. Infinite scroll, tabbed panels, "Load More" buttons, and login walls all need the scraper to act on the page before reading it. Firecrawl exposes this as Actions, which lets you reach content behind login walls or paginated buttons. This tier carries the highest latency and the most complexity, so use it only where the content genuinely requires it.
Rendering usually changes what you get charged, and the multipliers vary a lot. On ScrapingBee, rendering is enabled by default at 5 credits, and a stealth proxy request with JavaScript can reach 75. Firecrawl adds 4 credits per page for JSON mode and another 4 for enhanced proxy. Context.dev bills one flat credit whether the page is static or a heavy single-page app needing a full headless render, and failed or blocked requests are never billed. It also auto-detects whether a page needs a render at all, and roughly 70% of requests serve from cache without one.
Keep rendering and unblocking separate in your head, because they solve different problems. Rendering gets you the DOM after scripts run. Unblocking gets you a usable response in the first place, through residential proxies, fingerprint rotation, and CAPTCHA handling. A page can render perfectly and still be unreachable behind Cloudflare, and a site can let you straight in and still hand back an empty shell.
| Approach | What it captures | Speed and cost | Best for |
|---|---|---|---|
| Static HTML fetch | The server's initial response | Faster and cheaper | Server-rendered pages and known JSON endpoints |
| Headless browser render | The DOM after scripts execute | Slower and more resource-intensive | Single-page apps and client-rendered content |
| Browser render with actions | Content loaded after clicks, typing, or scrolling | Highest latency and complexity | Infinite scroll, tabs, forms, and interactive pages |
| Provider | Rendering approach | Rendering surcharge | Output you get |
|---|---|---|---|
| Context.dev | Auto-detects when a render is needed, escalates proxies automatically | None, one flat credit either way | Markdown, HTML, and schema JSON |
| Firecrawl | Headless browser with Actions for clicks, scrolling, and login walls | JSON mode and enhanced proxy each add credits per page | Markdown, HTML, and schema JSON |
| ScrapingBee | Single render parameter with selector and network wait controls | Rendering is on by default and costs more than a plain fetch | HTML you parse, AI extraction as an add-on |
| Zyte | Browser HTML, actions, sessions, and unblocking | Usage-based | Rendered HTML plus automatic extraction for supported types |
| Bright Data | Scraping Browser and Web Unlocker | Bandwidth-based | Raw HTML, conversion is yours |
| If this is your situation | Use |
|---|---|
| The page is server-rendered | Static fetch, skip the browser |
| Content appears only after scripts run | Headless render with wait-for-selector |
| Content appears after a click or a scroll | Render with actions |
| You get blocked before rendering matters | Proxy escalation and unblocking |
Last reviewed