TL;DR
- Check for real paginated URLs or an authorized data endpoint before automating clicks or scrolling. Either source may let you request each page directly.
- When interaction is necessary, use a bounded sequence of click, wait, or scroll actions. Context.dev Scrape supports one to five browser actions per request.
- Deduplicate results by stable item ID or URL. Stop when an explicit end marker appears or no new IDs arrive, and mark results incomplete if you hit an action or request cap.
- For public pages, use ordinary Scrape requests without assuming page state carries across calls. Browser profiles separately support reusable authenticated sessions on paid plans.
Why click-to-load and infinite scroll break static scraping
A plain HTTP request can miss items that a browser displays after you click or scroll. The server may return an initial HTML page, while JavaScript later fetches more items and inserts them into the page. A scraper that reads only the initial response never sees those additions.
Real pagination is the simplest case because each page has its own URL. A load-more button usually requests the next batch without changing the URL, while infinite scroll triggers requests as the viewport moves. Those differences determine whether you can request separate pages or need to reproduce an interaction.
Before automating a browser, inspect the page’s links and URL changes. Then use the DevTools Network tab to check whether clicks or scrolling produce an authorized, stable XHR or fetch response with the fields you need. If they do, you may be able to request that response directly within the site’s access and rate limits. If they do not, browser actions can trigger the interaction before extraction.
Decision flow: pick the simplest extraction path
Check for real page URLs before automating a browser. Open the target page, click its next-page control, and watch the address bar and links in DevTools. If each page has a stable URL, request those URLs directly and stop when the next link disappears.
| Pattern | How to detect it | Recommended approach | When browser automation is unnecessary |
|---|---|---|---|
| Real pagination | A next link or page number leads to a distinct URL | Scrape each URL | The URLs expose the needed items |
| Paginated JSON | DevTools Network shows an authorized fetch or XHR response with a cursor or page parameter | Request the response within the site’s permissions and rate limits | The response contains the needed fields |
| Load-more control | Earlier checks fail, and a click adds items to the page | Click, then wait for new content | An authorized data response supplies the same items |
| Infinite scroll | Earlier checks fail, and scrolling triggers new content | Scroll and wait in bounded steps | An authorized paginated feed supplies the items |
If URLs do not expose the content, inspect the Network tab before choosing the click or scroll path. A stable, authorized JSON response may give you explicit page boundaries and item IDs that the rendered page does not expose. Check the site’s terms and pace requests before using it.
When neither direct path works, use one click followed by a wait for a selector that identifies the added content. Use bounded scrolling only when content depends on scroll position. A page reaching the bottom does not establish that an infinite feed has ended.
Automating interactions with Context.dev Scrape
Context.dev Scrape browser actions run in order and return the page’s final state. For a public page with a load-more button, use perform to click it and waitFor to confirm that the expected item appeared. This example uses the TypeScript SDK and parse rules to return product names and links.
// scrape.mts: run with tsx after installing context.dev
import ContextDev from "context.dev";
const client = new ContextDev({
apiKey: process.env.CONTEXT_DEV_API_KEY,
timeout: 120_000,
maxRetries: 0,
});
const url = "https://example.com/products";
const parseParams = {
rules: {
items: {
selector: ".product",
type: "list" as const,
output: { name: "h2", url: "a@href" },
},
},
};
const clicked = await client.web.scrape({
url,
formats: { parse: true },
parseParams,
sharedParams: {
actions: [
{ type: "perform", action: "Click the Load more button" },
{ type: "waitFor", selector: ".product:nth-child(20)" },
],
},
});Choose a waitFor selector that identifies content the click should add. The example’s twentieth product is a page-specific check, not a selector you can reuse unchanged. For an infinite-scroll page, make a separate request with a fixed scroll limit.
const scrolled = await client.web.scrape({
url,
formats: { parse: true },
parseParams,
sharedParams: {
actions: [
{ type: "scroll", direction: "down", amount: "viewport", maxScrolls: 3 },
],
},
});Each request above stays within the documented limit of one to five actions. Neither request can inspect an intermediate DOM state and decide to click again until a button disappears. Ordinary separate Scrape requests start fresh navigation, so the scroll request does not continue from the click request.
Check each response before adding records. The example below rejects failed or partial output, converts relative links to absolute URLs, and stops collecting if a snapshot yields no new URLs.
const seen = new Map<string, { name: string; url: string }>();
for (const page of [clicked, scrolled]) {
if (page.isPartial || !page.parsed?.success) {
throw new Error("Scrape returned incomplete parsed output");
}
const data = page.parsed.data as {
items?: Array<{ name: string; url: string }>;
};
let added = 0;
for (const item of data.items ?? []) {
if (!item.url) continue;
const absoluteUrl = new URL(item.url, url).href;
if (seen.has(absoluteUrl)) continue;
seen.set(absoluteUrl, { ...item, url: absoluteUrl });
added++;
}
if (added === 0) break;
}A zero-new-URL stop can mean the feed ended, but it cannot prove completeness. Keep any additional requests under a fixed count and deadline, and mark results incomplete when a cap stops collection. For transient failures, cap 5xx retries and honor Retry-After on 429 responses. Do not retry 400, 401, or 403 responses as transient errors.
Validating output, deduplicating, and knowing when to stop
Check the parsed output before accepting any items. A successful HTTP response can still contain a failed output, so require parsed.success, inspect isPartial, and confirm that parsed.data.items matches the list your parse rules requested. Treat an empty list or partial output as a result to investigate, not proof that the page has no more items.
Deduplicate each batch before deciding whether to continue. Resolve each item’s relative link against the returned page URL, then use that absolute URL or a stable item ID as the key in a Set. Count newly added keys rather than rendered cards. A later request may return the same items again, and a virtualized feed may remove older cards from the DOM while new ones appear.
Bound every extraction run with MAX_PAGES, MAX_SCROLLS, MAX_ATTEMPTS, and a deadline. MAX_PAGES limits paginated requests, MAX_SCROLLS limits scrolling work, and MAX_ATTEMPTS limits retries for transient failures. Context.dev’s timeout and error guidance provides the basis for handling failed requests without retrying indefinitely.
Stop when a validated batch adds zero new IDs or when you verify an end marker, such as an exhausted next-page cursor. Zero new IDs can also mean the request repeated an earlier page, so record that stop as uncertain unless you verified the end. If any cap or the deadline ends the run first, mark the dataset incomplete rather than claiming you exhausted the feed.
Stateless extraction versus persistent browser sessions
Ordinary Context.dev Scrape requests do not resume the previous request’s page. Browser actions run within one request, which returns the page output after the action list finishes. A later request without a browser profile navigates again, so it cannot continue clicking through the earlier page’s DOM. For the public-page pattern in this guide, we use bounded actions and extract results from each response.
Context.dev also offers paid-plan browser profiles that preserve authenticated session state across Scrape requests. A profile supports one concurrent request and does not support MFA. It preserves a login, but it does not provide open-ended Playwright control. If your workflow must inspect intermediate states and decide each next action in a long-running session, a hosted browser/CDP service such as Browserless is a better fit.
Responsible pacing and the limits of guaranteed completeness
Set a per-host request limit before repeating clicks or scrolls. Respect published rate limits and space requests accordingly. When a host returns 429, pause and honor its Retry-After value before trying again. Retry transient 5xx responses only within your attempt limit and deadline.
A virtualized feed can remove older items from the page as new ones appear. Its DOM item count may stay constant even while the feed loads new records, so counting visible elements cannot establish completeness. Collect stable item IDs as you go and deduplicate them. If a bounded scroll yields no new IDs, stop that run and mark completeness unknown. Reaching the visible bottom does not prove the feed has ended.
When you need a complete dataset, prefer authorized paginated URLs or a stable data endpoint. An explicit end marker or exhausted cursor gives you an ending condition that scrolling alone cannot provide.
How ScrapingBee and Microlink handle the same problem
ScrapingBee fits an existing HTTP extraction pipeline when the page needs a predictable sequence of selector-driven actions. Its js_scenario instructions run in order within a request. You can click a button, wait for an element, and scroll a specified distance. ScrapingBee also documents an infinite_scroll option with a maximum scroll count and a delay, so you can bound a feed crawl without writing the scroll loop yourself.
Microlink fits pages that need custom logic to decide when to stop. Its load-more tutorial uses a Puppeteer-based function to check for the button, click it, and wait for the item count to grow. The function stops when the button disappears or the click ceiling is reached. For a single show-all control, Microlink documents a simpler click-and-wait option. For large feeds, Microlink recommends checking for an underlying JSON endpoint before relying on repeated browser interactions.
Comparison table
For bounded public-page interactions, Context.dev lets you request structured output without maintaining a browser. ScrapingBee fits an existing selector-driven HTTP pipeline, while Microlink gives you more control over when a page-driven loop stops.
| Tool | Interaction model | Documented limit | Output | Best fit |
|---|---|---|---|---|
| Context.dev | Ordered click, wait, and scroll actions in one Scrape request | 1–5 actions per request | Parsed JSON with field rules | Bounded interactions and structured extraction through one API |
| ScrapingBee | Ordered js_scenario instructions or infinite_scroll | No fixed step limit stated in the cited docs | HTML, Markdown, or extracted data | Selector-driven requests, especially when its scroll instruction fits |
| Microlink | Puppeteer function can check item counts between clicks | Tutorial uses a developer-set click ceiling | Function-defined extracted data | Custom page-driven loops with explicit stop checks |
FAQ
Can I click “Load more” until the button disappears in one request? No. Context.dev browser actions run in a fixed sequence, and your code receives the page only after the sequence finishes. You cannot inspect the button and decide whether to click again within that request. If the site exposes page URLs or an authorized data endpoint, use those for a repeatable loop.
How many actions can I chain? You can specify one to five actions per Scrape request. A click, a wait for a selector, and a scroll count as three actions.
Does scrolling to the bottom guarantee every item loaded? No. An infinite feed may fetch more items after a delay, and a virtualized list may remove older items from the DOM. Use bounded scrolls and waits, record each newly discovered item, and mark the extraction incomplete if you hit your limit without a clear end marker.
Do I need a browser profile for public pages? Usually not. Ordinary Scrape requests can perform bounded interactions without a saved session. Context.dev also documents browser profiles for reusable authenticated sessions on paid plans, but public-page extraction generally does not need one.
What if items have no stable IDs? Build a fingerprint from fields such as a normalized title and price, then compare fingerprints across results. Different items can share those fields, so a repeat or a request with no new fingerprints is a stopping signal, not proof that you captured the full list.
Conclusion
Prefer real paginated URLs or an authorized data endpoint when a site exposes them. Page boundaries or cursors give you a clearer way to establish completeness than clicks and scrolling, especially when a feed loads items unpredictably. Treat interaction automation as a bounded best effort, and report an incomplete result when you cannot verify the end of the list.
Our JavaScript scraping API roundup covers vendor selection. This guide covers the interaction logic beneath that choice.