How do website crawler APIs manage rate limits and failed pages?
Short answer
A managed crawler API handles rate limits by running your crawl inside a concurrency cap, rotating proxies, and retrying throttled or blocked fetches, so your code never implements backoff. It handles failed pages by reporting each page’s status code and success flag instead of aborting the crawl. On Context.dev, concurrency is capped per plan (Free 2, Developer 10, Pro 100, Growth 250, Scale 500), every crawl returns numSucceeded, numFailed, and numSkipped, failed or blocked pages are not billed, and Batches refunds the credits reserved for pages that fail.
Rate limits show up in two places: the limits the API puts on you, and the limits target sites put on the crawler. The first is a concurrency cap per account. Context.dev allows 2 concurrent requests on the free plan, 10 on Developer, 100 on Pro, 250 on Growth, and 500 on Scale, with no per-minute cap by default, and the dashboard shows your effective limits. The second is handled inside the service: when a target answers 429 or a bot challenge, the crawler rotates proxies, waits, and retries within the same request, which is why rendering and anti-bot handling are part of the base price rather than a flag you set.
Failed pages should be data, not exceptions. A Context.dev crawl returns every page with its statusCode and success flag, plus numSucceeded, numFailed, and numSkipped in the response metadata, so a dead link or a blocked page does not stop the rest of the site from coming back. Single scrapes behave the same way: a 404 returns a normal response whose output has success set to false and an error_code of NOT_FOUND. Your pipeline can log failures, retry them later, or drop them without wrapping every call in error handling.
Billing follows the same rule. Crawl charges 1 credit per successful page and nothing for failed or blocked pages. A single scrape is free when every output fails, except for a missing page (404), which costs the base price because the target was reached and reported missing. Batches reserves credits when you submit, charges only successful URLs, and refunds the rest when the job settles. Monitors apply it over time: failed or skipped runs are free. On a site with many dead or protected links, that rule changes the bill more than the plan price does.
For large jobs, move the retry problem off your servers entirely. Batches runs in the background within your concurrency limit, reports each item’s status, and sends a webhook when the job settles. Whatever endpoint you use, read the statusCode on every page, keep your own list of URLs to retry, and set maxPages or a credit budget so a misbehaving site cannot run up the bill.
| Endpoint | Throttled or blocked fetches | How failures are reported | Billing for failed pages |
|---|---|---|---|
| Scrape | Proxy rotation and anti-bot handling inside the request | success flag and error_code on every output | Free when every output fails; a missing 404 page costs the base price |
| Crawl | Same path as Scrape for every page | statusCode and success per page; numFailed and numSkipped totals | Not billed |
| Batches | Runs in the background within your concurrency limit | Status per item, plus a completion webhook | Not billed; reserved credits refunded when the batch settles |
| Monitors | Scheduled runs retry on the next interval | Run history and delivery attempts in the API | Failed and skipped runs are free |
| Plan | Concurrent requests | Per-minute cap |
|---|---|---|
| Free | 2 | None by default |
| Developer | 10 | None by default |
| Pro | 100 | None by default |
| Growth | 250 | None by default |
| Scale | 500 | None by default |
| Enterprise | Custom | Custom |
| If this is your situation | Use |
|---|---|
| Crawling sites with many dead or protected links | Crawl, which bills only successful pages |
| Thousands of URLs where you do not want to manage retries | Batches |
| You need more parallelism than your plan allows | A higher plan, or Enterprise custom limits |
| You want to know when a page comes back or changes | A page Monitor |
Last reviewed