Introducing web search monitors

How to Scrape Up to 100,000 URLs in One Job with the Context.dev Batches API

Context.dev Batches (POST https://api.context.dev/v1/batch/submit) scrapes up to 100,000 URLs, or crawls a site from a start URL or sitemap, as one asynchronous job, and returns Markdown, HTML or other Scrape Anything outputs for every page by polling, a signed webhook, or downloadable result files kept for 180 days. Credits are reserved when you submit, only successful pages are charged starting at 1 credit per page, and unused reservations are refunded when the batch settles.

Tools used in this guide

When should you use a batch instead of single scrapes or a crawl?

Context.dev Batches runs bulk web collection in the background when your workload is larger than a single request or a synchronous site crawl. Developers building RAG pipelines, training datasets, and AI agents can choose between four core data endpoints depending on how many URLs they have and whether they already know the specific URLs they need to fetch.

EndpointRouteScaleWhen to use it
ScrapePOST /v1/web/scrape1 page per requestFetch one known URL synchronously when your application or agent is waiting on the immediate response.
CrawlPOST /v1/web/crawlUp to 500 linked pages in one responseDiscover and extract up to 500 linked pages from a starting URL in a single synchronous call.
BatchesPOST /v1/batch/submitUp to 100,000 known URLs, or a crawl of more than 500 pagesRun a large URL list or a large site crawl asynchronously in the background and collect results by polling, webhook, or NDJSON files.
Map URLsGET /v1/web/urlsSite URL discoveryList a site's URLs from its sitemaps and links so you can filter them and submit the selected URLs in a batch.

Step 1: Submit a URL list

Context.dev Batches accepts between 1 and 100,000 URL records in a single POST https://api.context.dev/v1/batch/submit request authenticated with Authorization: Bearer $CONTEXT_DEV_API_KEY. The Batches overview specifies that you must set input.mode to "scrape" to process a URL list, or "crawl" if you need the engine to follow links from a start URL or read a domain's sitemap.

When working in scrape mode, you pass 1 to 100,000 URL objects in the input.data.urls array. Each entry requires a public url using the HTTP or HTTPS protocol. You can also attach an optional itemId string and an optional meta object to hold your own application records. Both itemId and meta are passed directly through to the results so you can match outputs back to your database.

Inside input.data, you must set at least one output format in formats to true. You can combine markdown, html, screenshot, images, bytes, parse, product, highlights, and json. Every page in the batch uses the exact same per-page options as the synchronous scrape endpoint. You can define sharedParams for custom headers, browser actions, content filters like mainContentOnly, geolocation options, wait conditions, and PDF parsing tools such as setting sharedParams.parsers.pdf.ocr to "auto".

You also gain precise control over cache and timeouts. The maxAgeMs parameter controls cache age in milliseconds. It defaults to three days (259,200,000 ms) and accepts values up to 365 days, or 0 to fetch fresh content. The timeoutOpts parameter dictates the per-page deadline, which defaults to 90,000 ms but can extend to 300,000 ms. Setting return-partial within timeoutOpts ensures that if one format times out, the successfully completed outputs are still preserved.

Submitting a valid batch payload returns an immediate HTTP 202 Accepted response containing the batch object. You must save the id property from this object to check progress later. The response object details input.submitted to count total URLs sent, input.reserved to count pages the batch will attempt after duplicate and invalid URLs are removed, and credits.reserved showing the initial credit hold. Any malformed links are grouped under invalid_urls, listing up to 100 rejected URLs with the exact rejection reason.

TypeScript SDK Example

TypeScript
import ContextDev from "context.dev";
 
const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });
 
const batch = await client.batch.submit({
  input: {
    mode: "scrape",
    data: {
      formats: { markdown: true, html: true },
      urls: [
        { url: "https://example.com/about", itemId: "about", meta: { section: "company" } },
        { url: "https://example.com/careers", itemId: "careers", meta: { section: "company" } },
      ],
      sharedParams: {
        mainContentOnly: true,
      },
    },
  },
  "Idempotency-Key": "site-import-2026-11-17",
});
 
console.log(batch.id, batch.status);

cURL Example

Shell
curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: site-import-2026-11-17" \
  -d '{
    "input": {
      "mode": "scrape",
      "data": {
        "formats": { "markdown": true, "html": true },
        "urls": [
          {
            "url": "https://example.com/about",
            "itemId": "about",
            "meta": { "section": "company" }
          }
        ],
        "sharedParams": {
          "mainContentOnly": true
        }
      }
    },
    "tags": ["site-import"]
  }'

Step 2: Make the submit safe to retry with an Idempotency-Key

Context.dev Batches supports an Idempotency-Key request header so your pipeline can safely retry a submission after a network timeout without creating duplicate jobs or reserving credits twice. This header can hold up to 200 characters and remains scoped uniquely within your organization.

If your network connection drops during a submit call, simply retry the exact same request. Retrying a request with the same Idempotency-Key and an identical request body returns the original batch object instead of executing a new batch. However, sending the same Idempotency-Key with a modified request body triggers an HTTP 409 error with the code IDEMPOTENCY_KEY_CONFLICT to protect against accidental overlaps.

Step 3: Track progress or receive a webhook

Context.dev Batches lets you track a running job either by polling GET /v1/batch/{batch_id} or by configuring a signed webhook that fires as soon as the batch reaches a final state. The lifecycle of a batch moves through active statuses (queued, running, cancelling) into final statuses (completed, cancelled, failed). A status of completed strictly means every accepted page was attempted. Individual pages inside a completed batch can still fail due to bad links or target website rules, so developers must evaluate progress.failed and the page_errors summary.

If you prefer polling, you can request the batch status every 10 to 30 seconds while the batch remains active. The response provides critical counters including progress.succeeded, progress.failed, and progress.pending. It also breaks down billing in the credits object, highlighting reserved, refunded, OCR charged, and net credits used. The page_errors array summarizes page failures grouped by specific error codes. The results property, which houses file download links, remains null until the batch becomes final.

To automate tracking, you can configure batch webhooks by adding a webhook object to your submit payload with your endpoint URL, such as webhook: { url: "https://app.example.com/hooks/context", retry: {} }. Including the retry object enables explicit delivery retries, while omitting it defaults to best-effort delivery. The API will respond to this submission with a webhook_secret exactly once. You must store this secret server-side to verify the X-Context-Signature header on incoming deliveries. The platform dispatches batch.completed, batch.failed, and batch.cancelled events containing the event type, id, creation timestamp, and batch data.

Step 4: Read the results

Context.dev Batches exposes a paginated JSON endpoint at GET /v1/batch/{batch_id}/results once the batch reaches a final status of completed, cancelled, or failed. Attempting to request this endpoint before the batch is final returns an HTTP 409 error with the code BATCH_NOT_COMPLETED.

The results pagination requires you to supply a limit between 1 and 100, which defaults to 25. You cycle through pages by passing the next_cursor string from the previous response as your cursor parameter as long as has_more evaluates to true. Result pages may close early to maintain payload sizes below 8 MB, so your loop must depend entirely on the has_more boolean rather than a strict record count.

Batch result records are not returned in submission order. You must match each record back to your source database using the itemId you provided during submission, reading back your custom meta object alongside it. A successful page record includes a status of "ok", the credits_used, your custom identifiers, and a result object holding the requested extraction formats. A failed record returns a status of "error", accompanied by an error_code, a diagnostic message, and your identifiers.

Page error codes include NOT_FOUND for missing pages, WEBSITE_BLOCKED when targets block automated access, and WEBSITE_ACCESS_ERROR for general network failures. Pages blocked by your defined access policies show ACCESS_POLICY_DENIED, while pages exceeding their deadline flag as REQUEST_TIMEOUT (timed-out pages always get one automatic retry before returning this error). Content issues trigger UNSUPPORTED_CONTENT or PDF_IMAGES_ONLY. You can resolve image-only PDFs by enabling sharedParams.parsers.pdf.ocr in your request. Unexpected system issues return INTERNAL_ERROR.

Step 5: Download result files for large batches

Context.dev Batches packages finished results into gzipped NDJSON files in results.files so you can download tens of thousands of pages in parallel without paging through JSON cursors. Once a batch concludes, the primary GET /v1/batch/{batch_id} route populates these file properties.

Every entry within results.files contains a URL to a compressed NDJSON file holding up to 50 individual JSON result records per file, one per line. For context, a batch of 100,000 URLs typically produces around 2,000 of these gzipped NDJSON result files.

Security restrictions apply to these downloads. The signed URLs in the payload expire 24 hours after the API call that generated them, indicated by results.expires_at. If your links expire before you finish downloading them, you simply retrieve the batch object again to receive a fresh batch of 24-hour links. The underlying files themselves are safely retained for 180 days after the batch finishes before they are permanently deleted.

Step 6: Retry failed pages

Context.dev Batches makes it straightforward to recover from transient target errors by filtering for failed records and submitting only those URLs as a new batch. You retrieve your results, isolate the objects where the status is "error", and inspect the specific error_code to see if parameter adjustments are necessary.

For example, if you encounter numerous REQUEST_TIMEOUT errors, you might boost the timeoutOpts to the maximum 300,000 ms before resubmitting. You can then submit this refined URL list through POST /v1/batch/submit using a brand new Idempotency-Key, making sure to preserve the original itemId and meta values so the retried results integrate cleanly into your system.

If you need to halt operations, sending a request to POST /v1/batch/{batch_id}/cancel stops the batch from starting new pages and transitions the status to cancelling. Any pages actively processing will finish, and then the batch assumes a cancelled state. If you attempt to cancel a job that has already finished, you receive a 409 error for BATCH_NOT_CANCELLABLE. To proactively remove your data before the 180-day window expires, calling DELETE /v1/batch/{batch_id} permanently purges the finished batch and all associated files.

Crawl a whole site as a batch

Context.dev Batches can also discover and scrape an entire website in the background when you set input.mode to "crawl" instead of passing a fixed URL list. Background async crawls surpass the 500-page limit of synchronous crawl requests while offering 180-day result file retention.

To configure a crawl, you start from a starting URL by defining source.type as "start_url" alongside the specific url, or you point it at a domain's sitemap. You define your page ceiling by passing maxUrls inside the controls object. The API sets input.reserved_is_ceiling to true because the batch might finish below your maxUrls limit if it runs out of in-scope pages. Unused credits are safely refunded when the job settles.

The active batch object surfaces specific metrics under the crawl key. This includes crawl.source, the max_pages allowed, the max_depth (which registers as 0 for sitemap crawls), follow_subdomains, and any url_pattern or exclude_url_pattern rules you defined.

TypeScript Crawl Example

TypeScript
import ContextDev from "context.dev";
 
const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });
 
const batch = await client.batch.submit({
  input: {
    mode: "crawl",
    data: {
      formats: { markdown: true, html: true },
      source: {
        type: "start_url",
        url: "https://example.com/docs",
        controls: {
          maxUrls: 1000,
          maxDepth: 3,
          followSubdomains: false,
          regex: "^https://example\\.com/docs/",
        },
      },
      sharedParams: {
        mainContentOnly: true,
      },
    },
  },
  tags: ["docs-crawl"],
  "Idempotency-Key": "docs-crawl-2026-11-17",
});
 
console.log(batch.id, batch.status);

Limits to plan around

Context.dev Batches enforces payload, concurrency, rate, and credit reservation limits so you can size your queues predictably. Each batch accommodates between 1 and 100,000 URLs while capping the total request body at 24 MiB.

Your subscription plan governs active concurrency for batches in the queued, running, or cancelling states. The Free plan permits 1 active batch, Developer allows 2, Pro and Growth allow 5, and the Scale and Scale 2.5M plans allow up to 20. Exceeding your concurrency allowance triggers an HTTP 403 error for BATCH_LIMIT_EXCEEDED.

The global rate limit allocates 1,000 units per minute per organization. Submitting a new batch consumes 50 units, whereas all other endpoints for status checks, cancellations, or deletions require only 1 unit. Before creating a batch, the system calculates the maximum credit cost to process all accepted pages. If your balance cannot cover this credits.reserved total, the API halts execution with an HTTP 402 USAGE_EXCEEDED error.

Security teams restricting API key scopes must apply batches:write to enable submit, cancel, or delete actions, and batches:read for status checks and results fetching. Batches do not support zero data retention because they keep result files for 180 days, so setting zdr: "enabled" returns ZDR_NOT_SUPPORTED.

Limit or RequirementValue / Behavior
URLs per batch1 to 100,000 URLs
Maximum request body size24 MiB
Active batches by planFree: 1, Developer: 2, Pro: 5, Growth: 5, Scale: 20, Scale 2.5M: 20 (403 BATCH_LIMIT_EXCEEDED above limit)
Rate limit budget1,000 units/minute per organization (POST /v1/batch/submit = 50 units, all other batch endpoints = 1 unit)
Up-front credit checkMust cover credits.reserved for all accepted pages (402 USAGE_EXCEEDED if insufficient)
Restricted API key scopesbatches:write for submit, cancel, delete. batches:read for status and results
Result file retention180 days (signed download URLs refresh on each GET /v1/batch/{batch_id} and last 24 hours)
Zero data retention (zdr)Only disabled is supported (enabled returns ZDR_NOT_SUPPORTED)

Complete TypeScript example

Context.dev Batches can be integrated end to end in a single TypeScript workflow that submits a URL array, polls until completion, pages through all result records, and separates Markdown outputs from failed URLs. The script submits the URLs with specific item identifiers, polls for the batch status every 15 seconds, and handles the necessary JSON cursor loops to collect the extracted content seamlessly.

TypeScript
import ContextDev from "context.dev";
 
const API_KEY = process.env.CONTEXT_DEV_API_KEY!;
const BASE_URL = "https://api.context.dev/v1";
const client = new ContextDev({ apiKey: API_KEY });
 
const FINAL_STATUSES = new Set(["completed", "cancelled", "failed"]);
 
function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}
 
async function runBatchScrape(urls: string[]) {
  // 1. Submit the URL list with an itemId per URL and an Idempotency-Key
  const submitted = await client.batch.submit({
    input: {
      mode: "scrape",
      data: {
        formats: { markdown: true },
        urls: urls.map((url, index) => ({
          url,
          itemId: `page-${index + 1}`,
        })),
      },
    },
    "Idempotency-Key": "example-batch-2026-11-17",
  });
 
  console.log(`Submitted batch ${submitted.id} (reserved: ${submitted.input.reserved})`);
 
  // 2. Poll GET /v1/batch/{batch_id} every 15 seconds until a final status
  let batchStatus = submitted;
  while (!FINAL_STATUSES.has(batchStatus.status)) {
    await sleep(15_000);
    const statusRes = await fetch(`${BASE_URL}/batch/${submitted.id}`, {
      headers: { Authorization: `Bearer ${API_KEY}` },
    });
    if (!statusRes.ok) {
      throw new Error(`Status check failed: ${statusRes.status}`);
    }
    batchStatus = await statusRes.json();
    console.log(
      `Status: ${batchStatus.status} | succeeded: ${batchStatus.progress.succeeded}, failed: ${batchStatus.progress.failed}, pending: ${batchStatus.progress.pending}`
    );
  }
 
  // 3. Page through GET /v1/batch/{batch_id}/results using cursor while has_more is true
  const markdownByItemId: Record<string, string> = {};
  const failedPages: Array<{ itemId?: string; url?: string; error_code: string }> = [];
 
  let cursor: string | undefined = undefined;
  let hasMore = true;
 
  while (hasMore) {
    const params = new URLSearchParams({ limit: "100" });
    if (cursor) {
      params.set("cursor", cursor);
    }
 
    const resultsRes = await fetch(
      `${BASE_URL}/batch/${submitted.id}/results?${params.toString()}`,
      {
        headers: { Authorization: `Bearer ${API_KEY}` },
      }
    );
    if (!resultsRes.ok) {
      throw new Error(`Results fetch failed: ${resultsRes.status}`);
    }
 
    const page = await resultsRes.json();
 
    for (const record of page.data ?? page.records ?? []) {
      if (record.status === "ok" && record.itemId) {
        markdownByItemId[record.itemId] = record.result?.markdown ?? "";
      } else if (record.status === "error") {
        failedPages.push({
          itemId: record.itemId,
          url: record.url,
          error_code: record.error_code,
        });
      }
    }
 
    hasMore = Boolean(page.has_more);
    cursor = page.next_cursor ?? undefined;
  }
 
  // 4. Log net credits charged and summary counts
  console.log(`Net credits charged: ${batchStatus.credits.net}`);
  console.log(`Collected Markdown for ${Object.keys(markdownByItemId).length} pages`);
  console.log(`Failed pages: ${failedPages.length}`, failedPages);
 
  return { markdownByItemId, failedPages, netCredits: batchStatus.credits.net };
}
 
const sampleUrls = [
  "https://example.com/about",
  "https://example.com/pricing",
  "https://example.com/docs",
];
 
runBatchScrape(sampleUrls).catch(console.error);

cURL Equivalent

Shell
curl -X POST https://api.context.dev/v1/batch/submit \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: example-batch-2026-11-17" \
  -d '{
    "input": {
      "mode": "scrape",
      "data": {
        "formats": { "markdown": true },
        "urls": [
          { "url": "https://example.com/about", "itemId": "page-1" },
          { "url": "https://example.com/pricing", "itemId": "page-2" },
          { "url": "https://example.com/docs", "itemId": "page-3" }
        ]
      }
    }
  }'

What it costs

Context.dev Batches follows Scrape Anything pricing per page, charging 1 credit per successful page for standard outputs like Markdown or HTML and 5 credits per page with JSON extraction. Complex processing requires additional credits mapped in the credits and pricing documentation. Generating product formats or highlights requires an additional 1 credit, while recovering content via PDF OCR charges a separate 1 credit per recovered page directly to credits.ocr_charged.

The system protects users from overcharging through intelligent reservation mechanics. The API reserves the maximum page cost across all accepted URLs at submission time. A 10,000-URL Markdown batch reserves up to 10,000 credits at submission and is charged only for the pages that succeed when the batch settles. Any unused extraction modifiers, unattempted URLs, or failed accesses are seamlessly returned to your pool as credits.refunded, yielding your actual consumption under credits.net. Developers can test these workflows on the Free tier available on the pricing page, which provides 1,000 credits every month with no credit card required.

Frequently asked questions

How do I scrape a list of URLs with an API? Context.dev Batches lets you submit between 1 and 100,000 URLs in a single POST https://api.context.dev/v1/batch/submit request with input.mode set to "scrape". You choose one or more Scrape Anything formats such as markdown, html, or json, and Context.dev processes the entire list asynchronously in the background. When the job finishes, you can read the extracted content through paginated JSON or download gzipped NDJSON result files.

How many URLs can I submit in one batch? Context.dev Batches accepts from 1 to 100,000 URL records in a single batch submission, up to a 24 MiB request body limit. Duplicate URLs are automatically removed before execution, and any malformed addresses are reported in invalid_urls with a rejection reason. If you need to collect pages from a website without a prebuilt list, you can also run a batch in crawl mode with a maxUrls ceiling.

How do I know when a batch is finished? Context.dev Batches moves through queued, running, or cancelling before reaching a final status of completed, cancelled, or failed. You can poll GET /v1/batch/{batch_id} every 10 to 30 seconds to inspect progress.succeeded, progress.failed, and progress.pending. Alternatively, you can pass a webhook object when submitting the batch to receive a signed batch.completed, batch.failed, or batch.cancelled event as soon as the job finishes.

How long are batch results kept? Context.dev Batches retains your batch records and downloadable gzipped NDJSON result files for 180 days after the batch reaches a final state. The signed download URLs inside results.files expire 24 hours after the GET /v1/batch/{batch_id} call that returned them, and calling GET /v1/batch/{batch_id} again generates fresh 24-hour links at any point during the 180-day retention window. If you want to remove the data earlier, calling DELETE /v1/batch/{batch_id} permanently deletes a finished batch and its files immediately.

How much does bulk scraping cost? Context.dev Batches uses Scrape Anything per-page pricing, which costs 1 credit per successful page for standard outputs like Markdown or HTML and 5 credits per page when JSON schema extraction succeeds. The API reserves the maximum credit cost for accepted pages when you submit and refunds unattempted pages, failed scrapes, and unused extraction modifiers when the batch settles. You can start on the Free plan with 1,000 credits every month and no credit card required.

Get started

Get an API key with 1,000 free monthly credits at https://www.context.dev and submit your first batch.

Start building with live web context.

Scrape, search, and research the web with one API. Bring the results straight into your product.

1,000free credits

Every month. No credit card required.

Start for freeRead the docs