Building a Production MCP Web Scraping Pipeline

TL;DR

• A production MCP scraping pipeline combines a narrow tool schema, a rendering and extraction layer, output validation, source provenance, and quality evaluation.
• The worked example uses Context.dev to retrieve JavaScript-rendered pages and return structured JSON without maintaining browsers, proxies, CAPTCHA handling, or site adapters.
• You will build a TypeScript MCP tool that validates requests, normalizes extracted data, and attaches the source URL, retrieval timestamp, extraction method, and content hash.
• You will add retries, rate limits, typed errors, security controls, and observability for failures and latency.
• A deployment rubric will measure extraction accuracy, schema validity, freshness, latency, blocked-request rate, and cost per successful result.

What an MCP web scraping pipeline actually needs to do

An MCP scraping tool should return a predictable data contract that an agent can use without interpreting page structure. The contract defines accepted inputs, required output fields, error states, and source metadata. Raw HTML lacks those guarantees and consumes context with navigation, scripts, cookie notices, and unrelated content.

Basic retrieval methods break when websites vary their markup or render content in the browser. A fetch call may receive an empty application shell instead of the product data visible to a user. Regex rules often fail after small template changes. Giving an agent unrestricted browser control creates a different problem because the agent must choose interactions, identify relevant content, and preserve evidence for each extracted claim.

A production pipeline needs five controls.

1. Narrow tool schemas constrain each tool to one intent and reject unsupported parameters before retrieval begins.

2. JavaScript rendering loads content that static HTTP requests cannot access, including data inserted after page load.

3. Validated output converts page content into typed JSON or clean Markdown. Schema validation prevents missing or malformed fields from reaching the agent.

4. Source provenance attaches the final URL, retrieval time, extraction method, and content hash to every result. Downstream systems can then cite and audit the source.

5. Measurable quality gates test field accuracy, schema validity, freshness, latency, blocked-request rate, and cost per successful result. These measurements reveal failures that a successful HTTP status cannot detect.

Designing the MCP server and tool schema

An MCP server should expose one tool for each agent intent. Register extract_structured_data, crawl_site, and url_to_markdown separately instead of adding optional crawl, render, action, and output flags to one scraper. Each schema then limits the parameters an agent can supply and gives the model a clearer basis for selecting the correct tool.

The following tool accepts a URL and returns a fixed company profile. A local adapter keeps the Context.dev request separate from the MCP boundary.

import { z } from "zod"
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"
import { extractCompanyWithContext } from "./context-adapter.js"
 
const Input = z.object({
  url: z.string().url()
}).strict()
 
const Output = z.object({
  companyName: z.string().min(1),
  summary: z.string().min(1),
  sourceUrl: z.string().url(),
  retrievedAt: z.string().datetime()
}).strict()
 
const server = new McpServer({
  name: "web-data",
  version: "1.0.0"
})
 
server.registerTool(
  "extract_structured_data",
  {
    description: "Extract a company profile from a public web page",
    inputSchema: Input.shape,
    outputSchema: Output.shape
  },
  async ({ url }) => {
    const extracted = await extractCompanyWithContext(url)
 
    const result = Output.parse({
      companyName: extracted.companyName,
      summary: extracted.summary,
      sourceUrl: extracted.finalUrl,
      retrievedAt: new Date().toISOString()
    })
 
    return {
      content: [
        {
          type: "text",
          text: JSON.stringify(result)
        }
      ],
      structuredContent: result
    }
  }
)
 
const transport = new StdioServerTransport()
await server.connect(transport)

Zod rejects invented inputs before the handler reaches Context.dev. Output parsing also prevents missing names, malformed URLs, or unstructured provider responses from crossing the tool boundary. A failed parse should become a typed tool error rather than an empty object that the agent may treat as valid data.

The client configuration starts this server as a local process or points to a remote MCP endpoint. During discovery, the client sends tools/list and receives the registered names, descriptions, and input schemas. The agent selects a tool and sends its arguments through tools/call. The MCP server validates the request, runs the matching handler, validates the result, and returns structured content.

Register crawl_site and url_to_markdown with independent handlers and output schemas. Separate registration prevents an agent requesting Markdown while supplying crawl-only parameters or attempting unsupported browser actions through the extraction tool.

Retrieving JavaScript-rendered pages

Static HTTP requests often miss content that JavaScript inserts after the initial document loads. Product grids, prices, and pagination may appear only after a browser executes scripts or triggers a network request. A retrieval layer should render these pages before extraction and support waits or browser actions when content loads after scrolling or interaction.

Self-managed rendering usually means operating a Playwright or Puppeteer browser pool. You must control concurrency, recycle crashed processes, cap memory use, rotate proxies, and detect blocked responses. Browser control remains appropriate when you need persistent login sessions or complex UI interaction. For routine extraction, browser operations add infrastructure that does not improve the final data format.

A managed endpoint moves rendering into the MCP tool handler’s retrieval stage. The following adapter keeps vendor configuration outside the tool schema, so the agent can provide a URL without controlling rendering credentials or internal options.

type RenderedPage = {
  url: string
  content: string
  contentType: "html" | "markdown"
}
 
async function retrieveRenderedPage(url: string): Promise<RenderedPage> {
  const endpoint = process.env.CONTEXT_RENDER_ENDPOINT
  const apiKey = process.env.CONTEXT_API_KEY
 
  if (!endpoint || !apiKey) {
    throw new Error("Context.dev rendering is not configured")
  }
 
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      authorization: `Bearer ${apiKey}`,
      "content-type": "application/json"
    },
    body: JSON.stringify({ url })
  })
 
  if (!response.ok) {
    throw new Error(`Rendering failed with status ${response.status}`)
  }
 
  return response.json() as Promise<RenderedPage>
}

The MCP handler calls retrieveRenderedPage after validating the requested URL and before extracting fields or validating output. Context.dev supplies the managed rendering and crawling layer, which removes browser, proxy, and CAPTCHA infrastructure from your application. Your pipeline must still enforce URL policies, validate extracted data, attach provenance, and handle failures.

Returning validated JSON or clean Markdown

The tool should choose its output shape according to the next consumer. Structured JSON supports agents that need named fields for decisions or storage. Markdown preserves headings, links, lists, and readable text for retrieval-augmented generation.

Treat every extraction response as untrusted until it passes validation. The following Zod schemas reject missing fields, unexpected fields, invalid URLs, and malformed timestamps before the MCP tool returns data.

import { z } from "zod"
 
const Product = z.object({
  name: z.string().trim().min(1),
  price: z.number().nonnegative().nullable(),
  currency: z.string().length(3).nullable(),
  availability: z.enum([
    "in_stock",
    "out_of_stock",
    "unknown"
  ])
}).strict()
 
const StructuredResult = z.object({
  data: Product,
  sourceUrl: z.string().url(),
  retrievedAt: z.string().datetime()
}).strict()
 
const MarkdownResult = z.object({
  markdown: z.string().trim().min(1),
  sourceUrl: z.string().url(),
  retrievedAt: z.string().datetime()
}).strict()
 
type ValidationResult<T> =
  | { ok: true, data: T }
  | {
      ok: false
      error: {
        code: "SCHEMA_VALIDATION_FAILED"
        issues: Array<{
          path: string
          message: string
        }>
      }
    }
 
function validate<T>(
  schema: z.ZodType<T>,
  raw: unknown
): ValidationResult<T> {
  const parsed = schema.safeParse(raw)
 
  if (!parsed.success) {
    return {
      ok: false,
      error: {
        code: "SCHEMA_VALIDATION_FAILED",
        issues: parsed.error.issues.map(issue => ({
          path: issue.path.join("."),
          message: issue.message
        }))
      }
    }
  }
 
  return { ok: true, data: parsed.data }
}

The MCP handler should pass the Context.dev structured extraction response into validate(StructuredResult, raw). When validation fails, the handler should return a typed tool error rather than an empty object or partially valid record. A typed failure lets the agent retry, choose another source, or report that extraction failed without inventing missing values.

For RAG ingestion, Context.dev’s URL-to-Markdown capability converts a live page into structured Markdown. The handler can validate that response with MarkdownResult before chunking or embedding it. Keep headings and source links because they preserve document structure, but remove scripts, navigation clutter, and repeated page furniture that would waste context.

Avoid making every JSON field optional to improve the apparent success rate. Model legitimate absence with nullable(), and treat other omissions as extraction failures. That distinction prevents incomplete data from crossing the tool boundary as a successful result.

End-to-end example: agent call to Context.dev to normalized output

The MCP client sends a narrow request that contains the target URL and extraction intent. The server owns the Context.dev credentials, rendering configuration, validation schema, and provenance fields.

import { z } from "zod";
 
const ToolInput = z.object({
  url: z.string().url(),
});
 
const Company = z.object({
  name: z.string().min(1),
  description: z.string().min(1),
  logoUrl: z.string().url().nullable(),
});
 
const NormalizedResult = z.object({
  data: Company,
  provenance: z.object({
    sourceUrl: z.string().url(),
    retrievedAt: z.string().datetime(),
    extractionMethod: z.literal("context.dev"),
  }),
});
 
type ToolInput = z.infer<typeof ToolInput>;
type NormalizedResult = z.infer<typeof NormalizedResult>;
 
// Hop 1. The agent runtime creates this MCP tool call.
const agentCall = {
  name: "extract_company",
  arguments: {
    url: "https://example.com/about",
  },
};
 
// Hop 2. Keep the vendor-specific request inside one adapter.
async function callContext(input: ToolInput): Promise<unknown> {
  const endpoint = process.env.CONTEXT_EXTRACT_URL;
  const apiKey = process.env.CONTEXT_API_KEY;
 
  if (!endpoint || !apiKey) {
    throw new Error("Context.dev configuration is missing");
  }
 
  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      authorization: `Bearer ${apiKey}`,
      "content-type": "application/json",
    },
    body: JSON.stringify({
      url: input.url,
      render: true,
      output: "json",
      schema: {
        name: "string",
        description: "string",
        logoUrl: "string|null",
      },
    }),
    signal: AbortSignal.timeout(30_000),
  });
 
  if (!response.ok) {
    throw new Error(`Context.dev request failed with ${response.status}`);
  }
 
  return response.json();
}
 
// Hop 3. The MCP handler validates input before making a network request.
async function extractCompanyTool(rawInput: unknown) {
  const input = ToolInput.parse(rawInput);
  const retrievedAt = new Date().toISOString();
  const contextResponse = await callContext(input);
 
  // Keep response mapping separate because API versions may wrap data differently.
  const extracted =
    typeof contextResponse === "object" &&
    contextResponse !== null &&
    "data" in contextResponse
      ? (contextResponse as { data: unknown }).data
      : contextResponse;
 
  // Hop 4. Invalid or incomplete extraction fails at the tool boundary.
  const data = Company.parse(extracted);
 
  // Hop 5. The server attaches provenance and validates the final object.
  return NormalizedResult.parse({
    data,
    provenance: {
      sourceUrl: input.url,
      retrievedAt,
      extractionMethod: "context.dev",
    },
  });
}
 
const result = await extractCompanyTool(agentCall.arguments);

The adapter calls a server-side Context.dev endpoint configured through CONTEXT_EXTRACT_URL. Keeping the endpoint and request mapping in one function lets you update API versions without changing the MCP tool contract.

A successful call returns one predictable object.

{
  "data": {
    "name": "Example Company",
    "description": "Example Company provides analytics software.",
    "logoUrl": "https://example.com/logo.svg"
  },
  "provenance": {
    "sourceUrl": "https://example.com/about",
    "retrievedAt": "2025-03-08T14:25:31.000Z",
    "extractionMethod": "context.dev"
  }
}

Zod rejects missing fields, invalid URLs, and unexpected output shapes before the agent receives them. The MCP server should convert those exceptions into typed tool errors rather than returning an empty object that the agent could mistake for a valid extraction.

Retries, rate limits, and failure handling

Retry only failures that another attempt can plausibly fix. Timeouts, connection resets, HTTP 429 responses, and most 5xx responses are transient. Blocked requests, unavailable rendering, invalid schemas, and unsupported URLs require a typed error because repeated calls will usually produce the same outcome. Empty extraction results can receive one retry before the handler reports EMPTY_RESULT.

A shared limiter should control concurrency and request spacing across every agent call. The example below uses Bottleneck to wrap the MCP tool handler and exponential backoff to retry transient failures.

import Bottleneck from "bottleneck";
 
type FailureCode =
  | "BLOCKED"
  | "RENDERER_UNAVAILABLE"
  | "INVALID_SCHEMA"
  | "EMPTY_RESULT"
  | "UPSTREAM_FAILURE";
 
class ToolFailure extends Error {
  constructor(
    readonly code: FailureCode,
    message: string,
    readonly retryable = false
  ) {
    super(message);
  }
}
 
const limiter = new Bottleneck({
  maxConcurrent: 5,
  minTime: 200
});
 
const sleep = (ms: number) =>
  new Promise(resolve => setTimeout(resolve, ms));
 
async function withRetry<T>(
  operation: () => Promise<T>,
  attempts = 3
): Promise<T> {
  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (error instanceof ToolFailure && !error.retryable) {
        throw error;
      }
 
      if (attempt === attempts - 1) {
        throw new ToolFailure(
          "UPSTREAM_FAILURE",
          "Extraction failed after retries"
        );
      }
 
      const delay = 500 * 2 ** attempt + Math.random() * 250;
      await sleep(delay);
    }
  }
 
  throw new ToolFailure("UPSTREAM_FAILURE", "Retry loop ended");
}
 
const limitedHandler = (input: ExtractInput) =>
  limiter.schedule(() =>
    withRetry(() => extractWithContext(input))
  );

The extraction adapter should classify upstream responses before withRetry receives them. Map HTTP 429, 502, 503, 504, and network timeouts to retryable ToolFailure instances. Map explicit blocking responses to BLOCKED, missing JavaScript rendering to RENDERER_UNAVAILABLE, and failed Zod parsing to INVALID_SCHEMA.

The MCP boundary should serialize final failures instead of returning null or an empty object.

try {
  return { ok: true, data: await limitedHandler(input) };
} catch (error) {
  const failure = error as ToolFailure;
 
  return {
    ok: false,
    error: {
      code: failure.code,
      message: failure.message,
      retryable: failure.retryable
    }
  };
}

Agents can then change the URL, request a different extraction mode, or report a specific failure without treating missing data as a valid result.

Preserving source URLs and citation metadata

Provenance belongs inside every extracted record because downstream agents may separate records from the original response. A top-level source URL can become ambiguous when a crawl returns several pages or refreshes only part of a dataset.

import { z } from "zod"
 
const Provenance = z.object({
  sourceUrl: z.string().url(),
  retrievedAt: z.string().datetime(),
  extractionMethod: z.enum([
    "managed-structured-v1",
    "managed-markdown-v1"
  ]),
  contentHash: z.string().regex(/^[a-f0-9]{64}$/)
})
 
const ExtractedRecord = z.object({
  title: z.string().min(1),
  summary: z.string().min(1),
  provenance: Provenance
})
 
const records = z.array(ExtractedRecord)
const ExtractionResult = z.object({ records })

Each field answers a different verification question. sourceUrl identifies the resolved page after redirects. retrievedAt shows how current the record was when extraction occurred. extractionMethod identifies the rendering and parsing path used. contentHash records a SHA-256 hash of the normalized source content, which helps you detect duplicate results or confirm that cited material has changed.

The retrieval handler should create these fields rather than asking the agent or model to generate them. Record the timestamp when Context.dev returns the page, use the final resolved URL, and calculate the hash after applying consistent whitespace and encoding normalization. Model-generated provenance can contain plausible URLs or timestamps that never appeared during retrieval.

Schema validation should cover provenance and extracted content in the same operation.

const result = ExtractionResult.safeParse(candidate)
 
if (!result.success) {
  throw new Error("Extraction output failed schema validation")
}
 
return result.data

A failed provenance check should reject the record instead of returning uncited content. When one record combines evidence from multiple pages, attach an array of provenance objects or split the output into source-specific records. Never assign one URL to claims assembled from several sources.

Observability and security boundaries

Production observability should connect each agent request to its extraction result. Assign a trace ID when the MCP server receives a tool call, and pass that ID through rendering, extraction, validation, and response delivery. Record the tool name, normalized domain, extraction method, latency, retry count, response status, and schema version. Log blocked-request events and validation failures as explicit error types rather than empty results.

Audit logs should preserve debugging evidence without storing sensitive page content by default. Record a content hash, result size, and provenance metadata instead of the full response. Redact authorization headers, cookies, query parameters, and extracted personal data. Access controls and retention limits should apply to logs because tool calls may reveal internal research targets.

URL controls should prevent an agent from turning the scraping tool into an arbitrary network client. Allowlist approved domains when the use case permits it. Otherwise, require HTTPS, restrict ports, resolve hostnames before connecting, and reject loopback, private, link-local, and cloud metadata addresses. The handler should repeat those checks after every redirect to prevent SSRF-style abuse through attacker-controlled URLs or DNS changes.

Browser actions need a narrower schema than arbitrary JavaScript execution. Define an enum of permitted actions such as click, wait, and scroll, then cap action counts and execution time. Reject file access, downloads, uploads, clipboard access, custom scripts, and navigation outside the approved domain. Keep session credentials outside agent-controlled parameters, and isolate each request so one tool call cannot read another call’s cookies or storage.

Context.dev can manage rendering and extraction infrastructure, but your MCP server still owns authorization and tool policy. Apply domain rules and action limits before calling Context.dev, then validate the returned schema before releasing data to the agent. Track latency, blocked requests, and validation failures by domain so operational problems remain visible even when the retrieval layer is managed.

Evaluating extraction quality before deployment

A repeatable evaluation harness should test labeled URLs that represent production traffic by domain, page template, rendering mode, and update cadence. Start with 50 to 100 URLs, store the expected output and source snapshot for each one, and run the suite before every release. Run it on a schedule to catch site changes and vendor regressions.

Score each run with operational metrics.

• Extraction accuracy measures correct extracted fields divided by expected fields. Normalize dates, prices, and whitespace before comparison. Review free-form text manually or with a documented semantic scoring rule. Weight fields that drive downstream actions more heavily.
• Schema validity rate measures responses that pass the complete output schema divided by completed tool calls. Missing required provenance should fail validation even when the extracted content looks correct.
• Freshness measures how long the pipeline takes to capture a source change. Scheduled pipelines should capture at least 95 percent of known changes within one scheduled run interval.
• Latency measures elapsed time between the MCP tool call and validated output. Track p50 for typical performance and p95 for slow requests. The p95 must remain below the agent runtime’s timeout budget.
• Blocked-request rate measures requests identified as blocked divided by initial retrieval attempts. Count challenges and access-denied pages as blocked responses. Retries should remain visible in this metric.
• Cost per successful result divides API fees and retry costs by outputs that pass both accuracy and schema checks. Failed or unusable responses still contribute to cost.

Reasonable starting gates are at least 97 percent field accuracy, 99.5 percent schema validity, and no critical-field errors. Keep blocked requests below 2 percent, and require p95 latency to stay at least 20 percent under the client timeout. Set the cost ceiling from your application economics rather than the cheapest observed test run.

Report every metric by domain and page template as well as across the full suite. An acceptable aggregate can hide a broken product-page template or one domain that blocks most requests. Preserve failed payloads, error types, latency traces, and Context.dev request identifiers so you can distinguish extraction errors from retrieval failures.

Production deployment should require every hard gate to pass across several scheduled runs. Treat new domains and templates as unevaluated until labeled examples enter the harness. Gradual rollout then confirms that the test set reflects real agent traffic.

Comparing architectures: managed API versus browser control versus self-hosted infrastructure

Context.dev is the better fit when your MCP tool needs structured web data or clean Markdown without operating scraping infrastructure. Playwright MCP, Puppeteer MCP, and Skyvern fit workflows where the agent must control a browser, complete login steps, or maintain a session. Firecrawl, Apify, and Bright Data each offer stronger specialization in particular parts of the scraping stack.

A managed extraction API reduces the operational surface of an MCP server. Context.dev handles page rendering, crawling, and structured output behind one API, so your tool handler can focus on input validation, provenance, and application logic. You do not need to run browser pools or integrate separate services for proxies and CAPTCHA handling. Context.dev works best when the browser serves as a means of retrieving data rather than the environment where the task occurs.

Firecrawl is a strong option for Markdown-oriented retrieval and RAG pipelines. Its mature MCP presence makes it easier to connect agents to search, scraping, and crawling workflows. If your pipeline primarily converts pages into clean text for indexing, Firecrawl may fit the output model directly. Context.dev becomes more attractive when you want one managed interface for rendering, crawling, and schema-shaped extraction.

Apify suits projects that benefit from its broad Actor marketplace. An existing Actor may already support a particular website or extraction task, which can shorten implementation time. Actor selection and configuration add another architectural layer, however. Context.dev offers a simpler fit when you prefer one extraction contract across many sites instead of choosing and operating task-specific Actors.

Bright Data suits companies that need enterprise scraping infrastructure and extensive control over collection components. Its product range can support demanding proxy and data-acquisition programs. That breadth may require more vendor-specific configuration than an agent-facing extraction tool needs. Context.dev favors faster deployment and lower infrastructure overhead when your main requirement is normalized web data for an LLM pipeline.

Browser-control tools remain the right choice when interaction defines the task. Playwright MCP and Puppeteer MCP let an agent click elements, fill forms, inspect page state, and manage navigation directly. Skyvern similarly targets browser workflows driven through user interfaces. These approaches support login flows and multi-step transactions, but you assume more responsibility for browser security, session handling, resource limits, and recovery from changing page behavior.

Self-hosted browser infrastructure gives you the most control over execution and data residency. You can tune browser versions, network routing, and session storage for a specific workload. You also own capacity planning, proxy integration, blocking diagnostics, and browser crashes. Self-hosting makes sense when those controls justify the engineering cost or when policy prevents a managed service.

Context.dev includes browser actions, but we position it primarily as a web-data extraction layer. Choose it when the desired result is sourced JSON or Markdown. Choose direct browser control when the agent must operate the site itself or preserve a long-lived authenticated session.

Comparison table: MCP scraping architectures at a glance

Architecture	Rendering	Primary output	Infrastructure burden	Best-fit use case
Context.dev	Managed JavaScript rendering	Structured JSON or clean Markdown	Low	AI agents that need sourced web data through one API
Firecrawl	Managed or self-hosted rendering	Markdown and structured extraction	Low to medium	RAG pipelines and Markdown-oriented MCP workflows
Bright Data	Managed browsers, proxies, and scraping APIs	HTML, JSON, or browser responses	Medium	Enterprise scraping programs that need extensive proxy infrastructure
Apify	Managed Actors and browser runtimes	Actor-specific datasets	Medium	Workflows served by its broad Actor marketplace
Playwright MCP	Direct browser control	Page content and interaction results	High	Login flows, testing, and multi-step browser interaction
Puppeteer MCP	Direct Chromium control	Page content and interaction results	High	Custom Chromium automation with precise browser control
Skyvern	Browser-based task automation	Task results and extracted data	Medium	Visual workflows that require navigation and form interaction

FAQ

How do MCP tool schemas differ from REST endpoints?

An MCP schema helps an agent discover a tool, understand its purpose, and construct valid arguments at runtime. A REST endpoint usually assumes that a developer has already programmed the caller. Keep MCP tools narrow, describe each field clearly, reject unknown parameters, and validate both input and output.

How should an MCP server handle pagination and large crawls?

The server should own crawl state rather than asking the agent to manage page tokens or URL queues. Return a cursor, enforce page and depth limits, deduplicate URLs, and process large jobs asynchronously. A managed layer such as Context.dev can handle rendering and crawling without requiring you to maintain browser pools, proxy rotation, or scheduling infrastructure.

How can I keep extracted data fresh?

Store the retrieval timestamp with every record and set refresh intervals according to how often each source changes. Scheduled recrawls work for predictable updates. Context.dev Monitors can watch a page, sitemap, or site and detect changes when fixed schedules would create unnecessary requests. Revalidate changed records against the same output schema before replacing stored data.

When should I choose browser automation over a managed extraction API?

Choose Playwright MCP, Puppeteer MCP, or another browser-control tool when your workflow requires persistent login sessions, multi-step forms, direct UI interaction, or precise browser state. Choose a managed extraction API when you need rendered page data, clean Markdown, or structured JSON without maintaining browsers and site adapters. Context.dev primarily serves extraction workflows, although its actions parameter supports browser actions.

How should cost per successful result affect vendor choice?

Calculate total vendor and infrastructure spend divided by the number of valid, usable results. Count blocked requests, schema failures, retries, and empty responses as costs rather than successes. Include engineering time when comparing a managed API with self-hosted browsers. A lower request price can produce a higher cost per successful result when failures require repeated calls or manual maintenance.

Conclusion

Reliable agent-facing scraping depends on the controls around retrieval. Narrow schemas constrain tool calls, validation keeps malformed records out of downstream systems, and provenance lets users verify each result. Scheduled evaluations reveal whether the pipeline remains accurate, fresh, and cost-effective as source sites change.

Your workload should determine the architecture. Context.dev fits structured extraction and LLM-ready output when you want managed rendering, crawling, and normalization through one API. Browser-control tools fit login flows, persistent sessions, and multi-step interaction.

Instrument the evaluation rubric before choosing an extraction layer. Measure accuracy, schema validity, freshness, latency, blocked requests, and cost per successful result against representative URLs. Then select the architecture that meets those measured requirements.