TL;DR
- Call the scrape tool when an agent needs current evidence from a specific page, rather than scraping every URL it encounters.
- Validate the URL against your allowed domains and reject internal destinations before sending a request.
- Retrieve rendered Markdown and schema-shaped JSON in one
client.web.scrapecall, and setmax_age_msso a cached copy cannot pass as a live one. - Return validated, schema-shaped data with the source URL and retrieval time so the agent can cite the page behind each claim.
- Treat missing, blocked, stale, or irrelevant pages as explicit failures. The agent should seek another permitted source or say it cannot verify the answer, rather than guess.
When an agent should call the scrape tool
An agent should scrape when a user's question depends on information that may have changed since the agent last checked. Current pricing, product availability, and a recently updated policy are examples. If the agent already has a relevant page retrieved within your application's freshness window, it can use that evidence instead. If it has no suitable URL, a search step should identify a source before the scrape tool runs.
Your application should decide which sources the agent may access. A support agent might scrape public pages on your documentation domain but reject a URL supplied in a prompt that points elsewhere. Search results do not grant permission either. The application must check each selected URL against its domain policy before passing it to a web scraping API.
A successful retrieval does not automatically answer the question. The agent should check whether the page covers the requested topic and whether its retrieval time meets the question's freshness needs. A page about last year's prices may load correctly but provide no evidence for today's price. In that case, the agent needs a newer permitted source or must say it could not verify the answer.
Designing the tool call contract
Give the agent a small, stable tool interface that stays separate from any scraping SDK. The request needs a url. An optional extraction field can name one of the schemas your application defines, such as product_price. Settings like the cache window and the page deadline belong to your application, not the model, so the model cannot invent arbitrary retrieval settings.
The response should carry source_url, status, and retrieved_at, plus the page content as Markdown, a structured object, or both. The tool should keep the page URL attached to the content either way. Record the final page URL after permitted redirects so the agent can cite the page it actually received. Use an explicit status such as ok, not_found, blocked, timeout, or empty. For a failed request, return no extracted facts rather than plausible-looking placeholders.
For example, a request might specify url as https://webscraper.io/test-sites/e-commerce/static/product/31 and extraction as product_price. A successful response would pair each extracted value with source_url and a UTC retrieved_at timestamp. If the page contains no price, the tool should report the missing field. The agent cannot treat the extraction request itself as evidence that a price exists.
Your application must validate URLs before it issues a tool call. Restrict permitted protocols, match hostnames against an allowlist, and reject loopback, private, and link-local destinations. Context.dev makes the request from its own infrastructure rather than yours, and the scrape endpoint takes public HTTP and HTTPS URLs only. In our tests, loopback and cloud metadata addresses came back as failed outputs instead of content. Keep the checks anyway wherever your own code fetches a URL, such as a fallback HTTP client or a link preview step. Redirects and DNS changes can turn an apparently public URL into an internal address, a known server-side request forgery risk. The allowlist is also your job: a managed retrieval API does not decide which destinations your agent is authorized to request.
Handling missing pages, blocks, and stale content
Your agent should classify each scrape result before using it as evidence. In the tool response, keep the source URL, retrieval time, and a status that distinguishes retrieval failures from content the agent cannot use. A successful request alone does not establish that the page answers the user's question.
- Removed page or 404. The agent should stop retrying that URL. It can check another permitted source or tell the user that the requested page is unavailable.
- Blocked request. The agent should report that it could not access the target and try a different permitted source if one exists. A block does not give the agent evidence about the page's contents.
- Timeout. The agent can retry a limited number of times. If the request still times out, it should report the retrieval failure rather than fill in the missing answer.
- Empty or low-content render. The agent can make one rendered retrieval attempt if the first response lacks usable page text. If the rendered result remains empty, it should seek another source or report the limitation.
- Irrelevant page. The agent should compare the retrieved text with the user's question. If a page does not support the requested claim, it should search for a more relevant permitted URL instead of citing that page.
A recent retrieval can still return old information. Caching is one way that happens: Context.dev reuses a cached result younger than max_age_ms, which defaults to three days, so a response can be a cache hit even though your application received it a moment ago. Set max_age_ms=0 to fetch live, or derive the real fetch time from the response, as the example below does. The agent should also distinguish when it fetched a page from when the page says its information was updated. If the user needs current facts and the page provides no adequate date or supporting detail, the agent should qualify its answer or find a better source.
Implementing the scrape call with Context.dev
Context.dev provides managed scraping and rendering through one API, so your agent can retrieve a page without maintaining its own browser infrastructure. The Python SDK exposes it as client.web.scrape, which loads the page in a browser and returns the formats you request. Install it along with Pydantic, which the next section uses:
pip install context.dev pydanticimport os
from datetime import datetime, timedelta, timezone
from urllib.parse import urlparse
from context.dev import ContextDev
client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])
ALLOWED_DOMAINS = {"webscraper.io"}
STATUS_BY_ERROR_CODE = {
"NOT_FOUND": "not_found",
"WEBSITE_NOT_FOUND": "not_found",
"WEBSITE_BLOCKED": "blocked",
"TIMEOUT": "timeout",
"REQUEST_TIMEOUT": "timeout",
}
PRODUCT_PRICE_SCHEMA = {
"type": "object",
"properties": {
"product_name": {"type": ["string", "null"]},
"price_text": {"type": ["string", "null"]},
},
"required": ["product_name", "price_text"],
}
def is_allowed(url: str) -> bool:
parsed = urlparse(url)
host = parsed.hostname or ""
return parsed.scheme in ("http", "https") and any(
host == domain or host.endswith(f".{domain}") for domain in ALLOWED_DOMAINS
)
def scrape_source(url: str, schema: dict, max_age_ms: int = 0) -> dict:
if not is_allowed(url):
return {"status": "not_allowed", "source_url": url}
page = client.web.scrape(
url=url,
formats={"markdown": True, "json": True},
json_params={
"schema": schema,
"instructions": "Copy values exactly as written on the page. Use null when a value is absent.",
},
max_age_ms=max_age_ms,
timeout_opts={"milliseconds": 20_000, "behavior": "return-partial"},
)
if not is_allowed(page.url):
return {"status": "not_allowed", "source_url": page.url}
failed = next((output for output in (page.markdown, page.json_) if not output.success), None)
if failed:
status = STATUS_BY_ERROR_CODE.get(failed.error_code, "failed")
elif not page.markdown.data.strip():
status = "empty"
else:
status = "ok"
result = {"status": status, "source_url": page.url}
if status == "ok":
result["retrieved_at"] = datetime.now(timezone.utc) - timedelta(milliseconds=page.cache_metadata.age_ms)
result["page_modified_time"] = page.metadata.modified_time
result["markdown"] = page.markdown.data
result["extracted"] = page.json_.data
return resultReplace ALLOWED_DOMAINS with the domains your agent may read. The example uses webscraper.io, a practice site built for scraping, so it runs as written. The function does the checking that keeps a bad page from becoming evidence:
- Policy first. The URL is checked before any request, and the final URL is checked again after redirects, because a redirect can leave your allowed domains.
- Per-output results.
formatstakes independent flags, so one call returns Markdown and JSON together, and each output carries its ownsuccessanderror_code. A missing page does not raise an exception. When we requested a nonexistent page on October 27, 2026,markdown.successwasFalsewith theNOT_FOUNDcode. Codes the table does not list map tofailed. - Deadline. With
return-partial, the API returns the outputs that finished before the 20-second deadline and reports the rest as failed, so a timeout reaches your code as an error code instead of an exception. - Honest timestamp.
cache_metadata.age_msis the age of the copy that was served, and0for a live fetch. Subtracting it from the current time gives the fetch time, to within the request's own latency. The defaultmax_age_ms=0forces a live fetch. Raise it to your freshness window for pages where a cached copy is fine. - Page date kept apart.
page_modified_timeis what the page says about itself. It isNoneunless the page declares a modified time, and it stays separate fromretrieved_at. - No placeholders. On any failure the function returns only the status and URL.
SDK exceptions for authentication, credits, rate limits, and network errors still propagate. Treat them as problems in your own infrastructure, not as evidence about the page.
Returning schema-shaped extraction with source attribution
The agent should receive validated evidence with its source attached, rather than an unlabelled string of scraped text. Keep the source URL and retrieval timestamp in the same object as the extracted fields so later tool calls cannot separate a claim from the page used to support it.
scrape_source already asked for JSON that matches PRODUCT_PRICE_SCHEMA. The schema marks both fields nullable, and the instructions tell the extraction model to copy values exactly as written and return null for anything absent. The model reading the page can still get a value wrong, so the code below validates the result before the agent sees it:
from datetime import datetime
from pydantic import AnyHttpUrl, BaseModel, ConfigDict, field_validator
class ProductEvidence(BaseModel):
model_config = ConfigDict(extra="forbid")
product_name: str
price_text: str
source_url: AnyHttpUrl
retrieved_at: datetime
@field_validator("product_name", "price_text")
@classmethod
def reject_blank(cls, value: str) -> str:
if not value.strip():
raise ValueError("Required evidence is blank")
return value
def to_evidence(result: dict) -> ProductEvidence:
if result["status"] != "ok":
raise ValueError(f"Cannot cite this page: {result['status']}")
extracted = result["extracted"]
evidence = ProductEvidence(
product_name=extracted.get("product_name"),
price_text=extracted.get("price_text"),
source_url=result["source_url"],
retrieved_at=result["retrieved_at"],
)
for value in (evidence.product_name, evidence.price_text):
if value not in result["markdown"]:
raise ValueError(f"{value!r} does not appear in the retrieved page")
return evidence
result = scrape_source(
"https://webscraper.io/test-sites/e-commerce/static/product/31",
PRODUCT_PRICE_SCHEMA,
)
print(to_evidence(result))When we ran this on October 27, 2026, it printed the product name Packard 255 G2, the price $416.99, the page URL, and a UTC timestamp. Pydantic rejects missing or blank required values instead of passing a partial product record to the LLM. to_evidence goes one step further: each extracted value must appear verbatim in the Markdown the tool returned, so a value the page does not contain becomes a failed extraction rather than a citation. The source URL comes from the response's url field, which is the final URL after redirects, and scrape_source has already checked it against your allowlist. If a response does not let you establish which page supplied the content, fail the extraction rather than attach the requested URL as an unverified citation.
Writing a grounded agent response
A grounded answer should contain only claims supported by the validated extraction, with the retrieved URL beside each claim. For example, if the extraction above succeeds, the agent could say, "The Packard 255 G2 is listed at $416.99 (source page, retrieved October 27, 2026). The page does not say whether tax or shipping is included, so I can't confirm the final price."
The agent should produce that answer only after checking the extraction. Your validation step should require a source URL and the fields needed to answer the user's question. Keep the retrieval timestamp with those fields so the agent can decide whether the evidence is recent enough. If a required field is missing, the agent should seek another permitted source or say it cannot verify the answer. Details the question does not depend on, such as shipping terms, can stay unanswered without blocking a supported claim, but the agent should say so rather than fill them in.
When a managed scrape call isn't enough
A managed scrape call fits an agent that needs evidence from a public page without maintaining its own retrieval and rendering infrastructure. Context.dev can handle that retrieval through one API, but your application still decides which URLs the agent may access and whether the returned content supports an answer.
Pages behind a login, or behind a short sequence of clicks, are in reach on paid plans. Browser profiles let later requests reuse a saved sign-in, and browser actions can click, scroll, or wait before capture, up to five steps per request. What a scrape call does not replace is a browser your agent drives step by step, where the next action depends on what the page just showed. Sites that require MFA, CAPTCHA, passkeys, or a popup login are unsupported by profiles, and a profile serves one request at a time, so multi-account workflows need one profile per account and their own scheduling. For those tasks, keep a controllable browser.
On-demand scraping also differs from collecting data across many sites on a schedule. For that design, see production web scraping pipelines in Python. If you are choosing how to expose retrieval to the agent, see MCP server vs REST API for web scraping. For a comparison of on-demand tool options, see best real-time web scraping tools for AI agents.
FAQ
How does this differ from a scheduled scraping pipeline? An agent calls the tool when a question needs current evidence from a permitted URL. A scheduled pipeline collects pages on a timetable, whether or not a user has asked about them.
When should the agent retry instead of failing? Retry a timeout or temporary block within a fixed limit. Treat a removed page as unavailable, and seek another permitted source when the retrieved page does not answer the question. The failure-handling contract should tell the agent which outcome occurred.
How do I keep a cached copy from passing as live evidence? Context.dev reuses a cached result younger than max_age_ms, which defaults to three days. Pass max_age_ms=0 when the question needs the page as it stands now, and compute retrieved_at from cache_metadata.age_ms so the timestamp reflects when the content was fetched, not when your code received it.
Can one call return both Markdown and structured JSON? Yes. Set formats={"markdown": True, "json": True} and pass a JSON Schema in json_params. Each output reports its own success, so check both: Markdown can succeed while JSON extraction fails. In the Python SDK, the JSON output is page.json_, with a trailing underscore.
How does retaining the source URL prevent invented citations? URL retention gives the agent a specific page to cite, but the URL alone cannot prove a claim. Your response layer should cite that URL only for claims supported by the validated extraction attached to it.
Conclusion
A source-aware scraping tool needs a clear request and response contract, plus explicit behavior when retrieval fails or the page lacks evidence. For developers adding on-demand retrieval to an agent loop, those checks let the agent cite what it found and say what it could not verify. Scheduled crawl infrastructure serves a different job.