The fastest way to give an agent web search is one Context.dev Search call with highlightsOptions enabled, which returns ranked URLs plus only the passages that match the query, from one API key. When the agent needs a specific known URL or full page text, the exact same API key also handles single-page Highlights, full Markdown scraping, and multi-page Answers.
Tools used in this guide
- Context.dev Search: Ranked results with relevance (
high,medium,low), plus optional Markdown or highlights per result. - Context.dev Highlights: Passages relevant to a question, returned in page order.
- Your agent framework or plain function calling: Works with OpenAI, Anthropic, or MCP tool definitions.
What should a web search API for AI agents return?
Context.dev Search is purpose-built for AI agents: every result includes a high, medium, or low relevance label, deterministic read status codes (SUCCESS, TIMEOUT, WEBSITE_BLOCKED), domain and freshness filters, and optional highlightsOptions that return only the passages matching the agent's query. An agent-ready web search API must return more than raw links so the agent can evaluate evidence directly without performing blind, expensive scraping passes across the live web.
Autonomous AI agents that browse and research the live web face a severe structural bottleneck. Full web pages routinely contain thousands of tokens of navigation menus, boilerplate, and unrelated text that saturate the model's context window, increase latency, and dilute reasoning accuracy. Giving an agent a separate search API and a separate scraper forces developers to manage multiple keys, stitch together ranking and extraction pipelines, and push entire page payloads into the prompt.
Academic research throughout 2026 highlights the precise impact of this context bloat. According to Kerboua et al. (2026) in their research on trimming the large context of web agents at INSA Lyon, Mila, and ServiceNow Research, web pages processed by LLM agents often exceed tens of thousands of tokens. This saturates context limits, increases computational cost, and exposes agents to prompt injection. Their experiments on the WorkArena and WebArena benchmarks demonstrated that extracting only task-relevant lines from web observations reduces observation size by over 50 percent while maintaining strong task success rates and reducing vulnerability to banner and pop-up prompt-injection attacks.
Furthermore, Li et al. (2026) from Tongji University and Stanford University found that real-world web observation spaces contain over 90 percent structural noise, such as navigation Chrome, ads, and boilerplate. By pruning environmental noise and keeping only sparse reasoning anchors, their framework improved Gemini-3.0-flash task success on WebArena from 36.4 percent to 41.2 percent while cutting token consumption by 8.8 percent. It also improved DeepSearch accuracy from 53.9 percent to 57.1 percent with an 8x token reduction.
Kolukuluru et al. (2026) added in their EMNLP research that long-horizon research agents accumulate context rapidly while the marginal value of additional raw text declines. Their stage-aware analysis found that early pruning at the retrieval stage yields the largest end-to-end savings, reducing token usage by up to 73 percent while preserving judged report quality and source grounding. Finally, Zhang et al. (2026) observed that each turn in an agentic search loop appends retrieved snippets, page contents, and tool outputs until the trajectory reaches tens of thousands of tokens, which severely degrades model attention if noisy observations are not filtered at retrieval time.
This is why Context.dev returns ranked URLs, titles, and descriptions alongside explicit relevance scores. Setting highlightsOptions.enabled: true returns query-relevant passages natively in the same API call, and per-result failure codes ensure the agent retains basic URL metadata even if a page read fails due to anti-bot protections or timeouts.
Step 1: search the web
To begin building your AI search capability, you will initialize the Context.dev client and execute a basic query. The standard workflow starts with installing the context.dev package via npm and utilizing the CONTEXT_DEV_API_KEY environment variable.
The API accepts POST requests at the https://api.context.dev/v1/web/search endpoint. When configuring your request, you control the result count via the numResults integer, which accepts values ranging from 1 to 100 and defaults to 10.
Here is how developers can search the web using the native TypeScript SDK:
import ContextDev from "context.dev";
const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });
const search = await client.web.search({
query: "Stripe API authentication",
numResults: 10
});
console.log(search.results);Alternatively, you can make the exact same request using cURL if you are integrating Context.dev into a different backend environment or a raw HTTP client:
curl https://api.context.dev/v1/web/search \
-H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "Stripe API authentication", "numResults": 10}'Step 2: return only the relevant passages
To send an LLM only the relevant passages from a web page, enable Context.dev Highlights via highlightsOptions.enabled on POST /v1/web/search or formats.highlights on POST /v1/web/scrape and set maxCharacters (100 to 20,000, default 4,000) to fit your context budget. Context.dev returns matching Markdown passages in page order with bracketed headings and source URLs intact.
When you set highlightsOptions.enabled to true in your search call, Highlights use the search query and read each resulting page according to your markdownOptions settings. This allows you to request focused, extracted text without needing to enable full Markdown extraction for the entire document.
The system strictly respects a character budget defined by highlightsOptions.maxCharacters. The default limit is 4,000 characters, with a valid range of 100 to 20,000 combined characters per result. You can also control the cache freshness of the underlying page read via markdownOptions.maxAgeMs. This value defaults to 180 days for full Markdown, or 365 days when only highlights are requested. It accepts up to 365 days (31,536,000,000 milliseconds), and setting a value of 0 forces a completely fresh scrape.
According to first-party metrics published in September 2026, Context.dev Highlights massively improve token efficiency and reasoning accuracy. On the adapted WebCode highlights benchmark (which measures whether returned text supports every substantive fact in the reference answer across 250 URL-and-question pairs), Context.dev Highlights achieved 76.0 percent groundedness using an average of only 229 output tokens. For comparison, a native full-page open baseline averaged 6,889 output tokens while only achieving 57.6 percent groundedness. On the SimpleQA Verified benchmark, Context.dev Highlights achieved 76.7 percent answer accuracy with an average of just 183 output tokens.
Production deployments mirror these savings. The coding agent platform freecode recorded a 9.8 percent reduction in input tokens across replays of real coding sessions, alongside an 8.9 percent lower total LLM and retrieval cost. The AI search engine Scira AI took under 10 minutes to add, test, and deploy live web search with Context.dev. According to Aarav Gupta, Founder and CEO of Spikonado, Context.dev allows them to equip agents with up-to-date context on hardware and frameworks, making them massively more reliable.
Here is an example request to return relevant result content:
{
"query": "Stripe API authentication",
"highlightsOptions": {
"enabled": true,
"maxCharacters": 1500
}
}And the corresponding response shape:
{
"results": [
{
"url": "<string>",
"title": "<string>",
"description": "<string>",
"relevance": "high",
"markdown": {
"markdown": "<string>",
"code": "NOT_REQUESTED",
"finalDOMState": "loaded"
},
"highlights": {
"highlights": [
"<string>"
],
"code": "SUCCESS"
}
}
],
"query": "<string>",
"cache_metadata": {
"status": "hit",
"age_ms": 1
},
"request_id": "3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91",
"partial": true,
"key_metadata": {
"credits_consumed": 123,
"credits_remaining": 123
}
}Step 3: get highlights from one page you already know
Context.dev also allows you to extract precise information when the agent already holds a target URL. You can send a direct request to POST https://api.context.dev/v1/web/scrape (or use client.web.scrape in the SDK) and enable formats.highlights: true. You must supply highlightsParams.query, which accepts between 1 and 2,000 characters. You can also optionally provide highlightsParams.maxCharacters to strictly limit the token output, which defaults to 4,000 and ranges from 100 to 20,000 characters across the returned passages.
The response will contain the relevant passages in page order within highlights.data, retaining all useful Markdown formatting. The API automatically inserts bracketed headings (for example, [Example Domain]) to add structural context to each isolated passage, and ellipses mark where unrelated text was omitted.
A major advantage of this single-page endpoint is that it natively supports YouTube transcripts. A YouTube video URL can use the exact same JSON request shape to find relevant passages within spoken audio text, giving the agent video understanding without transcribing massive media files manually.
The system enforces an input processing limit where Highlights read the first 400,000 characters of page Markdown after content filters are applied. A page without text will return an empty array. Developers must always verify highlights.success before instructing the agent to read highlights.data to ensure highlights from a known URL are completely valid.
Request and abridged response example:
{
"url": "https://example.com",
"formats": {
"highlights": true,
"markdown": true
},
"highlightsParams": {
"query": "What is this domain used for?",
"maxCharacters": 1500
}
}{
"highlights": {
"requested": true,
"success": true,
"data": [
"[Example Domain] This domain is for use in illustrative examples in documents."
]
}
}Step 4: handle each result's outcome
Context.dev requires developers to intentionally check each search result's highlights.code and markdown.code before blindly passing content to the model. An agent-ready platform must gracefully handle the unpredictability of the live web, and Context.dev provides deterministic status codes to simplify this logic.
The API returns explicit status codes for every result:
| Code | Meaning |
|---|---|
SUCCESS | Page content is available. |
NOT_REQUESTED | The output was not enabled. |
TIMEOUT | The page exceeded its deadline. |
CONTENT_TOO_LARGE | The page exceeded the content limit. |
WEBSITE_ACCESS_ERROR | The page could not be accessed. |
WEBSITE_BLOCKED | The site served an anti-bot challenge or access wall. |
ERROR | Another extraction failure. |
Critical resilience rules apply for agents operating autonomously. A search result remains incredibly useful even when its page content cannot be read. You must keep the url, title, and description intact when a read fails so the agent still knows the candidate exists and can log the explicit failure state without hallucinating page contents.
Furthermore, highlights can return code: "SUCCESS" with an empty array ([]) when the page was read successfully but no passage on the page was relevant to the query. Finally, markdown.finalDOMState reports `