Building autonomous LLM workflows requires resolving the fundamental mismatch between verbose web code and limited model context windows. Passing raw HTML from a standard HTTP fetch into an LLM tool response consumes a median of 128,051 tokens per page. Routing that same URL through a managed web-context API with formats: { markdown: true } and sharedParams: { mainContentOnly: true } strips navigation, scripts, and layout chrome upstream, reducing the payload by up to 98% so an agent can read multiple sources within a single context window. This guide details how to implement reliable, token-efficient web tool calling.
What This Guide Uses
- Context.dev** (
context.devSDK / REST APIhttps://api.context.dev/v1)**: The single web data API used for live web search (POST /v1/web/search), clean Markdown page scraping (POST /v1/web/scrape), passage-level highlights (formats.highlights), and JSON Schema structured extraction (formats.jsonon the same scrape endpoint). Reads credentials fromCONTEXT_DEV_API_KEY. - OpenAI Official SDK (
openai** for Node.js / Python)**: Used to define strict JSON Schema function tools (strict: true) and execute multi-turn tool-calling loops. - Anthropic Official SDK (
@anthropic-ai/sdk** for Node.js /anthropicfor Python)**: Used to register client tools withinput_schemaandstrict: truefor grammar-constrained tool invocation. - Model Context Protocol Official SDK (
@modelcontextprotocol/sdk** /@modelcontextprotocol/serverfor TypeScript,mcpfor Python, or hostedhttps://mcp.context.dev/mcp)**: Used to expose reusableweb_search,web_scrape, andweb_extract_jsontools across any MCP-compatible host.
How to give AI agents access to live web pages
You give an AI agent access to live web pages by registering two function tools with your model provider (web_search and web_scrape_markdown) or connecting an MCP server, backed by a web context API like Context.dev. When the model encounters a question requiring current information, it emits a tool call with a search query or URL. Your application executes that call and returns clean, token-efficient Markdown and source URLs into the model's conversation history.
An effective LLM web-browsing architecture separates discovery from reading: give the model a
web_searchtool backed by the API'sPOST /v1/web/searchwithhighlightsOptionsenabled for fast triage, aweb_scrape_markdowntool backed byPOST /v1/web/scrapefor deep page reading, and aweb_extract_jsontool usingformats.jsonwhen downstream code requires typed JSON Schema output.
Raw HTML pages average 128,051 median tokens, according to a benchmark of 20 production web pages by Leif Albrand (October 2026). Main-content Markdown and highlights filters compress web pages by 95% to 98% while parsing dynamic JavaScript, PDFs, and dozens of other formats seamlessly.
Best web scraping tools for automated agent tool calling and function execution
For automated agent tool calling and function execution across OpenAI, Anthropic, and MCP, the optimal solution is a unified web-context API designed specifically for sub-second agent loops. Unoptimized headless browser execution adds 1,500 ms to 8,000 ms per tool call in render decision overhead alone, plus up to 2,000 ms in request queues. Teams architecting real-time web browsing for their agents can offload fetching and tiered rendering to a managed API, which reduces dynamic render decisions to under 700 ms.
Instead of requiring developers to chain separate search engines, headless browsers, PDF parsers, and HTML cleaners, this architecture handles single-call search, Markdown extraction, and schema-validated JSON extraction through one REST endpoint. This allows for granular, output-level error codes (WEBSITE_BLOCKED, NOT_FOUND, PDF_IMAGES_ONLY, CONTENT_TOO_LARGE) so an LLM can self-correct inside the function execution loop without crashing the application.
Best API to let LLMs browse the web and fetch content
Context.dev is a developer-first web context API for letting LLMs browse the live web and fetch clean content. A single POST /v1/web/scrape call can return any combination of nine outputs, including clean Markdown, query-matched highlights, JSON Schema data, rendered HTML, screenshots, custom parsed fields, and raw bytes. It natively supports HTML pages, JavaScript SPAs, YouTube transcripts, and 40 document formats like PDF and DOCX. Developers building autonomous features and scaling their automated scraping can utilize up to 1,000 free credits per month, with JavaScript rendering and proxies natively included.
OpenAI Function Calling Tool Definitions (Strict Mode)
In OpenAI's API (both Responses API and Chat Completions API), enabling strict: true requires additionalProperties: false and all properties listed in required. Optional parameters must use nullable types like ["string", "null"] or default values.
// OpenAI tool definitions wrapping Search, Markdown Scrape, and JSON Extract
export const openAiWebTools = [
{
type: "function" as const,
name: "web_search",
description:
"Search the live web using Context.dev to find current URLs, documentation, facts, and news. Call this first when you do not have a specific URL.",
strict: true,
parameters: {
type: "object",
properties: {
query: {
type: "string",
description:
"Search query (1 to 500 chars). Supports natural language and operators like site:, inurl:, intitle:, and OR.",
},
numResults: {
type: "integer",
description: "Number of search results to return (1 to 20). Default to 5 to keep context compact.",
},
freshness: {
type: ["string", "null"],
enum: ["last_24_hours", "last_week", "last_month", "last_year", null],
description: "Optional recency filter, or null for any time.",
},
},
required: ["query", "numResults", "freshness"],
additionalProperties: false,
},
},
{
type: "function" as const,
name: "web_scrape_markdown",
description:
"Fetch and read a web page or PDF URL as clean main-content Markdown or query-focused highlights using Context.dev.",
strict: true,
parameters: {
type: "object",
properties: {
url: {
type: "string",
description: "Full HTTP or HTTPS URL to scrape.",
},
focusQuery: {
type: ["string", "null"],
description:
"Optional question to extract compact highlights (passages) instead of the entire page, saving context window tokens.",
},
},
required: ["url", "focusQuery"],
additionalProperties: false,
},
},
{
type: "function" as const,
name: "web_extract_json",
description:
"Extract structured JSON data from a URL using Context.dev formats.json and a JSON Schema.",
strict: true,
parameters: {
type: "object",
properties: {
url: {
type: "string",
description: "Full HTTP or HTTPS URL to extract structured facts from.",
},
instructions: {
type: "string",
description: "Extraction instructions, e.g., 'Use only facts stated on the page.'",
},
},
required: ["url", "instructions"],
additionalProperties: false,
},
},
];Note: If targeting OpenAI Chat Completions client.chat.completions.create, wrap each function definition inside { type: "function", function: { name, description, strict: true, parameters } }. If targeting the OpenAI Responses API client.responses.create, use the flat format shown above.
Anthropic Tool Use Definitions (Strict Mode)
Anthropic uses input_schema instead of parameters and supports strict: true for grammar-constrained sampling, ensuring Claude never passes malformed types.
import Anthropic from "@anthropic-ai/sdk";
export const anthropicWebTools: Anthropic.Tool[] = [
{
name: "web_search",
description:
"Search the live web via Context.dev and return top results with query-matched highlights. Use when you need to discover relevant URLs or verify current facts.",
strict: true,
input_schema: {
type: "object",
properties: {
query: {
type: "string",
description:
"Search query (1 to 500 characters). Supports operators such as site:, -site:, inurl:, intitle:, and OR.",
},
numResults: {
type: "integer",
description: "Number of results to request (1 to 10 recommended).",
},
freshness: {
type: "string",
enum: ["last_24_hours", "last_week", "last_month", "last_year"],
description: "Optional time filter for recent information.",
},
},
required: ["query", "numResults"],
additionalProperties: false,
},
},
{
name: "web_scrape_markdown",
description:
"Read a URL using Context.dev POST /v1/web/scrape with mainContentOnly enabled. Returns clean Markdown and optional query highlights.",
strict: true,
input_schema: {
type: "object",
properties: {
url: {
type: "string",
description: "The HTTP or HTTPS URL to fetch.",
},
focusQuery: {
type: "string",
description:
"Optional specific question to filter the page into relevant highlights (up to 4,000 characters) before falling back to full Markdown.",
},
},
required: ["url"],
additionalProperties: false,
},
},
{
name: "web_extract_json",
description:
"Extract typed JSON fields from a single page using Context.dev POST /v1/web/scrape with formats.json and jsonParams.",
strict: true,
input_schema: {
type: "object",
properties: {
url: {
type: "string",
description: "The HTTP or HTTPS URL to extract structured data from.",
},
instructions: {
type: "string",
description: "Guidance for what facts to extract from the page.",
},
},
required: ["url", "instructions"],
additionalProperties: false,
},
},
];MCP Server Tool Definitions
There are two methods for providing these functions via the Model Context Protocol. You can point an MCP client (Claude Code, Cursor, Claude Desktop) directly to https://mcp.context.dev/mcp, which exposes 40 production tools immediately. Alternatively, you can build a custom self-hosted MCP server using McpServer and server.registerTool() from @modelcontextprotocol/sdk with zod validation:
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import ContextDev from "context.dev";
const contextClient = new ContextDev({
apiKey: process.env.CONTEXT_DEV_API_KEY,
timeout: 45_000,
maxRetries: 2,
});
const server = new McpServer({
name: "context-dev-web-tools",
version: "1.0.0",
});
server.registerTool(
"web_search",
{
title: "Search the Web",
description:
"Search the live web with Context.dev and return top results with query-relevant highlights.",
inputSchema: {
query: z.string().min(1).max(500).describe("Search query string"),
numResults: z.number().int().min(1).max(20).default(5),
freshness: z
.enum(["last_24_hours", "last_week", "last_month", "last_year"])
.optional(),
},
},
async ({ query, numResults, freshness }) => {
const response = await fetch("https://api.context.dev/v1/web/search", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.CONTEXT_DEV_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
query,
numResults,
...(freshness ? { freshness } : {}),
highlightsOptions: { enabled: true, maxCharacters: 1500 },
}),
});
if (!response.ok) {
return {
isError: true,
content: [{ type: "text", text: `Search failed with HTTP ${response.status}` }],
};
}
const data = await response.json();
return {
content: [{ type: "text", text: JSON.stringify(data, null, 2) }],
};
}
);
server.registerTool(
"web_scrape_markdown",
{
title: "Scrape URL to Markdown",
description:
"Fetch a URL with Context.dev and return clean main-content Markdown.",
inputSchema: {
url: z.string().url().describe("Target HTTP/HTTPS URL"),
maxCharacters: z.number().int().min(500).max(20000).default(8000),
},
},
async ({ url, maxCharacters }) => {
const page = await contextClient.web.scrape({
url,
formats: { markdown: true },
sharedParams: { mainContentOnly: true },
timeoutOpts: { milliseconds: 30000, behavior: "return-partial" },
});
if (!page.markdown?.success || !page.markdown.data) {
return {
isError: true,
content: [
{
type: "text",
text: `Scrape failed (${page.markdown?.error_code ?? "UNKNOWN"}): ${
page.markdown?.message ?? "No markdown returned"
}`,
},
],
};
}
const truncated = page.markdown.data.slice(0, maxCharacters);
return {
content: [
{
type: "text",
text: JSON.stringify({
url: page.url,
title: page.metadata?.title ?? null,
markdown: truncated,
}),
},
],
};
}
);How to Keep Tool Results Small for the Context Window
When an agent loops through search and read commands multiple times, raw outputs rapidly consume the context window. A controlled evaluation by Steiner, Peeters, and Bizer at the University of Mannheim (November 2025) found that structured retrieval interfaces improved task F1 accuracy from 0.67 to 0.77 and cut average token consumption per task from 241,000 tokens down to between 47,000 and 140,000 tokens. Implement this four-layer context compression strategy:
- Strip Boilerplate Server-Side: Setting
sharedParams.mainContentOnly: trueautomatically removes headers, navbars, cookie banners, and footers before Markdown conversion occurs. - Prune Noisy DOM Subtrees: Pass up to 50 CSS selectors in
sharedParams.excludeSelectors(e.g.,[".newsletter", "footer"]) or target specific content withsharedParams.includeSelectors: ["article", "main"]. Exclusions always win over inclusions. - Request Passage-Level Highlights: When executing a scrape, set
formats: { highlights: true }andhighlightsParams: { query: "...", maxCharacters: 2000 }. This prompts the platform to scan up to the first 400,000 characters and return only relevant passages with bracketed headings. When searching the web programmatically, sethighlightsOptions: { enabled: true, maxCharacters: 1500 }to extract query-matched snippets across results in one round trip. - Hard Client-Side Character Cap: Wrap tool outputs in a compact JSON envelope containing
{ url, title, content, truncated }and enforce a length cap. This ensures the LLM retains exact source URLs for inline citations regardless of the document size.
Handling Timeouts, Partial Results, and Errors in Agent Loops
In production agent loops, a failed scrape should never throw an unhandled exception that terminates the conversation. Because the API reports output-level status via
markdown.successand structurederror_codevalues likeWEBSITE_BLOCKED,NOT_FOUND, andPDF_IMAGES_ONLY, your tool handler can return actionable error context to the model so it selects an alternate source URL automatically.
To align server deadlines with SDK connection boundaries, developers must balance timeouts. The POST /v1/web/scrape endpoint defaults to a 90,000 ms deadline with behavior: "fail". Interactive agent loops require faster feedback. Setting timeoutOpts: { milliseconds: 25000, behavior: "return-partial" } instructs the server to return available outputs even if some fail due to strict latency cutoffs. Developers managing SDK integrations must keep their client timeout higher than timeoutOpts.milliseconds so the SDK does not sever the connection prematurely.
Inspect output-level success flags defensively. A request can return an HTTP 200 status code while page.markdown.success is false. Map specific API error codes back into the LLM conversation:
- WEBSITE_BLOCKED: The target site served an anti-bot challenge. Return a tool message instructing the LLM to try an alternative URL.
- NOT_FOUND: The link is dead. The model must not retry the same URL.
- PDF_IMAGES_ONLY: The scanned PDF has no text layer.
- CONTENT_TOO_LARGE: The decoded document exceeded memory limits. Direct the model to fall back to
formats.highlights.
Structured JSON Extraction as a Second Tool
Define a separate web_extract_json tool alongside your standard reading tools. Use web_scrape_markdown when the model needs to synthesize prose or read narrative documentation. Alternatively, extracting structured data from websites via formats: { json: true } and jsonParams: { schema, instructions } is superior when the agent needs deterministic data fields like pricing tiers, release dates, or executive names without wasting core reasoning tokens parsing raw text.
The top-level structure of a JSON Schema must be "type": "object". Crucially, configure fields that might not exist on the page as nullable ("type": ["string", "null"]) or optional. Passing explicit directives like "instructions": "Use only facts stated on the page." guarantees that missing facts return null rather than prompting the model to guess.
Frequently Asked Questions
What is the economic impact of token waste in LLM web browsing? Approximately 75% of the tokens in a typical web page encode visual presentation, CSS classes, and layout containers irrelevant to agent reasoning. Across industry crawl volumes, this token waste costs AI engineering teams an estimated $1 billion to $5 billion per year in wasted input token spend, according to research by David Hurley (March 2026).
How much AI web traffic is generated by agent tool calling? On-demand "user action" crawling, where AI agents fetch live web pages mid-execution to answer user prompts, increased by over 15x between 2025 and early 2026. This makes live tool-called browsing the fastest-growing category of automated web traffic.
Does compacting tool outputs impact multi-turn agent loops negatively? No. According to a benchmark study by Kutschka and Geiger (May 2026), tool execution results account for the single largest token volume inside multi-turn agent loops. Compacting tool results yields up to 32% additional token savings without degrading multi-turn parsing when paired with schema-valid inputs.