Website Crawler {API}
Crawl any website and extract Markdown from every page.
Point the crawler at any URL and get back clean Markdown for every page it discovers. Control depth, page limits, URL filters, and subdomain following — all with a single POST request.
1 credit per page crawled. Up to 500 pages per request.
What You Get
Crawl entire sites and get structured Markdown for every page.
Multi-page Markdown extraction
Crawl up to 500 pages and get clean Markdown for every page discovered
Depth & page limits
Control crawl depth from the start URL and cap the total number of pages
URL regex filtering
Only follow and scrape URLs matching your custom regex pattern
Subdomain following
Optionally crawl across subdomains of the starting domain
How It Works
We handle link discovery, page fetching, and Markdown conversion for you.
POST a starting URL
Send a fully qualified URL with optional depth, page limit, and regex filters
Links are discovered
The crawler follows same-domain links respecting your depth and regex constraints
Pages converted to Markdown
Each page is fetched and its content is extracted as clean Markdown
Results returned
Get all pages with Markdown content, metadata, crawl depth, and status codes
API Response
Crawl results for docs.context.dev
POST /v1/web/crawl{
"results": [
{
"markdown": "# Context.dev Documentation\n\nWelcome to the Context.dev API docs.\nLearn how to extract brand data, scrape websites,\nand query any domain with a single API call.\n\n## Getting Started\n\n- [Authentication](/authentication)\n- [Quick Start](/quickstart)\n- [API Reference](/api-reference)\n...",
"metadata": {
"url": "https://docs.context.dev",
"title": "Context.dev Documentation",
"crawlDepth": 0,
"statusCode": 200,
"success": true
}
},
{
"markdown": "# Authentication\n\nAll API requests require a Bearer token...\n",
"metadata": {
"url": "https://docs.context.dev/authentication",
"title": "Authentication - Context.dev",
"crawlDepth": 1,
"statusCode": 200,
"success": true
}
}
],
"metadata": {
"numUrls": 12,
"maxCrawlDepth": 2,
"numSucceeded": 11,
"numFailed": 1
}
}Context at scale
Join 5,000+ businesses using Context.dev to enrich their products with structured web data.