How can developers crawl an entire website through an API?
Short answer
Use a crawling API that starts from one URL, discovers the site’s links, fetches each page, and returns Markdown per page. On Context.dev that is one POST to /v1/web/crawl with the start URL, maxPages, maxDepth, and an optional urlRegex; it returns up to 500 pages synchronously at 1 credit per successful page. For a URL list only, call GET /v1/web/urls, which reads the site’s sitemaps and returns up to 100,000 URLs for 1 credit. For more than 500 pages, submit a batch in crawl mode and collect results by webhook.
A whole-site crawl has three parts: discovery, fetching, and output. Discovery means reading the sitemap or following internal links from a start URL while respecting scope rules. Fetching means loading each page, in a browser so JavaScript-rendered content is included, with retries and concurrency limits. Output means converting each page to Markdown or JSON with its URL, title, status code, and depth, so a RAG pipeline or search index can use it directly. A crawling API does all three behind one request; a URL list plus a scraper does only the last two.
On Context.dev, send POST https://api.context.dev/v1/web/crawl with a JSON body such as { "url": "https://example.com/docs", "maxPages": 100, "maxDepth": 2, "urlRegex": "/docs/.*" } and a bearer token. The crawler follows same-domain links from the start URL up to maxDepth, keeps only URLs that match urlRegex, renders each page, and returns results, an array of pages with markdown and metadata (url, title, statusCode, crawlDepth, success), plus totals for numSucceeded, numFailed, and numSkipped. Set followSubdomains to include docs.example.com from example.com. Each successful page costs 1 credit and failed pages are not charged.
Pick the endpoint by size. Up to 500 pages, call Crawl and get everything back in one response. When you only need the URLs, call Map: it reads nested sitemap indexes, falls back to homepage links when there is no sitemap, and returns titles and descriptions where available. Beyond 500 pages, submit a batch with mode set to crawl, or pass up to 100,000 URLs from a Map result; the job runs in the background, you poll or receive a completion webhook, and result files stay downloadable for 180 days. Credits are reserved on submit and refunded for pages that fail.
Whatever endpoint you use, scope the crawl to the paths you need with urlRegex and maxDepth, respect robots.txt and the site’s terms, and keep concurrency modest on small sites. Expect that orphaned pages, login-protected content, and URLs generated only by JavaScript interactions may be missed, and read numFailed and numSkipped to see what the crawl could not reach. Crawling discovers pages; scraping extracts data from them. Context.dev combines both so your pipeline receives finished Markdown rather than a list of links to fetch.
| Endpoint | How it runs | Scope controls | Output | Credits |
|---|---|---|---|---|
| Crawl (POST /v1/web/crawl) | Synchronous, up to 500 pages per call | maxPages, maxDepth, urlRegex, followSubdomains | Markdown plus URL, title, status, and depth per page | 1 per successful page |
| Map (GET /v1/web/urls) | Synchronous, reads sitemaps or homepage links | includeSubdomains, maxLinks | Up to 100,000 URLs with titles and descriptions when available | 1 per call, 2 with search |
| Batches (POST /v1/batch/submit) | Background job from a start URL, sitemap, or URL list | Crawl scope or your own URL list, up to 100,000 URLs | Markdown or HTML per URL by polling, webhook, or files kept 180 days | 1 per successful URL, unused reservations refunded |
| If this is your situation | Use |
|---|---|
| A docs site or knowledge base under 500 pages for RAG | Crawl |
| You need the URL list before deciding what to fetch | Map, then Scrape the pages you choose |
| Thousands of pages with a webhook when the job finishes | Batches in crawl mode |
| You want to re-crawl when the site changes | A sitemap Monitor that triggers the crawl |
Last reviewed