How to Extract Raw HTML from Any URL with a Single API Call

Sometimes you just need the HTML.

Not parsed Markdown or a structured JSON extraction or an AI summary. Just the raw, fully-rendered HTML of a webpage, the same DOM a real browser sees after JavaScript execution, redirects, and client-side rendering.

You might be feeding it into a custom parser, diffing two versions of a page, or building a monitoring tool that watches for DOM changes. Whatever the reason, "give me the HTML at this URL" sounds simple, and it's way more painful to do reliably than it should be.

This guide covers the practical ways to extract raw HTML from a URL, the problems you'll hit doing it yourself, and how a single API call can replace a lot of infrastructure code.

Why "Just Fetch the HTML" Is Harder Than It Sounds

If you've ever written fetch(url) or requests.get(url) and called it a day, you've probably already been burned. The HTML you get back from a simple HTTP request is often not what you see in a browser. Here's why.

JavaScript-Rendered Pages

The modern web runs on client-side JavaScript. React, Next.js, Vue, Angular: these frameworks render content after the initial page load. A basic HTTP request returns the bare shell:

<!DOCTYPE html>
<html>
	<body>
		<div id="root"></div>
		<script src="/bundle.js"></script>
	</body>
</html>

That's not the HTML you wanted. The actual content (product listings, article text, pricing tables) is injected by JavaScript after the page loads. To get the real HTML, you need a headless browser that executes JavaScript and waits for the DOM to stabilize.

Anti-Bot Protections

Cloudflare, Akamai, PerimeterX, DataDome: a growing number of sites use bot detection that blocks automated requests outright. Your curl command returns a CAPTCHA page or a 403. Your Python script gets rate-limited after three requests.

Getting past these protections requires rotating proxies, browser fingerprinting, TLS fingerprint matching, and sometimes residential IP addresses. This isn't a weekend project. It's ongoing infrastructure.

Redirects, Cookies, and Authentication Walls

Many URLs don't serve content directly. They redirect through chains of shortened URLs, tracking pixels, and consent gates. Some pages require cookies from a previous visit. Others sit behind login walls or geo-restricted CDNs.

A simple GET request doesn't handle any of this. A headless browser with proper session management does.

Dynamic Content and Lazy Loading

Even with JavaScript execution, some content only appears after scrolling, clicking, or waiting for async API calls to resolve. Images lazy-load. Infinite scroll pages only render the first batch. Tabs hide content until clicked.

Getting the "full" HTML means knowing when to stop waiting, and that's harder than it sounds.

The DIY Approach: Headless Browsers

The standard solution for developers who need rendered HTML is to run a headless browser, typically Puppeteer (Chrome) or Playwright (Chrome, Firefox, or WebKit).

Here's a minimal Puppeteer example:

import puppeteer from 'puppeteer';
 
const extractHtml = async (url: string): Promise<string> => {
	const browser = await puppeteer.launch({ headless: true });
	const page = await browser.newPage();
 
	await page.goto(url, { waitUntil: 'networkidle2' });
	const html = await page.content();
 
	await browser.close();
	return html;
};
 
const html = await extractHtml('https://example.com');
console.log(html);

This works for a demo. It falls apart in production:

Resource overhead. Each browser instance consumes 100-300MB of RAM. If you're extracting HTML from 50 URLs concurrently, you need serious compute. Chrome processes leak memory over time, so you need restart logic.

Startup latency. Launching a browser takes 1-3 seconds. Navigating to a page, waiting for JavaScript to execute, and waiting for network idle adds another 3-10 seconds. For a "just get the HTML" use case, this is painfully slow.

Anti-bot detection. Headless Chrome is detectable. Sites check navigator.webdriver, the Chrome DevTools protocol fingerprint, and dozens of other signals. You need puppeteer-extra-plugin-stealth and constant updates to stay ahead.

Infrastructure management. You need to keep Chrome/Chromium updated, manage browser pools, handle crashes and timeouts, and set up retry logic. On serverless platforms, you need special Chrome builds (chrome-aws-lambda) because the standard binary is too large.

Proxy management. If you're scraping at any scale, you need rotating proxies to avoid IP bans. That means proxy providers, rotation logic, error handling for dead proxies, and billing management.

All of this to answer the question: "What's the HTML at this URL?"

The API Approach: One Request, Done

The alternative is to let someone else manage the headless browsers, proxy infrastructure, and anti-bot bypasses, and just call an API. You send a URL, it renders the page in a real browser, deals with anti-bot protections, and hands back the complete HTML. One request, one response.

Here's what that looks like with Context.dev Scrape with HTML output:

cURL

curl --request POST "https://api.context.dev/v1/web/scrape" \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://example.com","formats":{"html":true}}'

An abbreviated response looks like this. Read html.data after checking html.success; other format envelopes and metadata are omitted here:

{
  "url": "https://example.com",
  "html": {
    "requested": true,
    "success": true,
    "data": "<!DOCTYPE html><html>...</html>"
  }
}

The url field returns the resolved URL after any redirects, so you always know where the content actually came from.

Python

Using the Context.dev Python SDK:

from context.dev import ContextDev
 
client = ContextDev(api_key="YOUR_API_KEY")
 
response = client.web.scrape(
    url="https://example.com",
    formats={"html": True},
)
if response.html.data is None:
    raise RuntimeError("HTML extraction failed")
print(response.html.data)

The SDK sends the same unified Scrape request and exposes the rendered document at response.html.data.

JavaScript

import ContextDev from 'context.dev';
 
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
 
const response = await client.web.scrape({
  url: 'https://example.com',
  formats: { html: true },
});
if (response.html.data === null) throw new Error('HTML extraction failed');
console.log(response.html.data);

TypeScript

import ContextDev from 'context.dev';
 
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
 
const scrapeHtml = async (url: string): Promise<string> => {
  const response = await client.web.scrape({ url, formats: { html: true } });
  if (response.html.data === null) {
    throw new Error(`HTML extraction failed for ${url}`);
  }
  return response.html.data;
};
 
const html = await scrapeHtml('https://example.com');

Ruby

require "context_dev"
 
client = ContextDev::Client.new(api_key: "YOUR_API_KEY")
 
response = client.web.scrape(
  url: "https://example.com",
  formats: {html: true},
)
raise "HTML extraction failed" if response.html.data.nil?
puts response.html.data

Using Axios (No SDK)

If you prefer not to use an SDK, the REST API works with any HTTP client:

import axios from 'axios';
 
const scrapeHtml = async (url: string) => {
  const response = await axios.post(
    'https://api.context.dev/v1/web/scrape',
    { url, formats: { html: true } },
    { headers: { Authorization: 'Bearer YOUR_API_KEY' }, timeout: 60000 },
  );
  if (!response.data.html.success) throw new Error('HTML extraction failed');
  return response.data.html.data;
};

What Happens Behind the Scenes

When you request formats.html from Scrape, Context.dev handles the rendering pipeline.

The URL gets loaded in a headless browser that executes all JavaScript and waits for the DOM to stabilize, so you get the rendered HTML, not the pre-render source. If a request is blocked, the system escalates through proxy tiers (datacenter, residential, mobile) until it gets through. All redirects are followed, and the final resolved URL comes back alongside the HTML so you know what page was actually rendered. Browser fingerprinting, TLS fingerprint matching, and session management are all automatic. No stealth plugins to configure on your end.

The result is the same HTML you'd see if you opened Chrome DevTools and copied the <html> element, delivered in the JSON response to POST /web/scrape.

Common Use Cases for Raw HTML Extraction

Custom Parsing Pipelines

If you're building a scraper for a specific data format (say, extracting structured product data from e-commerce sites or pulling financial data from SEC filings), you need the raw HTML to run your own parsing logic. CSS selectors, XPath queries, and regex patterns all operate on HTML. An API that returns clean HTML lets you skip the browser infrastructure and focus on the parsing.

DOM Diffing and Change Detection

Monitoring websites for changes (price drops, content updates, new job postings) requires comparing HTML snapshots over time. You need consistent, fully-rendered HTML for diffs to be meaningful. If your scraper sometimes returns JavaScript-rendered content and sometimes returns the raw shell, your diffs are useless.

SEO Auditing and Analysis

SEO tools need to see what search engines see. That means fully rendered HTML with all meta tags, Open Graph data, structured data (JSON-LD), and canonical URLs intact. Extracting raw HTML is the first step in any programmatic SEO audit.

Archiving and Compliance

Some industries require archiving web content for compliance. Financial services, legal, and healthcare organizations often need to store snapshots of web pages as they appeared at a specific point in time. Raw HTML is the most complete representation.

Feeding Data into AI Pipelines

While Markdown conversion is often better for LLM consumption, there are cases where you want the raw HTML: for example, when training models on web layout structure, building accessibility tools, or extracting data from HTML tables that lose structure in Markdown conversion.

Going Beyond HTML: Other Scrape Formats and Map URLs

Once you're extracting HTML from URLs, you'll probably need related capabilities too. Select additional formats in the same Scrape request, or use Map URLs to discover pages first:

URL to Markdown

If your goal is to feed web content into an LLM or RAG pipeline, raw HTML is usually more than you need. The Markdown format returns GitHub Flavored Markdown. Use sharedParams.mainContentOnly to focus on the page body:

response = client.web.scrape(
    url="https://example.com/blog/post",
    formats={"markdown": True},
    shared_params={"main_content_only": True},
    markdown_params={"include_links": True, "include_images": False},
)
if response.markdown.data is None:
    raise RuntimeError("Markdown extraction failed")
print(response.markdown.data)

You can control whether links and images are preserved, useful for RAG pipelines where you want text-only content, versus agent workflows where links matter.

Image Extraction

The images format returns page images in images.data, with url and alt fields. Optional enrichment can add dimensions, classification, and hosted file URLs:

response = client.web.scrape(
    url="https://example.com",
    formats={"images": True},
)
if response.images.data is None:
    raise RuntimeError("Image extraction failed")
for image in response.images.data or []:
    print(image.url, image.alt)

URL Discovery

Before scraping individual pages, you often need to know what pages exist. Map URLs returns indexed URLs as objects with a url and optional page metadata. Bound the list with maxLinks:

site = client.web.map_urls(domain="example.com", max_links=20)
for entry in site.urls:
    page = client.web.scrape(url=entry.url, formats={"html": True})
    if page.html.data is not None:
        print(entry.url, page.html.data)

Schema-Based Data Extraction

If you're extracting HTML just to parse specific fields, the Scrape JSON format can return fields from that page using jsonParams.schema:

response = client.web.scrape(
    url="https://example.com/pricing",
    formats={"json": True},
    json_params={
        "schema": {
            "type": "object",
            "properties": {"pricing": {"type": ["string", "null"]}},
            "required": ["pricing"],
        },
        "instructions": "Extract the pricing shown on this page; use null if absent.",
    },
)
if response.json_.data is None:
    raise RuntimeError("JSON extraction failed")
print(response.json_.data)

Validate response.json_.data in Python before using it. JSON output adds 4 credits to the base Scrape charge when extraction succeeds. For research across multiple pages, use Answers with a task and an example json_format object.

DIY vs. API: When to Use Which

Not every use case needs an API. Here's how I'd think about it:

Use a simple HTTP request (requests.get / fetch) when:

• The target page is server-rendered (no JavaScript frameworks)
• There's no bot protection
• You're scraping a small number of pages infrequently
• You control the target site

Use a headless browser (Puppeteer/Playwright) when:

• You need to interact with the page (click, scroll, fill forms)
• You're building a browser automation tool, not a data pipeline
• You're scraping your own application for testing

Use an HTML extraction API when:

• You need rendered HTML from JavaScript-heavy sites
• Target sites have anti-bot protections
• You're scraping at scale (dozens to thousands of URLs)
• You don't want to manage browser infrastructure
• Reliability matters. You need consistent results, not "works most of the time"

For most production use cases, the API wins on both developer time and operational cost. Running headless Chrome at scale gets expensive fast, and not just in compute. It's the engineering hours spent keeping it working that really add up.

Getting Started

Context.dev offers SDKs for TypeScript, Python, Ruby, and Go, plus a REST API that works with any HTTP client.

To start extracting HTML:

1. Sign up for a free account
2. Grab your API key from the dashboard
3. Make your first request:

curl --request POST "https://api.context.dev/v1/web/scrape" \
  --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://example.com","formats":{"html":true}}'

The response includes rendered HTML in html.data, the resolved url, and a per-format html.success flag. No browsers to manage, no proxies to configure, no stealth plugins to maintain. Just the HTML.

Frequently Asked Questions

Does the API execute JavaScript?

Yes. formats.html returns the rendered DOM. Set sharedParams.waitFor or browser actions when content needs more time or interaction before capture. For the original HTTP response instead, request formats.bytes.

How does it handle pages that block scrapers?

Context.dev uses automatic proxy escalation and browser fingerprinting when a request is blocked. No universal success percentage is claimed without a dated benchmark and target definition.

What's the rate limit?

Rate limits depend on your plan. The API returns standard HTTP 429 responses when limits are hit, with retry-after headers. Check the pricing page for details.

Can I scrape pages behind a login?

Scrape works with publicly accessible URLs. For pages behind authentication, you'd need to pass session cookies or use a headless browser with login automation.

What's the difference between HTML and Markdown output?

Both are formats on POST /web/scrape. HTML preserves the rendered DOM; Markdown converts page content into readable text and links. Request both with formats: { html: true, markdown: true } and read html.data and markdown.data. Shared content filters apply to both.

Is there a maximum page size?

Standard web pages work fine. Extremely large pages (multi-megabyte DOMs) may time out, but you can set a custom timeout up to 300 seconds (5 minutes) with timeoutOpts.milliseconds.