XML Sitemap Parsing at Scale: From 100 to 100,000 URLs

Parsing one XML sitemap is easy. You fetch the file, run it through an XML parser, and pull out the <loc> tags. Twenty lines of Python or Node.js, and you're done.

Parsing 10,000 sitemaps across thousands of domains, where some are gzipped, some are nested three levels deep in sitemap index files, some return JavaScript-rendered pages, and some are just broken XML, is a different problem entirely.

This guide covers what it takes to build an XML sitemap parser that works at scale. We'll go through the basics, hit every wall you'll run into between 100 and 100,000 URLs, and show how Context.dev Map URLs supplies an indexed inventory when your goal is URL discovery.

The basics: parsing a single XML sitemap

Before we get into scale, let's start with the baseline. An XML sitemap follows the Sitemaps protocol, and the structure is simple:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/page-1</loc>
    <lastmod>2026-03-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/page-2</loc>
    <lastmod>2026-02-20</lastmod>
  </url>
</urlset>

A minimal parser in Node.js using fast-xml-parser looks like this:

import { XMLParser } from 'fast-xml-parser';
 
async function parseSitemap(url: string): Promise<string[]> {
	const response = await fetch(url);
	const xml = await response.text();
	const parser = new XMLParser();
	const result = parser.parse(xml);
 
	const urls = result.urlset?.url;
	if (!urls) return [];
 
	return Array.isArray(urls) ? urls.map((u: any) => u.loc) : [urls.loc];
}

In Python, the equivalent with lxml:

import requests
from lxml import etree
 
def parse_sitemap(url: str) -> list[str]:
    response = requests.get(url, timeout=30)
    root = etree.fromstring(response.content)
    namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
    return [loc.text for loc in root.findall('.//ns:loc', namespace)]

This works for a single, well-formed sitemap. But the moment you start hitting real websites at scale, every assumption in this code breaks.

Wall #1: Sitemap discovery

The first problem isn't parsing. It's finding the sitemap in the first place.

The protocol says sitemaps should live at /sitemap.xml. In practice, websites put them everywhere. Here are real patterns you'll encounter across thousands of domains:

/sitemap.xml (standard)
/sitemap_index.xml
/sitemap/sitemap-index.xml
/sitemaps/main.xml
/wp-sitemap.xml (WordPress)
/sitemap.php
/server-sitemap-index.xml (Next.js)
/page-sitemap.xml, /post-sitemap.xml (Yoast)

Your parser needs a discovery layer. The most reliable approach checks two sources:

1. robots.txt parsing. The Sitemap: directive in robots.txt is the most authoritative signal. But not every site includes it, and some robots.txt files contain multiple sitemap directives or point to sitemap index files.

async function findSitemapsFromRobotsTxt(domain: string): Promise<string[]> {
	try {
		const response = await fetch(`https://${domain}/robots.txt`);
		const text = await response.text();
		return text
			.split('\n')
			.filter((line) => line.toLowerCase().startsWith('sitemap:'))
			.map((line) => line.split(':', 2).slice(1).join(':').trim());
	} catch {
		return [];
	}
}

2. Common path fallback. If robots.txt doesn't help, you probe a list of known sitemap paths with HEAD requests. More HTTP requests, more latency, but necessary if you want decent coverage.

At scale, this discovery step alone can account for 40-60% of your total request volume. Every domain needs multiple probes before you even start parsing.

Wall #2: Sitemap index recursion

Large sites split their URL inventory across sitemap index files, which are sitemaps that point to other sitemaps (for a deeper primer on the format, see our guide on what a sitemap is and why it matters). Those child sitemaps can themselves be index files, so you end up with a recursive tree.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-products-1.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products-2.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml</loc>
  </sitemap>
</sitemapindex>

Your parser needs to differentiate between a <urlset> (contains actual URLs) and a <sitemapindex> (contains pointers to other sitemaps), then recursively resolve the full tree.

Here's where it gets tricky at scale:

Depth limits. You need a recursion depth limit to avoid infinite loops from circular references. Three to four levels is typically safe.
Fanout. A single sitemap index might point to 500 child sitemaps. If each of those is also an index, you're suddenly making 250,000 HTTP requests for a single domain.
Concurrency control. You can't fire all these requests at once without hammering the target server. You need per-domain rate limiting and connection pooling.
Partial failures. What happens when 3 out of 500 child sitemaps return 404? You need to decide whether to fail the entire parse or return partial results.

async function resolveAllUrls(url: string, depth: number = 0): Promise<string[]> {
	if (depth > 3) return [];
 
	const xml = await fetchAndParse(url);
 
	// It's a sitemap index: recurse
	if (xml.sitemapindex) {
		const children = Array.isArray(xml.sitemapindex.sitemap) ? xml.sitemapindex.sitemap : [xml.sitemapindex.sitemap];
 
		const results = await Promise.allSettled(children.map((child: any) => resolveAllUrls(child.loc, depth + 1)));
 
		return results.filter((r) => r.status === 'fulfilled').flatMap((r) => (r as PromiseFulfilledResult<string[]>).value);
	}
 
	// It's a urlset: extract URLs
	if (xml.urlset?.url) {
		const urls = Array.isArray(xml.urlset.url) ? xml.urlset.url : [xml.urlset.url];
		return urls.map((u: any) => u.loc);
	}
 
	return [];
}

This is the recursive resolution code. It handles the happy path. But production code needs timeout handling, retry logic, deduplication, and memory management on top of this.

Wall #3: Gzip compression

Large sitemaps are often served compressed. You'll encounter two patterns:

Gzipped files at .xml.gz URLs. The URL itself tells you it's compressed.
Gzip content-encoding on .xml URLs. The server compresses the response transparently via HTTP headers.

Your parser needs to handle both. The first case requires explicit decompression. The second is usually handled by your HTTP client's Accept-Encoding header, but not always. Some servers serve gzipped content without the correct Content-Encoding header, and you just get garbled data that fails XML parsing.

import { gunzipSync } from 'zlib';
 
async function fetchSitemapContent(url: string): Promise<string> {
	const response = await fetch(url);
	const buffer = Buffer.from(await response.arrayBuffer());
 
	// Check for gzip magic number (1f 8b)
	if (buffer[0] === 0x1f && buffer[1] === 0x8b) {
		return gunzipSync(buffer).toString('utf-8');
	}
 
	return buffer.toString('utf-8');
}

The gzip magic number check is a defensive pattern. Instead of trusting the URL extension or HTTP headers, you inspect the actual bytes. This catches the edge cases that break naive implementations.

At scale, gzip handling also affects memory management. A large uncompressed sitemap can be much smaller on the wire, then expand sharply during decompression. If you're processing hundreds of these concurrently, the memory spike can crash your process.

Wall #4: Malformed XML

The XML spec is strict. Real-world sitemaps are not. Here's what you'll actually see across thousands of domains:

Unescaped ampersands in URLs: & instead of &
Missing XML declarations or incorrect encoding headers
Namespace mismatches where the namespace is omitted or non-standard
BOM characters at the start of the file
HTML mixed into XML because the server returns a full HTML page with embedded XML when the sitemap is dynamically generated
Truncated files where the server cut the response mid-stream
UTF-8 encoding errors with invalid byte sequences

A strict XML parser rejects all of these. A production sitemap parser has to deal with them:

function sanitizeXml(raw: string): string {
	let xml = raw;
 
	// Strip BOM
	xml = xml.replace(/^\uFEFF/, '');
 
	// Strip content before the XML declaration or first tag
	const xmlStart = xml.indexOf('<?xml');
	const tagStart = xml.indexOf('<urlset');
	const sitemapStart = xml.indexOf('<sitemapindex');
	const start = Math.min(...[xmlStart, tagStart, sitemapStart].filter((i) => i >= 0));
	if (start > 0) xml = xml.substring(start);
 
	// Fix unescaped ampersands in URLs
	xml = xml.replace(/&(?!amp;|lt;|gt;|quot;|apos;|#)/g, '&amp;');
 
	return xml;
}

Even with sanitization, some sitemaps are so mangled that no XML parser will accept them. At that point you fall back to regex, pulling URLs directly from the raw text with <loc>(.*?)</loc>. Not pretty, but it works.

Wall #5: Rate limiting, blocking, and anti-bot measures

When you're hitting sitemaps across hundreds or thousands of domains, you're making a lot of HTTP requests. Web servers notice.

Rate limiting. Many servers return 429 status codes after a burst of requests. Your parser needs exponential backoff with per-domain tracking. A global retry strategy doesn't work because Domain A's rate limit has nothing to do with Domain B's.

IP blocking. Some servers block IPs that make too many requests too quickly. Running from a single server, you'll eventually get blocked by sites with aggressive bot detection. That pushes you into IP rotation, proxy pools, or residential proxy networks, which cost money and add complexity.

Cloudflare and bot detection. A growing number of sites sit behind Cloudflare or similar CDNs that serve challenge pages to automated requests. Your sitemap request returns a 200 OK, but the body is a JavaScript challenge, not XML. Without a headless browser to solve the challenge, you get no data.

Dynamic rendering. Some sites generate sitemaps through JavaScript. A standard HTTP GET returns an empty shell or a loading page. You need a headless browser (Puppeteer, Playwright) to render the page and get the actual XML. This comes up a lot with SPAs and headless CMS setups.

Each of these problems needs its own infrastructure: proxy management, headless browser pools, challenge-solving services, per-domain rate limiting databases. This is where the engineering cost of a DIY sitemap parser starts to spiral.

Wall #6: Memory and performance at scale

Parsing a large XML file into a DOM tree can eat far more memory than the raw file size suggests. Process a few large sitemaps concurrently and you'll blow through your memory budget fast. Streaming XML parsers fix this.

SAX-style parsers (like sax-js in Node.js or iterparse in Python's lxml) process XML as a stream of events instead of building a full in-memory tree:

from lxml import etree
 
def parse_sitemap_streaming(content: bytes) -> list[str]:
    urls = []
    namespace = 'http://www.sitemaps.org/schemas/sitemap/0.9'
 
    for event, element in etree.iterparse(
        io.BytesIO(content), events=('end',)
    ):
        if element.tag == f'{{{namespace}}}loc':
            urls.append(element.text)
            element.clear()  # Free memory as we go
 
    return urls

Beyond memory, there are performance bottlenecks at every layer:

DNS resolution. Thousands of domains means thousands of DNS lookups. Caching and pre-resolution help, but DNS can still be a bottleneck.
Connection overhead. TLS handshakes are expensive. Connection pooling with keep-alive helps for multiple requests to the same domain, but across thousands of domains, every request is a cold start.
Deduplication. Sitemaps frequently contain duplicate URLs, especially when a sitemap index has overlapping child sitemaps. At 100,000+ URLs, the deduplication set alone becomes a nontrivial data structure to manage in memory.

The build vs. buy inflection point

A basic sitemap parser takes a day to build. A production parser that handles all the edge cases above takes weeks. And then you maintain it forever: new anti-bot measures, parsing failures on sites you haven't seen before, proxy infrastructure, uptime monitoring.

The inflection point usually hits around 1,000 domains. Below that, a scrappy script with some error handling works fine. Above that, you're building infrastructure that has nothing to do with your actual product.

Discover URLs with Context.dev Map URLs

Map URLs returns URLs from Context.dev's index with available page metadata. It is useful when you want a URL inventory without running an XML discovery pipeline. For an exact view of a site's declared sitemap, keep the direct parsing workflow above.

How it works

One API call, one endpoint:

curl -X GET "https://api.context.dev/v1/web/urls?domain=example.com" \
  -H "Authorization: Bearer YOUR_API_KEY"

The response is an object whose urls array contains objects with url and optional title, description, keywords, and language fields.

With the official SDK:

import ContextDev from 'context.dev';
 
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const result = await client.web.mapUrls({ domain: 'shopify.com', maxLinks: 1000 });
 
console.log(result.urls.map((entry) => entry.url));
console.log(result.urls.length);

from context.dev import ContextDev
 
client = ContextDev(api_key="YOUR_API_KEY")
result = client.web.map_urls(domain="shopify.com", max_links=1000)
 
for entry in result.urls:
    print(entry.url, entry.title)

Each SDK entry is an object. Read entry.url for the URL; do not pass the whole object into Scrape.

Inventory limits and metadata

Use maxLinks to bound the response and urlRegex to select useful paths. Metadata may be absent when it is not on record. An indexed inventory can differ from the live sitemap, and partial or capped responses should not be used to conclude that a page was removed.

Real workflow patterns

Map URLs works as a discovery layer feeding into other operations. A few common patterns:

Map to scrape. Select up to 20 matching URL objects, then request Markdown from each URL with unified Scrape:

const site = await client.web.mapUrls({
  domain: 'competitor.com',
  urlRegex: '/products/|/pricing',
  maxLinks: 20,
});
 
const pages = await Promise.all(
  site.urls.map(async ({ url }) => {
    const page = await client.web.scrape({ url, formats: { markdown: true } });
    if (page.markdown.data === null) throw new Error(`Markdown extraction failed for ${url}`);
    return { url: page.url, markdown: page.markdown.data };
  }),
);

Indexed URL monitoring. Compare Map URL inventories over time to find newly indexed or missing URLs. An index change is a discovery signal; confirm removals against the live page or sitemap before deleting stored content:

const site = await client.web.mapUrls({ domain, maxLinks: 100000 });
if (site.partial || site.urls.length === 100000) {
  throw new Error('Inventory may be incomplete; do not compare snapshots');
}
const currentUrls = new Set(site.urls.map((entry) => entry.url));
const previousUrls = new Set<string>(loadPreviousSnapshot(domain));
 
const newPages = [...currentUrls].filter((url) => !previousUrls.has(url));
const missingFromIndex = [...previousUrls].filter((url) => !currentUrls.has(url));

Map to brand enrichment. Combine Map URLs with Context.dev's brand data API to build company profiles with logos, colors, products, and page inventories:

const [site, response] = await Promise.all([
  client.web.mapUrls({ domain: 'target.com', maxLinks: 1000 }),
  client.brand.retrieve({ type: 'by_domain', domain: 'target.com' }),
]);
 
console.log(site.urls.map((entry) => entry.url));
console.log(response.brand?.logos, response.brand?.colors, response.brand?.industries);

Map URLs, Scrape, and Brand retrieval share one API key, so discovery can feed both content ingestion and company enrichment.

Compare the data each approach returns

Direct sitemap parsing and Map URLs answer different questions. Choose based on the source data your workflow needs.

Requirement	Direct sitemap parsing	Context.dev Map URLs
Input	Sitemap URL or a discovery process	Domain with optional filters
URL source	Publisher-declared XML files	Context.dev's index
Metadata	XML fields such as `lastmod` when supplied	Available title, description, keywords, and language
Coverage limit	Files successfully fetched and parsed	Index coverage, `maxLinks`, and partial responses
XML processing	Handle compression, recursion, and parsing	No XML returned to your application
Page content	Fetch each selected page separately	Use Scrape or Crawl after discovery

Measure coverage against your own required URLs. A successful index lookup does not establish that every live page or sitemap entry is present.

When DIY still makes sense

There are cases where building your own parser is the right call:

Single domain, internal use. If you're parsing your own sitemap for internal monitoring, a simple script is fine. You control the format and there are no anti-bot issues.
Highly custom parsing logic. If you need to extract and process sitemap metadata (like <lastmod>, <changefreq>, or custom XML extensions) in domain-specific ways, a custom parser gives you full control.
Airgapped environments. If your infrastructure can't make external API calls, you'll need a self-hosted solution.

For everything else (multi-domain extraction, competitive intelligence, AI pipelines, SEO auditing at scale) the API approach wins on engineering time, reliability, and total cost.

Getting started

Start with an indexed URL inventory:

1. Get an API key. Sign up at context.dev and grab your API key from the dashboard.

2. Install the SDK.

# Node.js / TypeScript
npm install context.dev
 
# Python
pip install context.dev

3. Request indexed URLs for a domain.

import ContextDev from 'context.dev';
 
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const result = await client.web.mapUrls({ domain: 'stripe.com', maxLinks: 1000 });
 
console.log(`Returned ${result.urls.length} URLs`);
result.urls.forEach((entry) => console.log(entry.url));

4. Pipe into your workflow. Feed the URLs into scraping, analysis, indexing, or whatever your application needs.

XML sitemap parsing looks simple until it isn't. The gap between parsing one well-formed sitemap and reliably extracting URLs from thousands of real-world domains is filled with recursive index files, gzip, broken XML, anti-bot measures, and infrastructure overhead that has nothing to do with the problem you're actually trying to solve.

Context.dev Map URLs supplies an indexed URL inventory through one request. Use it for discovery, inspect its limits, and reconcile with live sitemaps when completeness matters.

If you need URL discovery across domains, get your API key and try Map URLs. Use the XML workflow above when you need the original sitemap data.

Parsing one XML sitemap is easy. You fetch the file, run it through an XML parser, and pull out the <loc> tags. Twenty lines of Python or Node.js, and you're done.

The basics: parsing a single XML sitemap

Before we get into scale, let's start with the baseline. An XML sitemap follows the Sitemaps protocol, and the structure is simple:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/page-1</loc>
    <lastmod>2026-03-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/page-2</loc>
    <lastmod>2026-02-20</lastmod>
  </url>
</urlset>

A minimal parser in Node.js using fast-xml-parser looks like this:

import { XMLParser } from 'fast-xml-parser';
 
async function parseSitemap(url: string): Promise<string[]> {
	const response = await fetch(url);
	const xml = await response.text();
	const parser = new XMLParser();
	const result = parser.parse(xml);
 
	const urls = result.urlset?.url;
	if (!urls) return [];
 
	return Array.isArray(urls) ? urls.map((u: any) => u.loc) : [urls.loc];
}

In Python, the equivalent with lxml:

import requests
from lxml import etree
 
def parse_sitemap(url: str) -> list[str]:
    response = requests.get(url, timeout=30)
    root = etree.fromstring(response.content)
    namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
    return [loc.text for loc in root.findall('.//ns:loc', namespace)]

This works for a single, well-formed sitemap. But the moment you start hitting real websites at scale, every assumption in this code breaks.

Wall #1: Sitemap discovery

The first problem isn't parsing. It's finding the sitemap in the first place.

The protocol says sitemaps should live at /sitemap.xml. In practice, websites put them everywhere. Here are real patterns you'll encounter across thousands of domains:

/sitemap.xml (standard)
/sitemap_index.xml
/sitemap/sitemap-index.xml
/sitemaps/main.xml
/wp-sitemap.xml (WordPress)
/sitemap.php
/server-sitemap-index.xml (Next.js)
/page-sitemap.xml, /post-sitemap.xml (Yoast)

Your parser needs a discovery layer. The most reliable approach checks two sources:

async function findSitemapsFromRobotsTxt(domain: string): Promise<string[]> {
	try {
		const response = await fetch(`https://${domain}/robots.txt`);
		const text = await response.text();
		return text
			.split('\n')
			.filter((line) => line.toLowerCase().startsWith('sitemap:'))
			.map((line) => line.split(':', 2).slice(1).join(':').trim());
	} catch {
		return [];
	}
}

2. Common path fallback. If robots.txt doesn't help, you probe a list of known sitemap paths with HEAD requests. More HTTP requests, more latency, but necessary if you want decent coverage.

At scale, this discovery step alone can account for 40-60% of your total request volume. Every domain needs multiple probes before you even start parsing.

Wall #2: Sitemap index recursion

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-products-1.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products-2.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml</loc>
  </sitemap>
</sitemapindex>

Your parser needs to differentiate between a <urlset> (contains actual URLs) and a <sitemapindex> (contains pointers to other sitemaps), then recursively resolve the full tree.

Here's where it gets tricky at scale:

Depth limits. You need a recursion depth limit to avoid infinite loops from circular references. Three to four levels is typically safe.
Fanout. A single sitemap index might point to 500 child sitemaps. If each of those is also an index, you're suddenly making 250,000 HTTP requests for a single domain.
Concurrency control. You can't fire all these requests at once without hammering the target server. You need per-domain rate limiting and connection pooling.
Partial failures. What happens when 3 out of 500 child sitemaps return 404? You need to decide whether to fail the entire parse or return partial results.

async function resolveAllUrls(url: string, depth: number = 0): Promise<string[]> {
	if (depth > 3) return [];
 
	const xml = await fetchAndParse(url);
 
	// It's a sitemap index: recurse
	if (xml.sitemapindex) {
		const children = Array.isArray(xml.sitemapindex.sitemap) ? xml.sitemapindex.sitemap : [xml.sitemapindex.sitemap];
 
		const results = await Promise.allSettled(children.map((child: any) => resolveAllUrls(child.loc, depth + 1)));
 
		return results.filter((r) => r.status === 'fulfilled').flatMap((r) => (r as PromiseFulfilledResult<string[]>).value);
	}
 
	// It's a urlset: extract URLs
	if (xml.urlset?.url) {
		const urls = Array.isArray(xml.urlset.url) ? xml.urlset.url : [xml.urlset.url];
		return urls.map((u: any) => u.loc);
	}
 
	return [];
}

This is the recursive resolution code. It handles the happy path. But production code needs timeout handling, retry logic, deduplication, and memory management on top of this.

Wall #3: Gzip compression

Large sitemaps are often served compressed. You'll encounter two patterns:

Gzipped files at .xml.gz URLs. The URL itself tells you it's compressed.
Gzip content-encoding on .xml URLs. The server compresses the response transparently via HTTP headers.

import { gunzipSync } from 'zlib';
 
async function fetchSitemapContent(url: string): Promise<string> {
	const response = await fetch(url);
	const buffer = Buffer.from(await response.arrayBuffer());
 
	// Check for gzip magic number (1f 8b)
	if (buffer[0] === 0x1f && buffer[1] === 0x8b) {
		return gunzipSync(buffer).toString('utf-8');
	}
 
	return buffer.toString('utf-8');
}

The gzip magic number check is a defensive pattern. Instead of trusting the URL extension or HTTP headers, you inspect the actual bytes. This catches the edge cases that break naive implementations.

Wall #4: Malformed XML

The XML spec is strict. Real-world sitemaps are not. Here's what you'll actually see across thousands of domains:

Unescaped ampersands in URLs: & instead of &
Missing XML declarations or incorrect encoding headers
Namespace mismatches where the namespace is omitted or non-standard
BOM characters at the start of the file
HTML mixed into XML because the server returns a full HTML page with embedded XML when the sitemap is dynamically generated
Truncated files where the server cut the response mid-stream
UTF-8 encoding errors with invalid byte sequences

A strict XML parser rejects all of these. A production sitemap parser has to deal with them:

function sanitizeXml(raw: string): string {
	let xml = raw;
 
	// Strip BOM
	xml = xml.replace(/^\uFEFF/, '');
 
	// Strip content before the XML declaration or first tag
	const xmlStart = xml.indexOf('<?xml');
	const tagStart = xml.indexOf('<urlset');
	const sitemapStart = xml.indexOf('<sitemapindex');
	const start = Math.min(...[xmlStart, tagStart, sitemapStart].filter((i) => i >= 0));
	if (start > 0) xml = xml.substring(start);
 
	// Fix unescaped ampersands in URLs
	xml = xml.replace(/&(?!amp;|lt;|gt;|quot;|apos;|#)/g, '&amp;');
 
	return xml;
}

Wall #5: Rate limiting, blocking, and anti-bot measures

When you're hitting sitemaps across hundreds or thousands of domains, you're making a lot of HTTP requests. Web servers notice.

Wall #6: Memory and performance at scale

SAX-style parsers (like sax-js in Node.js or iterparse in Python's lxml) process XML as a stream of events instead of building a full in-memory tree:

from lxml import etree
 
def parse_sitemap_streaming(content: bytes) -> list[str]:
    urls = []
    namespace = 'http://www.sitemaps.org/schemas/sitemap/0.9'
 
    for event, element in etree.iterparse(
        io.BytesIO(content), events=('end',)
    ):
        if element.tag == f'{{{namespace}}}loc':
            urls.append(element.text)
            element.clear()  # Free memory as we go
 
    return urls

Beyond memory, there are performance bottlenecks at every layer:

DNS resolution. Thousands of domains means thousands of DNS lookups. Caching and pre-resolution help, but DNS can still be a bottleneck.
Connection overhead. TLS handshakes are expensive. Connection pooling with keep-alive helps for multiple requests to the same domain, but across thousands of domains, every request is a cold start.
Deduplication. Sitemaps frequently contain duplicate URLs, especially when a sitemap index has overlapping child sitemaps. At 100,000+ URLs, the deduplication set alone becomes a nontrivial data structure to manage in memory.

The build vs. buy inflection point

Discover URLs with Context.dev Map URLs

How it works

One API call, one endpoint:

curl -X GET "https://api.context.dev/v1/web/urls?domain=example.com" \
  -H "Authorization: Bearer YOUR_API_KEY"

The response is an object whose urls array contains objects with url and optional title, description, keywords, and language fields.

With the official SDK:

import ContextDev from 'context.dev';
 
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const result = await client.web.mapUrls({ domain: 'shopify.com', maxLinks: 1000 });
 
console.log(result.urls.map((entry) => entry.url));
console.log(result.urls.length);

from context.dev import ContextDev
 
client = ContextDev(api_key="YOUR_API_KEY")
result = client.web.map_urls(domain="shopify.com", max_links=1000)
 
for entry in result.urls:
    print(entry.url, entry.title)

Each SDK entry is an object. Read entry.url for the URL; do not pass the whole object into Scrape.

Inventory limits and metadata

Real workflow patterns

Map URLs works as a discovery layer feeding into other operations. A few common patterns:

Map to scrape. Select up to 20 matching URL objects, then request Markdown from each URL with unified Scrape:

const site = await client.web.mapUrls({
  domain: 'competitor.com',
  urlRegex: '/products/|/pricing',
  maxLinks: 20,
});
 
const pages = await Promise.all(
  site.urls.map(async ({ url }) => {
    const page = await client.web.scrape({ url, formats: { markdown: true } });
    if (page.markdown.data === null) throw new Error(`Markdown extraction failed for ${url}`);
    return { url: page.url, markdown: page.markdown.data };
  }),
);

const site = await client.web.mapUrls({ domain, maxLinks: 100000 });
if (site.partial || site.urls.length === 100000) {
  throw new Error('Inventory may be incomplete; do not compare snapshots');
}
const currentUrls = new Set(site.urls.map((entry) => entry.url));
const previousUrls = new Set<string>(loadPreviousSnapshot(domain));
 
const newPages = [...currentUrls].filter((url) => !previousUrls.has(url));
const missingFromIndex = [...previousUrls].filter((url) => !currentUrls.has(url));

Map to brand enrichment. Combine Map URLs with Context.dev's brand data API to build company profiles with logos, colors, products, and page inventories:

const [site, response] = await Promise.all([
  client.web.mapUrls({ domain: 'target.com', maxLinks: 1000 }),
  client.brand.retrieve({ type: 'by_domain', domain: 'target.com' }),
]);
 
console.log(site.urls.map((entry) => entry.url));
console.log(response.brand?.logos, response.brand?.colors, response.brand?.industries);

Map URLs, Scrape, and Brand retrieval share one API key, so discovery can feed both content ingestion and company enrichment.

Compare the data each approach returns

Direct sitemap parsing and Map URLs answer different questions. Choose based on the source data your workflow needs.

Requirement	Direct sitemap parsing	Context.dev Map URLs
Input	Sitemap URL or a discovery process	Domain with optional filters
URL source	Publisher-declared XML files	Context.dev's index
Metadata	XML fields such as `lastmod` when supplied	Available title, description, keywords, and language
Coverage limit	Files successfully fetched and parsed	Index coverage, `maxLinks`, and partial responses
XML processing	Handle compression, recursion, and parsing	No XML returned to your application
Page content	Fetch each selected page separately	Use Scrape or Crawl after discovery

Measure coverage against your own required URLs. A successful index lookup does not establish that every live page or sitemap entry is present.

When DIY still makes sense

There are cases where building your own parser is the right call:

Single domain, internal use. If you're parsing your own sitemap for internal monitoring, a simple script is fine. You control the format and there are no anti-bot issues.
Highly custom parsing logic. If you need to extract and process sitemap metadata (like <lastmod>, <changefreq>, or custom XML extensions) in domain-specific ways, a custom parser gives you full control.
Airgapped environments. If your infrastructure can't make external API calls, you'll need a self-hosted solution.

For everything else (multi-domain extraction, competitive intelligence, AI pipelines, SEO auditing at scale) the API approach wins on engineering time, reliability, and total cost.

Getting started

Start with an indexed URL inventory:

1. Get an API key. Sign up at context.dev and grab your API key from the dashboard.

2. Install the SDK.

# Node.js / TypeScript
npm install context.dev
 
# Python
pip install context.dev

3. Request indexed URLs for a domain.

import ContextDev from 'context.dev';
 
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const result = await client.web.mapUrls({ domain: 'stripe.com', maxLinks: 1000 });
 
console.log(`Returned ${result.urls.length} URLs`);
result.urls.forEach((entry) => console.log(entry.url));

4. Pipe into your workflow. Feed the URLs into scraping, analysis, indexing, or whatever your application needs.

Context.dev Map URLs supplies an indexed URL inventory through one request. Use it for discovery, inspect its limits, and reconcile with live sitemaps when completeness matters.

If you need URL discovery across domains, get your API key and try Map URLs. Use the XML workflow above when you need the original sitemap data.

The basics: parsing a single XML sitemap

Wall #1: Sitemap discovery

Wall #2: Sitemap index recursion

Wall #3: Gzip compression

Wall #4: Malformed XML

Wall #5: Rate limiting, blocking, and anti-bot measures

Wall #6: Memory and performance at scale

The build vs. buy inflection point

Discover URLs with Context.dev Map URLs

How it works

Inventory limits and metadata

Real workflow patterns

Compare the data each approach returns

When DIY still makes sense

Getting started

Related articles

Multi-Site Product Data Extraction with Python and Pydantic

Why Your Web Scraper Returns Empty Results: Five Causes and Fixes

How to Scrape Logos from Websites [For Developers]

Production Web Scraping Pipelines in Python

Ship an agent that actually knows things.

The basics: parsing a single XML sitemap

Wall #1: Sitemap discovery

Wall #2: Sitemap index recursion

Wall #3: Gzip compression

Wall #4: Malformed XML

Wall #5: Rate limiting, blocking, and anti-bot measures

Wall #6: Memory and performance at scale

The build vs. buy inflection point

Discover URLs with Context.dev Map URLs

How it works

Inventory limits and metadata

Real workflow patterns

Compare the data each approach returns

When DIY still makes sense

Getting started

Related articles

Multi-Site Product Data Extraction with Python and Pydantic

Why Your Web Scraper Returns Empty Results: Five Causes and Fixes

How to Scrape Logos from Websites [For Developers]

Production Web Scraping Pipelines in Python

Ship an agent that actually knows things.