A sitemap is a file that lists every important URL on a website, giving search engines and applications a structured roadmap of the site's content. Think of it as a table of contents for the internet - instead of forcing a crawler to discover pages by following links one at a time, a sitemap hands over the full inventory upfront.
For developers building AI agents, data pipelines, or SEO tools, sitemaps are one of the most powerful (and underused) entry points into a website's data. In this guide, we'll cover everything you need to know: what sitemaps are, how they work, the different formats, why they matter for SEO, and how to retrieve any website's sitemap programmatically using an API.
How Sitemaps Work
At the most basic level, a sitemap is an XML file hosted at a predictable URL - usually https://example.com/sitemap.xml. When a search engine like Google or Bing visits a website, one of the first things it does is check for a sitemap. The file tells the crawler which pages exist, when they were last updated, how often they change, and how important they are relative to other pages on the site.
Here's a minimal example of what an XML sitemap looks like:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-03-15</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://example.com/about</loc>
<lastmod>2026-01-10</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>Each <url> entry contains a <loc> tag with the full URL, and optionally includes metadata like the last modification date (<lastmod>), the expected change frequency (<changefreq>), and a relative priority score (<priority>). Search engines use this data to decide how frequently to re-crawl specific pages and how to allocate their crawl budget across the site.
The sitemap protocol was jointly developed by Google, Yahoo, and Microsoft in 2006 and has since become a universal standard across all major search engines.
Types of Sitemaps
Not all sitemaps are created equal. Depending on the type of content a website serves, different sitemap formats come into play.
XML Sitemaps
The standard format. XML sitemaps are the most common and the most important for SEO. They follow the protocol defined at sitemaps.org and are designed specifically for search engine consumption. Most CMS platforms like WordPress, Shopify, and Webflow generate XML sitemaps automatically.
Sitemap Index Files
Large websites often have thousands or even millions of pages, so they use a sitemap index file that points to multiple individual sitemaps. This is essentially a sitemap of sitemaps.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemap-posts.xml</loc>
<lastmod>2026-03-20</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-products.xml</loc>
<lastmod>2026-03-18</lastmod>
</sitemap>
</sitemapindex>Major e-commerce sites, news outlets, and content platforms almost always use sitemap index files. Amazon, for instance, has hundreds of individual sitemaps nested under its index.
HTML Sitemaps
An HTML sitemap is a human-readable page on a website that lists links organized by category or hierarchy. While less useful for search engines than XML sitemaps, HTML sitemaps help users navigate complex websites and can improve internal linking. You've probably seen these in the footer of large corporate sites.
Image and Video Sitemaps
Google supports extensions to the standard sitemap protocol for images and videos. An image sitemap adds <image:image> tags inside each URL entry, pointing to hosted images on that page. A video sitemap does the same for video content, including metadata like title, description, duration, and thumbnail URL. These are critical for publishers and media companies who want their visual content indexed in Google Images and Google Video search results.
News Sitemaps
Google News has its own sitemap format that includes publication name, language, publication date, and article title. News sitemaps are essential for any publisher that wants to appear in Google News results and the "Top Stories" carousel. Articles must have been published within the last 48 hours to be included.
Why Sitemaps Matter for SEO
Sitemaps play a foundational role in search engine optimization. Here's why every website should have one - and why SEO professionals care about them so much.
Faster Indexing
When you publish a new page, there's no guarantee Google will discover it immediately. If the page isn't well-linked from other parts of your site (or from external sources), it might sit undiscovered for weeks. A sitemap ensures that search engines know about every page from the moment it's published.
Crawl Budget Optimization
Search engines allocate a finite "crawl budget" to each website - the number of pages they'll crawl in a given time period. A well-structured sitemap helps search engines prioritize the most important pages and skip low-value ones, making more efficient use of that budget.
Surfacing Orphan Pages
An orphan page is a page that exists on a website but isn't linked to from any other page. Search engines can't discover orphan pages through normal crawling. Sitemaps are the safety net - they ensure orphan pages still get crawled and indexed.
Understanding Site Architecture
For SEO professionals auditing a website, the sitemap is one of the first things they examine. It reveals the full scope of a site's content, exposes URL patterns, highlights content gaps, and shows how frequently different sections are updated. It's essentially a blueprint of the entire site.
Supporting Large and Dynamic Sites
For websites with hundreds of thousands of pages - e-commerce stores, marketplaces, job boards, real estate platforms - sitemaps aren't optional. Without them, search engines would struggle to discover and index the full breadth of content. Dynamic sites that generate pages based on user queries or database entries are especially dependent on sitemaps for comprehensive indexation.
Where to Find a Website's Sitemap
Finding a sitemap isn't always straightforward. While many sites host their sitemap at the standard /sitemap.xml path, others use non-standard locations, compress their sitemaps with gzip, or bury them behind sitemap index files.
Here are the common places to check. The sitemap might be at example.com/sitemap.xml, example.com/sitemap_index.xml, or example.com/sitemap/sitemap-index.xml. You can also check the site's robots.txt file (usually at example.com/robots.txt), which often contains a Sitemap: directive pointing to the sitemap's actual location. Google Search Console also shows the sitemaps that have been submitted for a verified domain.
But if you're trying to retrieve sitemaps programmatically - for data pipelines, SEO audits, competitive analysis, or AI agent workflows - manually checking these locations for every domain doesn't scale. That's where an API comes in.
Discovering Website URLs with Context.dev
If your goal is URL discovery across many domains, Context.dev Map URLs returns an indexed inventory with available page metadata. If you specifically need the publisher's sitemap XML, fetch and parse that sitemap directly.
Choose the source your workflow needs
A live sitemap pipeline must discover the XML files, follow sitemap indexes, decompress responses, parse XML, and handle fetch failures. It preserves publisher-supplied fields such as lastmod when available.
Map URLs queries indexed URL data instead. This removes XML handling from your application when discovery is the goal, but it does not return the publisher's original sitemap or prove complete coverage.
How Context.dev Map URLs Works
Map URLs takes a domain and returns urls objects with a url and optional title, description, keywords, and language. Use maxLinks, urlRegex, and includeSubdomains to control the inventory. It reads an index, so results are not proof of complete live sitemap coverage.
Here's a quick example using curl:
curl -X GET "https://api.context.dev/v1/web/urls?domain=example.com" \
-H "Authorization: Bearer YOUR_API_KEY"Read each entry's url and available page metadata. Map URLs does not return sitemap lastmod or changefreq fields. Inspect response limits and partial markers before using the list as a coverage baseline.
Use Cases for Map URLs
Map URLs is a discovery building block. Here are some of the most common ways developers and companies use it.
SEO Auditing and Monitoring. Compare indexed URL inventories to find pages worth reviewing. Confirm missing URLs against live pages or publisher sitemaps before treating them as removals.
AI Agent Workflows. Use indexed URLs to select pages for research, lead generation, or competitive intelligence. Feed each selected URL into Scrape and retain its source URL with the extracted content.
Competitive Intelligence. Compare URL inventories across competitors to find product, pricing, and content pages for further analysis. An index update is a discovery signal, not a publication timestamp.
Data Pipelines and ETL. Filter indexed URLs for product, blog, or documentation paths before ingestion. Reconcile with publisher sitemaps or bounded crawling when the pipeline needs broader coverage.
Lead Generation and Enrichment. In B2B sales, understanding a company's web presence is valuable context. Review a company's indexed URLs to discover product lines, geographic pages, job openings, and content topics - all from publicly available data.
Programmatic SEO. Compare your expected URL list with publisher sitemaps and search-engine reporting. Map URLs can supply another discovery view, but its index does not establish Google or Bing indexation.
Sitemap Best Practices
Whether you're creating sitemaps for your own website or consuming them from others, these best practices will save you headaches.
Split Large Sitemaps
If your site has a large URL inventory, use a sitemap index file to split URLs across multiple sitemaps. Organize the child sitemaps logically - by content type, category, or language.
Only Include Canonical URLs
Every URL in your sitemap should be the canonical version of that page. Don't include URLs with tracking parameters, session IDs, or query strings that lead to duplicate content. If a page has a rel=canonical tag pointing to a different URL, use the canonical URL in the sitemap.
Keep lastmod Accurate
The <lastmod> tag should reflect when the page's content actually changed - not when the sitemap was regenerated. Search engines use this signal to decide whether to re-crawl a page. If you update <lastmod> on every page every time the sitemap rebuilds, you're diluting the signal and wasting crawl budget.
Submit Your Sitemap to Search Engines
Hosting a sitemap at /sitemap.xml is a good start, but you should also explicitly submit it through Google Search Console and Bing Webmaster Tools. This ensures search engines know about it immediately and gives you access to indexation reporting that shows which URLs were successfully crawled.
Reference the Sitemap in robots.txt
Add a Sitemap: directive to your robots.txt file pointing to your sitemap (or sitemap index). This is how many crawlers discover sitemaps automatically.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlUse HTTPS URLs
All URLs in your sitemap should use HTTPS. If your site has been migrated from HTTP to HTTPS, make sure the sitemap reflects the current protocol. Mixing HTTP and HTTPS URLs creates confusion for search engines and can lead to indexation issues.
Gzip Large Sitemaps
For large sitemaps, compress them with gzip. Search engines support gzipped sitemaps natively, and compression can reduce file size substantially. Just make sure your server returns the correct Content-Encoding: gzip header.
Common Sitemap Mistakes
Even experienced developers and SEO professionals make these mistakes. Avoiding them will keep your site's indexation healthy.
Including pages that return 4xx or 5xx status codes is one of the most common errors. Every URL in your sitemap should return a 200 OK response. Sitemaps that are bloated with broken links erode trust with search engines over time.
Listing non-canonical URLs or pages blocked by robots.txt sends contradictory signals to crawlers. If you're telling Google not to crawl a page in robots.txt but listing it in your sitemap, you're creating confusion. These signals should be aligned.
Failing to update the sitemap when content changes is another frequent issue. A sitemap that never changes tells search engines that your site is static, which can reduce crawl frequency over time. If you're publishing new content regularly, your sitemap should reflect that.
Forgetting to handle pagination properly affects large sites with paginated content. If your blog has 200 pages of archived posts, each paginated page should either be in the sitemap or the sitemap should contain the individual post URLs directly - not both.
Sitemaps and the Future of Web Data
Sitemaps have been around since 2006, but their relevance is growing - not shrinking. As AI agents, LLM-powered tools, and automated data pipelines become more prevalent, sitemaps are becoming the de facto discovery layer for programmatic access to web content.
Traditional search engine crawling was designed for a world where Google was the primary consumer of web data. Today, the consumers are much more diverse: AI research agents, competitive intelligence platforms, e-commerce aggregators, data enrichment services, and developer tools that need structured access to the full breadth of a website's content.
This shift is exactly why APIs like Context.dev exist. The sitemap is the starting point - the index of everything a website has to offer. But extracting, parsing, and normalizing sitemaps at scale requires infrastructure that most teams don't want to build and maintain themselves. Context.dev provides that infrastructure as a clean, reliable API, so developers can focus on what they're building rather than the plumbing underneath.
Whether you're an SEO professional running audits, a developer building data pipelines, or a founder building the next AI-powered product, understanding sitemaps - and having programmatic access to them - is a fundamental building block. Context.dev Map URLs gives your pipeline an indexed inventory to use alongside the sitemap when planning page retrieval.
Conclusion
A sitemap is one of the simplest yet most important files on any website. It's the bridge between a website's content and the systems that need to discover, index, and process that content - whether those systems are search engines, AI agents, or data pipelines.
For website owners, maintaining a clean, accurate, and well-structured sitemap is table stakes for SEO. For developers and data teams, sitemaps are the natural entry point for any workflow that needs a complete inventory of a website's pages.
And for anyone who needs to retrieve sitemaps programmatically - across many domains, at scale, without worrying about edge cases - Context.dev provides the API infrastructure to do it with a single call. No XML parsing, no sitemap hunting, no infrastructure overhead. Just clean, structured data, ready to power whatever you're building next.