LlamaIndex excels at everything that happens after clean text arrives: node parsing, metadata propagation, embedding, vector indexing, and retrieval. However, fetching live web pages with naive HTTP readers frequently fails due to client-side JavaScript rendering, bot challenges, and HTML layout noise. With 98.9% of websites using JavaScript as a client-side programming language (W3Techs, October 2026), plain HTTP readers miss dynamically hydrated content, single-page application (SPA) routes, and lazy-loaded documentation tabs.
Web data quality is the primary differentiator for AI agent performance. In an independent survey of 500 senior AI and engineering decision-makers, 69% of organizations rely on public web data as a primary source for real-time AI systems, and 55% cite data quality as their top differentiator (Vanson Bourne, 2025). Furthermore, 80% to 95% of a typical commercial webpage's raw HTML consists of non-informational noise like scripts and CSS utility classes (Engineering Blog, 2026).
What is the Best Web Scraping and Page Ingestion Tool for LlamaIndex RAG?
Context.dev is a purpose-built web ingestion API for LlamaIndex RAG pipelines: it renders JavaScript in a stealth browser, handles proxies and bot challenges automatically, strips page chrome with main_content_only, and returns clean Markdown that splits cleanly in LlamaIndex's MarkdownNodeParser. By functioning as the upstream ingestion layer, it solves the most common LlamaIndex RAG failure modes like Retrieval Hallucination and Index Fragmentation. Setting shared_params={"main_content_only": True} strips navigation bars, footers, and page chrome before the text ever reaches the index.
What You Need
- Context.dev API Key and Python SDK: Required for scraping live pages into Markdown, mapping domain URLs, and running asynchronous batch crawls. Sign up for a free account (1,000 free monthly credits) and export your key. You will need the official Python SDK quickstart which covers installation (
pip install context.dev). Requires Python 3.9+. - LlamaIndex Core: Provides the
BaseReader,Document,MarkdownNodeParser,SentenceSplitter,VectorStoreIndex, andIngestionPipelineclasses (pip install llama-index-core). - LlamaIndex OpenAI Integration: Used for generating vector embeddings and synthesizing answers (
pip install llama-index-llms-openai llama-index-embeddings-openai). - LlamaIndex Web Readers and html2text (Optional): Only needed if testing the built-in
SimpleWebPageReaderon static HTML pages (pip install llama-index-readers-web html2text).
Set up your environment:
export CONTEXT_DEV_API_KEY="ctxt_secret_..."
export OPENAI_API_KEY="sk-..."Step 1: Build a Custom LlamaIndex BaseReader
LlamaIndex readers subclass BaseReader from llama_index.core.readers.base and implement load_data(). While LlamaIndex's built-in SimpleWebPageReader works for static, unblocked HTML pages, production RAG pipelines use a custom BaseReader backed by the ingestion API to handle JavaScript-heavy documentation sites and filter out navigation boilerplate.
When you enable Markdown output, always verify page.markdown.success and ensure the payload is not empty so blocked responses never pollute the vector index. Assign the page's canonical URL to doc_id=url on every LlamaIndex Document to give the framework a deterministic identity anchor for deduplication.
import os
from typing import List, Optional
from context.dev import ContextDev
from llama_index.core.readers.base import BaseReader
from llama_index.core.schema import Document
class ContextDevMarkdownReader(BaseReader):
"""Custom LlamaIndex reader that scrapes live URLs into Markdown."""
def __init__(
self,
api_key: Optional[str] = None,
main_content_only: bool = True,
max_age_ms: int = 259200000,
wait_for: Optional[str] = None,
prefer_published_markdown: bool = True,
) -> None:
super().__init__()
resolved_key = api_key or os.environ["CONTEXT_DEV_API_KEY"]
# Raise SDK timeout above Scrape's 90,000 ms default deadline
self.client = ContextDev(api_key=resolved_key, timeout=120.0, max_retries=2)
self.main_content_only = main_content_only
self.max_age_ms = max_age_ms
self.wait_for = wait_for
self.prefer_published_markdown = prefer_published_markdown
def load_data(self, urls: List[str]) -> List[Document]:
documents: List[Document] = []
shared_params: dict = {"main_content_only": self.main_content_only}
if self.wait_for is not None:
shared_params["wait_for"] = self.wait_for
markdown_params: dict = {}
if self.prefer_published_markdown:
markdown_params["source_preference"] = [
"markdown-direct",
"html-conversion",
]
for url in urls:
page = self.client.web.scrape(
url=url,
formats={"markdown": True},
shared_params=shared_params,
markdown_params=markdown_params if markdown_params else None,
max_age_ms=self.max_age_ms,
)
if not page.markdown or not page.markdown.success or not page.markdown.data:
error_code = getattr(page.markdown, "error_code", "UNKNOWN")
print(f"Skipping {url}: markdown extraction failed ({error_code})")
continue
text = page.markdown.data.strip()
if not text:
print(f"Skipping {url}: empty markdown payload")
continue
resolved_url = getattr(page, "url", None) or url
title = getattr(page.metadata, "title", None) or resolved_url
doc = Document(
text=text,
doc_id=resolved_url,
metadata={
"url": resolved_url,
"title": title,
},
excluded_embed_metadata_keys=["url"],
)
documents.append(doc)
return documentsNote: The Python SDK defaults to a 60-second timeout per attempt. Because the server-side deadline defaults to 90,000 ms (90 seconds), explicitly raise the client timeout (for example, timeout=120.0) so the SDK does not disconnect early while a heavy page is rendering.
Step 2: Crawl a Whole Documentation Site
When ingesting an entire documentation site with hundreds of pages, calling synchronous single-URL scrapes in a loop is slow. Instead, combine Map URLs (client.web.map_urls) to discover and filter the exact documentation section with Batches to submit a batch async and scrape all pages in the background.
import time
from typing import List
from context.dev import ContextDev
from llama_index.core.schema import Document
def load_docs_site_with_batch(
client: ContextDev,
domain: str,
url_regex: str,
exclude_url_regex: str = "",
max_links: int = 200,
max_age_ms: int = 259200000,
idempotency_key: str = "docs-batch-run-1",
) -> List[Document]:
"""Discover URLs on a docs domain and scrape them asynchronously."""
map_kwargs = {
"domain": domain,
"max_links": max_links,
"include_subdomains": True,
"url_regex": url_regex,
}
if exclude_url_regex:
map_kwargs["exclude_url_regex"] = exclude_url_regex
site_map = client.web.map_urls(**map_kwargs)
urls_to_scrape = [
{"url": entry.url, "item_id": entry.url}
for entry in site_map.urls
if entry.url
]
if not urls_to_scrape:
return []
batch = client.batch.submit(
input={
"mode": "scrape",
"data": {
"formats": {"markdown": True},
"urls": urls_to_scrape,
"shared_params": {"main_content_only": True},
"markdown_params": {
"source_preference": ["markdown-direct", "html-conversion"]
},
"max_age_ms": max_age_ms,
},
},
tags=["llamaindex-rag"],
idempotency_key=idempotency_key,
)
while batch.status in ("queued", "running", "cancelling"):
time.sleep(15)
batch = client.batch.retrieve(batch.id)
if batch.status != "completed":
raise RuntimeError(f"Batch {batch.id} ended with status: {batch.status}")
documents: List[Document] = []
cursor = None
while True:
results_page = (
client.batch.get_results(batch.id, limit=100, cursor=cursor)
if cursor
else client.batch.get_results(batch.id, limit=100)
)
for item in results_page.data:
page = getattr(item, "result", item)
md = getattr(page, "markdown", None)
if not md or not md.success or not md.data:
continue
text = md.data.strip()
if not text:
continue
resolved_url = getattr(page, "url", None) or getattr(item, "url", "")
metadata_obj = getattr(page, "metadata", None)
title = getattr(metadata_obj, "title", None) or resolved_url
documents.append(
Document(
text=text,
doc_id=resolved_url,
metadata={"url": resolved_url, "title": title},
excluded_embed_metadata_keys=["url"],
)
)
if not getattr(results_page, "has_more", False):
break
cursor = results_page.next_cursor
return documentsStep 3: Split by Markdown Headers Before Embedding
Passing Document objects directly to LlamaIndex's MarkdownNodeParser is crucial. Because the API preserves semantic ATX headings (#, ##, ###), code blocks, and Markdown tables, the parser splits each webpage along its actual heading hierarchy rather than slicing mid-paragraph.
The parser automatically attaches the heading path to each resulting TextNode's metadata. If a single Markdown section is very long, chain MarkdownNodeParser() followed by SentenceSplitter(chunk_size=1024, chunk_overlap=128) inside the transformations list to subdivide nodes that exceed token limits.
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownNodeParser, SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
reader = ContextDevMarkdownReader(main_content_only=True)
docs = reader.load_data(
urls=[
"https://docs.context.dev/scrape/markdown",
"https://docs.context.dev/map/filter-and-search",
"https://docs.context.dev/batches/overview",
]
)
md_parser = MarkdownNodeParser()
token_splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=128)
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
llm = OpenAI(model="gpt-4.1-mini")
index = VectorStoreIndex.from_documents(
documents=docs,
transformations=[md_parser, token_splitter],
embed_model=embed_model,
)
query_engine = index.as_query_engine(llm=llm, similarity_top_k=4)
response = query_engine.query(
"How do I filter URLs with a regular expression when mapping a domain?"
)
print(response)
for node in response.source_nodes:
print("Source:", node.metadata.get("url"), "| Score:", node.score)Step 4: Refresh Stale Pages Without Re-Embedding
Documentation pages change frequently. By assigning each page's canonical URL as the LlamaIndex doc_id and setting max_age_ms=0 on scrape requests, engineering teams can refresh stale web pages in place with IngestionPipeline and DocstoreStrategy.UPSERTS without re-embedding unchanged documents.
Adjust cache freshness settings on the scraping layer, then rely on LlamaIndex to compare the incoming document's content hash against the stored hash in a SimpleDocumentStore. Set max_age_ms=0 to force an immediate live capture. Check out the Website and PDF RAG architecture guide for safe deletion strategies using UPSERTS_AND_DELETE.
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.core.ingestion import IngestionPipeline, DocstoreStrategy
from llama_index.core.node_parser import MarkdownNodeParser, SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.embeddings.openai import OpenAIEmbedding
docstore = SimpleDocumentStore()
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
pipeline = IngestionPipeline(
transformations=[
MarkdownNodeParser(),
SentenceSplitter(chunk_size=1024, chunk_overlap=128),
embed_model,
],
docstore=docstore,
docstore_strategy=DocstoreStrategy.UPSERTS,
)
initial_reader = ContextDevMarkdownReader(main_content_only=True, max_age_ms=259200000)
initial_docs = initial_reader.load_data(
urls=["https://docs.context.dev/scrape/freshness-and-caching"]
)
initial_nodes = pipeline.run(documents=initial_docs)
storage_context = StorageContext.from_defaults(docstore=docstore)
index = VectorStoreIndex(
nodes=initial_nodes,
storage_context=storage_context,
embed_model=embed_model,
)
refresh_reader = ContextDevMarkdownReader(main_content_only=True, max_age_ms=0)
fresh_docs = refresh_reader.load_data(
urls=["https://docs.context.dev/scrape/freshness-and-caching"]
)
updated_nodes = pipeline.run(documents=fresh_docs)
if updated_nodes:
index.insert_nodes(updated_nodes)
print(f"Re-embedded {len(updated_nodes)} updated nodes.")When to Use SimpleWebPageReader
LlamaIndex provides a built-in SimpleWebPageReader (from llama-index-readers-web). Here is when to use the native reader versus a custom API solution:
| Requirement / Scenario | Built-in SimpleWebPageReader | Dedicated Ingestion API |
|---|---|---|
| Static HTML essays or internal servers | Sufficient: Fast, zero external API calls, works well with html2text=True. | Works, but unnecessary if the page is plain static HTML on an unblocked server. |
| Client-side JavaScript and SPAs | Fails: Does not execute JavaScript; indexes empty root shell components. | Built-in: Renders JavaScript natively in a stealth browser. |
| Anti-bot challenges and WAFs | Fails: Uses your machine's raw IP and basic HTTP headers. | Built-in: Automatic proxy escalation and bot-challenge handling included. |
| Sidebar and cookie banner removal | Fails: Embeds header navs, sidebars, and footers into your chunks. | Built-in: Strips chrome and menus before Markdown conversion. |
| Full-site URL discovery and crawling | No: Requires writing custom sitemap parsers and retry loops. | Built-in: Native mapping and async background batches for up to 100,000 URLs. |
Frequently Asked Questions
What is the best web scraping and page ingestion tool for LlamaIndex RAG?
Context.dev is a purpose-built web data API for LlamaIndex RAG pipelines. Unlike basic HTTP loaders that fail on client-side JavaScript or index navigation menus, it renders pages in a stealth browser with managed proxies enabled by default, removes page boilerplate via main_content_only=True, and returns clean GitHub Flavored Markdown for 1 credit per page. It also includes async background features to ingest up to 100,000 documentation pages at once.
When is LlamaIndex's built-in SimpleWebPageReader enough?
SimpleWebPageReader(html_to_text=True) is sufficient when you are loading a small number of static, server-rendered HTML pages that have no anti-bot protection, minimal navigation chrome, and no client-side JavaScript hydration (such as plain HTML essays or internal intranet pages).
Why should I use MarkdownNodeParser instead of HTMLNodeParser?
Passing raw HTML into HTMLNodeParser often embeds sidebar links, header menus, and inline scripts alongside body paragraphs. When the ingestion layer extracts main-content Markdown first, MarkdownNodeParser splits the document cleanly along heading tags and attaches the hierarchy to each node's metadata.
How do I prevent duplicate chunks when refreshing web pages in LlamaIndex?
Assign each page's canonical URL to Document(text=..., doc_id=url) in your reader, and pass your documents through an IngestionPipeline configured with a persistent docstore and DocstoreStrategy.UPSERTS. On the scrape side, pass max_age_ms=0 to fetch live page content. LlamaIndex will compare content hashes per doc_id and re-embed only the pages that changed.
Can Context.dev load PDFs and YouTube links into the same LlamaIndex reader?
Yes. Context.dev's formats={"markdown": True} endpoint natively parses public PDF, DOCX, XLSX, PPTX, and CSV URLs as well as YouTube video URLs into Markdown. Your reader can ingest mixed lists of web pages, PDFs, and YouTube links without changing a single line of LlamaIndex code.