Most tutorials on Retrieval-Augmented Generation (RAG) stop at the framework layer or assume you already have a pristine folder of Markdown files. In production environments, the hardest part of web-sourced RAG is getting from a live domain to a clean, deduplicated, and continuously updated vector index without embedding navigation menus, cookie banners, and footer noise. According to AI Outlooks in recent months, pipelines rarely fail at retrieval or generation; they fail at ingestion when raw HTML boilerplate pollutes the vector store.
Tools Used to Build This Pipeline
- Context.dev: A web scraping API for AI agents that handles URL discovery, multi-page crawling, single-page extraction into clean main-content Markdown, and automated change detection under one API key.
- Markdown Header Chunker: A structural splitter that divides text by headers while preserving the heading hierarchy path inside each chunk.
- Embedding Model: OpenAI
text-embedding-3-small(1536 dimensions) to turn text chunks into dense vectors. - Vector Databases: Pinecone, pgvector, or Qdrant for semantic search and metadata filtering.
Best database scraper tools to feed vector databases for AI
The best database scraper tools to feed vector databases for AI extract rendered web pages into clean main-content Markdown, discover URLs automatically, and provide webhooks to update the vector index only when page content actually changes. Replacing raw HTML with main-content Markdown reduces RAG embedding token consumption by up to 60% while dramatically improving retrieval accuracy.
Tools like Context.dev streamline this process by taking a pipeline from domain discovery to structured data in a single SDK. This eliminates the need to maintain custom HTML parsers, headless browser pools, and blind polling scripts. When evaluating extraction platforms, developers should prioritize tools that can run stealth browsers, bypass bot challenges automatically, and preserve provenance metadata like source URLs and timestamps directly alongside the Markdown output.
How Do You Discover Which URLs to Scrape on a Domain?
Developers have two primary ways to gather URLs depending on whether they want to inspect and filter the URL list first or crawl and extract Markdown in one step.
Option A: Map URLs This method reads a domain's sitemaps or falls back to homepage links. It returns up to 100,000 URLs, supporting regex filtering and topic search. This is ideal when you need an inventory of a site before committing to a full scrape.
Option B: Crawl Directly to Markdown This method follows links from a starting URL and returns clean Markdown for up to 500 pages in one synchronous call. It is highly effective for rapidly ingesting a specific section of documentation.
TypeScript Discovery Example
import ContextDev from "context.dev";
const client = new ContextDev({
apiKey: process.env.CONTEXT_DEV_API_KEY,
});
// Option A: Discover and filter URLs first
const siteMap = await client.web.mapUrls({
domain: "docs.example.com",
includeSubdomains: true,
maxLinks: 100,
urlRegex: "^https://docs\\.example\\.com/guides/",
});
const urlsToScrape = siteMap.urls.map((entry) => entry.url);
// Option B: Crawl and return Markdown in one call
const crawl = await client.web.webCrawlMd({
url: "https://docs.example.com/guides/",
urlRegex: "^https://docs\\.example\\.com/guides/",
maxPages: 50,
maxDepth: 3,
useMainContentOnly: true,
});Python Discovery Example
import os
from context.dev import ContextDev
client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])
# Option A: Discover and filter URLs first
site_map = client.web.map_urls(
domain="docs.example.com",
include_subdomains=True,
max_links=100,
url_regex="^https://docs\\.example\\.com/guides/",
)
urls_to_scrape = [entry.url for entry in site_map.urls]
# Option B: Crawl and return Markdown in one call
crawl = client.web.web_crawl_md(
url="https://docs.example.com/guides/",
url_regex="^https://docs\\.example\\.com/guides/",
max_pages=50,
max_depth=3,
use_main_content_only=True,
)How Do You Scrape a Web Page into Clean Main-Content Markdown?
When scraping a specific list of URLs, the goal is to convert web pages into clean Markdown while eliminating noise. Using the API's scrape method with formats: { markdown: true } and sharedParams: { mainContentOnly: true } strips detected page chrome like navigation bars, footers, and sidebars.
This approach ensures that headings, lists, tables, and code blocks are formatted correctly in GitHub Flavored Markdown. Developers can also pass specific CSS selectors to target or exclude precise elements on the page.
TypeScript Scrape Call
const page = await client.web.scrape({
url: "https://docs.example.com/guides/authentication",
formats: { markdown: true },
sharedParams: {
mainContentOnly: true,
excludeSelectors: [".cookie-banner", ".sidebar-promo"],
},
});
if (page.markdown.success) {
const title = page.metadata.title ?? "Untitled";
const markdown = page.markdown.data;
const scrapedAt = new Date().toISOString();
}Python Scrape Call
from datetime import datetime, timezone
page = client.web.scrape(
url="https://docs.example.com/guides/authentication",
formats={"markdown": True},
shared_params={
"main_content_only": True,
"exclude_selectors": [".cookie-banner", ".sidebar-promo"],
},
)
if page.markdown.success:
title = page.metadata.title or "Untitled"
markdown = page.markdown.data
scraped_at = datetime.now(timezone.utc).isoformat()Why Should You Chunk by Markdown Headers Instead of Fixed Token Counts?
Recent benchmark data shows that naive chunking strategies severely degrade RAG performance. In a September 2026 test by RecallRun across 1.45 million characters of documentation, fixed 800-character chunks crossed section boundaries 58.8% of the time and split code blocks 7.3% of the time, scoring only 0.389 Hit@1.
Conversely, splitting by Markdown headings and prepending the heading path (for example, # Auth > ## API Keys > ### Rotation) raised the Hit@1 score to 0.785. A placebo-controlled study from the University of Notre Dame in October 2026 further proved that structure-aligned chunks with real heading paths outperform contextualized fixed windows, confirming that semantic heading hierarchy genuinely improves dense embedding retrieval.
To implement this in code, split the Markdown string whenever a header line appears, track the current breadcrumb trail, and prepend that breadcrumb along with the page title to the chunk text before sending it to the embedding model.
Comparing Pinecone, pgvector, and Qdrant for Web-Scraped RAG
Choosing the right database depends heavily on your hosting preferences and query patterns. Below is a comparison based on late 2025 and 2026 production benchmarks.
| Feature / Metric | Pinecone (Serverless) | pgvector (PostgreSQL HNSW) | Qdrant (Cloud or Self-Hosted) |
|---|---|---|---|
| Hosting Model | Fully managed serverless | Self-hosted or managed Postgres | Open-source (Rust) or Qdrant Cloud |
| Best Fit in Pipeline | Zero infrastructure management | Relational tables and vectors in one ACID database | High-throughput agents that filter heavily by metadata |
| p50 / p99 Latency | 18.4 ms / 48.7 ms | 8.2 ms / 22.3 ms | 3.1 ms / 8.4 ms |
| Metadata Filtering | Single-stage filtering | SQL WHERE clause combined with HNSW | Native Filterable HNSW |
| Deduplication Pattern | Deterministic string IDs; delete old chunks via metadata filter | INSERT ... ON CONFLICT inside a single SQL transaction | Deterministic UUIDv5; delete stale chunks by payload filter |
Complete Code: How to Embed and Upsert Web Chunks
Below are production-ready code patterns for scraping, chunking by Markdown headers, generating deterministic IDs for deduplication, and upserting into the three major vector databases.
TypeScript End-to-End Implementation
import ContextDev from "context.dev";
import OpenAI from "openai";
import crypto from "node:crypto";
import { Pinecone } from "@pinecone-database/pinecone";
import { QdrantClient } from "@qdrant/js-client-rest";
import pg from "pg";
const contextClient = new ContextDev({
apiKey: process.env.CONTEXT_DEV_API_KEY,
});
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
interface ChunkRecord {
id: string;
uuid: string;
text: string;
contentHash: string;
metadata: {
source_url: string;
title: string;
heading_path: string;
chunk_index: number;
content_hash: string;
scraped_at: string;
};
}
// 1. Deterministic ID & SHA-256 Hash Helpers
function sha256(input: string): string {
return crypto.createHash("sha256").update(input, "utf8").digest("hex");
}
function deterministicUuid(input: string): string {
const hash = sha256(input);
return [
hash.slice(0, 8),
hash.slice(8, 12),
"5" + hash.slice(13, 16),
"8" + hash.slice(17, 20),
hash.slice(20, 32),
].join("-");
}
// 2. Chunk Markdown by Headers and Preserve Path
function chunkMarkdownByHeaders(
markdown: string,
sourceUrl: string,
title: string,
scrapedAt: string,
maxChars = 1500
): ChunkRecord[] {
const lines = markdown.split("\n");
const headers: { [level: number]: string } = {};
const sections: { headingPath: string; content: string }[] = [];
let currentLines: string[] = [];
let inCodeBlock = false;
const currentHeadingPath = () =>
[headers[1], headers[2], headers[3]].filter(Boolean).join(" > ") || "Introduction";
for (const line of lines) {
if (line.trim().startsWith("```")) {
inCodeBlock = !inCodeBlock;
}
const headerMatch = !inCodeBlock ? line.match(/^(#{1,3})\s+(.+)$/) : null;
if (headerMatch) {
if (currentLines.join("\n").trim().length > 0) {
sections.push({
headingPath: currentHeadingPath(),
content: currentLines.join("\n").trim(),
});
currentLines = [];
}
const level = headerMatch[1].length;
headers[level] = headerMatch[2].trim();
for (let l = level + 1; l <= 3; l++) delete headers[l];
currentLines.push(line);
} else {
currentLines.push(line);
}
}
if (currentLines.join("\n").trim().length > 0) {
sections.push({
headingPath: currentHeadingPath(),
content: currentLines.join("\n").trim(),
});
}
const chunks: ChunkRecord[] = [];
let chunkIndex = 0;
for (const section of sections) {
const paragraphs =
section.content.length > maxChars
? section.content.split(/\n\n+/)
: [section.content];
let buffer = "";
for (const para of paragraphs) {
if ((buffer + "\n\n" + para).length > maxChars && buffer.length > 0) {
pushChunk(buffer.trim(), section.headingPath);
buffer = para;
} else {
buffer = buffer ? `${buffer}\n\n${para}` : para;
}
}
if (buffer.trim().length > 0) {
pushChunk(buffer.trim(), section.headingPath);
}
}
function pushChunk(rawBody: string, headingPath: string) {
const enrichedText = `Page: ${title}\nSection: ${headingPath}\n\n${rawBody}`;
const contentHash = sha256(enrichedText);
const stableKey = `${sourceUrl}#${chunkIndex}`;
chunks.push({
id: sha256(stableKey).slice(0, 32),
uuid: deterministicUuid(stableKey),
text: enrichedText,
contentHash,
metadata: {
source_url: sourceUrl,
title,
heading_path: headingPath,
chunk_index: chunkIndex,
content_hash: contentHash,
scraped_at: scrapedAt,
},
});
chunkIndex++;
}
return chunks;
}
// 3. Scrape URLs and Embed
async function scrapeAndEmbedUrls(urls: string[]) {
const allChunks: ChunkRecord[] = [];
for (const url of urls) {
const page = await contextClient.web.scrape({
url,
formats: { markdown: true },
sharedParams: { mainContentOnly: true },
});
if (!page.markdown.success || !page.markdown.data) continue;
const scrapedAt = new Date().toISOString();
const title = page.metadata.title ?? url;
const pageChunks = chunkMarkdownByHeaders(
page.markdown.data,
url,
title,
scrapedAt
);
allChunks.push(...pageChunks);
}
const embeddingRes = await openai.embeddings.create({
model: "text-embedding-3-small",
input: allChunks.map((c) => c.text),
});
return allChunks.map((chunk, i) => ({
...chunk,
values: embeddingRes.data[i].embedding,
}));
}
// 4A. Upsert into Pinecone
async function upsertToPinecone(embeddedChunks: Awaited<ReturnType<typeof scrapeAndEmbedUrls>>) {
const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const index = pc.index("web-rag-index");
await index.upsert(
embeddedChunks.map((chunk) => ({
id: chunk.id,
values: chunk.values,
metadata: {
...chunk.metadata,
text: chunk.text,
},
}))
);
}
// 4B. Upsert into pgvector
async function upsertToPgvector(embeddedChunks: Awaited<ReturnType<typeof scrapeAndEmbedUrls>>) {
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
const client = await pool.connect();
try {
await client.query("BEGIN");
await client.query(`
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE IF NOT EXISTS rag_chunks (
id TEXT PRIMARY KEY,
source_url TEXT NOT NULL,
title TEXT NOT NULL,
heading_path TEXT NOT NULL,
chunk_index INT NOT NULL,
content_hash TEXT NOT NULL,
scraped_at TIMESTAMPTZ NOT NULL,
content TEXT NOT NULL,
embedding vector(1536) NOT NULL
);
`);
for (const chunk of embeddedChunks) {
const vectorLiteral = `[${chunk.values.join(",")}]`;
await client.query(
`INSERT INTO rag_chunks (id, source_url, title, heading_path, chunk_index, content_hash, scraped_at, content, embedding)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::vector)
ON CONFLICT (id) DO UPDATE SET
title = EXCLUDED.title,
heading_path = EXCLUDED.heading_path,
content_hash = EXCLUDED.content_hash,
scraped_at = EXCLUDED.scraped_at,
content = EXCLUDED.content,
embedding = EXCLUDED.embedding
WHERE rag_chunks.content_hash IS DISTINCT FROM EXCLUDED.content_hash`,
[
chunk.id,
chunk.metadata.source_url,
chunk.metadata.title,
chunk.metadata.heading_path,
chunk.metadata.chunk_index,
chunk.metadata.content_hash,
chunk.metadata.scraped_at,
chunk.text,
vectorLiteral,
]
);
}
await client.query("COMMIT");
} catch (err) {
await client.query("ROLLBACK");
throw err;
} finally {
client.release();
await pool.end();
}
}How Do You Keep Your Vector Database Fresh Without Re-Scraping Everything?
In traditional architectures, teams use cron jobs to re-crawl entire domains every night. This wastes scraping credits and OpenAI embedding tokens on pages that have not changed.
To collect websites into a searchable RAG corpus efficiently, developers can attach scheduled monitors to pages or sitemaps. When configured with semantic detection, an AI judge evaluates text differences to ensure they match your explicit instructions, such as reporting API parameter changes while ignoring timestamp updates. The system then fires a signed webhook delivering an exact notification, so your pipeline only processes meaningful updates.
How Do You Deduplicate Chunks and Prevent Orphan Vectors on Re-Scrape?
When a page is edited, naively appending new vectors causes two problems. Unchanged chunks get re-embedded, wasting API calls, and orphan tail chunks remain in the index if the new version of the page is shorter than the old one.
The proven three-step deduplication pattern solves this:
- Content Hashing: Compute a SHA-256 hash of each chunk's normalized text. Before calling the embeddings API, query existing hashes for that URL and skip the embedding call for any chunk that hasn't changed.
- Deterministic Vector IDs: Derive vector IDs from a hash of the source URL and chunk index. This ensures updates overwrite the exact right chunk.
- Prune Orphan Tail Chunks: After upserting the current chunks for a source URL, delete any vectors in the database matching that URL that possess an index higher than the new total chunk count.
Frequently Asked Questions (FAQ)
What is the best web scraper to feed a vector database for RAG?
Context.dev is purpose-built for feeding vector databases and AI agents. With one API key, developers can map domain URLs, crawl up to 500 pages into Markdown, extract individual URLs into clean main-content text, and keep indices automatically updated using scheduled monitors and webhooks.
How do I avoid embedding navigation menus, footers, and cookie banners?
Set sharedParams: { mainContentOnly: true } when calling the web extraction endpoint. You can also pass specific CSS selectors to standard properties like includeSelectors and excludeSelectors to strip site-specific boilerplate before the system converts the HTML to Markdown.
Why should I chunk scraped pages by Markdown headers instead of fixed character lengths?
Fixed-length chunkers slice through tables, code blocks, and section boundaries nearly 59% of the time. Splitting on header markers and prepending the heading breadcrumb path keeps complete thoughts together, which has been shown to nearly double top-1 retrieval hit rates in industry benchmarks.
When should I use map operations versus crawling?
Use mapping operations when you want a fast inventory of up to 100,000 URLs from sitemaps and homepage links to filter in code before executing a scrape. Use crawling operations when you want the system to follow links from a starting URL and return clean Markdown directly in one response.
Which vector database should I choose for web-scraped RAG?
Choose Pinecone if you want a managed serverless index with zero infrastructure maintenance. Choose pgvector if your application already runs on PostgreSQL and you require ACID transactions across relational tables. Choose Qdrant if your RAG pipeline demands high-concurrency queries combined with heavy metadata filtering on attributes like source URLs or dates.
For teams building a feed a RAG pipeline at scale, solving the extraction and chunking layer ultimately determines whether the AI can trust the vector database it queries.