How to Scrape Load-More Buttons and Infinite Scroll Pages
Check for real pagination or a data endpoint first, then use bounded click, wait, and scroll actions. Deduplicate by stable ID and mark results incomplete when a cap stops the run.
All articles.
The complete archive of stories, guides, and updates from Context.dev.
Playwright MCP browser_evaluate Hangs? The CSP Sandbox Bug Explained
Playwright MCP tools like browser_evaluate and browser_click can stop returning on pages whose timers never run, such as raw.githubusercontent.com files served with a CSP sandbox. We reproduced it on @playwright/mcp 0.0.84, explain the cause, and list what works today.
Best Web Search MCP Servers for Cursor and Claude in 2026
Compare Context.dev, Firecrawl, Exa, and Brave Search as web search MCP servers for Cursor and Claude: search, full-page reads, JavaScript rendering, schema-shaped JSON, auth, and setup.
Google /goto Links Broke SERP APIs Again on October 30, 2026: What Happened
On October 30, 2026, Google tightened its google.com/goto redirects and SERP APIs started returning wrapped links again. What changed, the full /goto timeline since July, and how to keep search results reliable.
How to Load Web Pages into LlamaIndex with Context.dev for RAG
Learn how to load web pages into LlamaIndex with Context.dev for RAG pipelines. Get clean markdown data, bypass bot challenges, and optimize your AI performance.
Introducing web search monitors
Watch the whole web for new developments on any topic. Web search monitors run your search on a schedule and alert you only when something new and relevant shows up. Set one up in a sentence through the Context MCP.
Web Scraping API Benchmark for AI Agents: Context.dev, Firecrawl, Apify, Zyte, ScrapingBee
How we will benchmark Context.dev, Firecrawl, Apify, Zyte, and ScrapingBee on static, JavaScript-rendered, and blocked pages, plus what two independent scraping benchmarks already report.
Best Company Enrichment APIs in 2026: Website Context vs Firmographic Data
Context.dev, Apollo, People Data Labs, and Crustdata compared on data source, fields, freshness, and pricing, plus a test plan for your own domains.
How to Build a Web Scraping Tool for AI Agents That Cites Its Sources
Build an on-demand scrape tool for AI agents that validates URLs, returns Markdown and JSON with a source URL and fetch time, and fails explicitly on stale pages.
How to Give an LLM Web Browsing with Function Calling: OpenAI, Anthropic and MCP Tool Definitions
Learn how to give your AI agents live web browsing capabilities using function calling and the Context.dev API. Optimize your LLM workflows with clean markdown.
What Is a Screen Scraper? Extensions vs. Python vs. APIs
Screen scraping extracts content after a browser renders the page. Compare browser extensions, Python and Playwright, and managed APIs for dynamic sites.
How to Scrape Websites into a Vector Database for RAG (Pinecone, pgvector, Qdrant)
Build a robust RAG pipeline by scraping clean Markdown from live websites. This guide shows you how to integrate Context.dev with Pinecone, pgvector, and Qdrant.
Start building with live web context.
Scrape, search, and research the web with one API. Bring the results straight into your product.