Google Custom Search JSON API Alternative: Migration Guide
Replace Google Custom Search JSON API before January 1, 2027. Get web results, Markdown, or relevant passages with Context.dev, plus a practical migration checklist.
Developer Deep Dive articles.
Every Context.dev article filed under Developer Deep Dive.
Claude Code ROBOTS_DISALLOWED: Why WebFetch Cannot Read Some Links
WebFetch returns ROBOTS_DISALLOWED when a site refuses automated access. Learn how to tell it apart from permission denials and preflight failures, and how to finish your research with a primary source you can cite.
How to Scrape Load-More Buttons and Infinite Scroll Pages
Check for real pagination or a data endpoint first, then use bounded click, wait, and scroll actions. Deduplicate by stable ID and mark results incomplete when a cap stops the run.
Playwright MCP browser_evaluate Hangs? The CSP Sandbox Bug Explained
Playwright MCP tools like browser_evaluate and browser_click can stop returning on pages whose timers never run, such as raw.githubusercontent.com files served with a CSP sandbox. We reproduced it on @playwright/mcp 0.0.84, explain the cause, and list what works today.
How to Build a Web Scraping Tool for AI Agents That Cites Its Sources
Build an on-demand scrape tool for AI agents that validates URLs, returns Markdown and JSON with a source URL and fetch time, and fails explicitly on stale pages.
What Is a Screen Scraper? Extensions vs. Python vs. APIs
Screen scraping extracts content after a browser renders the page. Compare browser extensions, Python and Playwright, and managed APIs for dynamic sites.
Lists Crawler: Python List Crawling Guide (2026)
Build a lists crawler in Python with Beautiful Soup, Scrapy, and Context.dev. Handle pagination, JavaScript, deduplication, and validated JSON output.
YouTube Transcript API: Get Any Video Transcript as Markdown
Get YouTube transcripts with one API call. Context.dev detects video URLs and returns timestamped transcripts, metadata, and descriptions as Markdown, with no proxies to manage.
Introducing Rusteerio
Why we ported Cheerio to Rust, how LLMs made it worth doing, and what changed in our HTML processing benchmarks.
Website to JSON: Extract Structured Data from Live Pages
Extract structured JSON from live websites with Context.dev Scrape. Learn how rendering, JSON Schema, missing fields, validation, and batch workflows affect the output.
MCP Server vs REST API for Web Scraping: When to Use Each
Compare MCP servers and REST APIs for web scraping across setup, tool discovery, authentication, observability, batching, retries, latency, and portability, plus architecture examples for agent, pipeline, and hybrid deployments.
Browser Fingerprinting Explained: Headless Browsers and Anti-Detect Profiles
Learn how browser fingerprinting exposes automation, what anti-detect profiles solve, and when a managed scraping API fits your workflow.
Start building with live web context.
Scrape, search, and research the web with one API. Bring the results straight into your product.