Web Scraping & Crawling
What is a headless browser?
A real browser engine running without a visible UI, controlled programmatically through an automation API.
A headless browser is Chrome, Firefox, or WebKit running in a server process with no window. It executes JavaScript, builds the DOM, makes network requests, and fires events the same way a normal browser does, you just drive it with code (Playwright, Puppeteer, Selenium) instead of a mouse.
For scraping, headless browsers solve the JavaScript problem. Single-page apps and most modern sites render their important content client-side, which means a plain HTTP fetch returns near-empty HTML. A headless browser waits for the JS to run, then dumps the populated DOM.
The trade-off is cost. A headless browser session uses 50-200x the memory and CPU of a static fetch. Production scraping infrastructure typically tries the static path first and only escalates to a headless browser when the response is missing the data it needs.
In the wild
- →Scraping a Next.js or React app whose content only appears after hydration
- →Taking full-page screenshots that match what a real user sees
- →Automating end-to-end tests against a deployed app
How Brand.dev uses headless browser
Endpoints in the Brand.dev API where this concept comes up directly.
FAQ
Headless Chrome vs Puppeteer vs Playwright?
Headless Chrome is the browser. Puppeteer and Playwright are libraries that drive it. Playwright also drives Firefox and WebKit and has better cross-browser ergonomics; Puppeteer is Chrome-first and lighter.
Is a headless browser detectable?
Yes, sites can probe navigator.webdriver, missing plugins, and timing fingerprints. Stealth plugins help, but evasion is an arms race.
When should I avoid using a headless browser?
Whenever the page is server-rendered. A static HTTP fetch is faster, cheaper, more reliable, and easier to scale.
Related terms
Programmatically extracting structured data from websites that were designed to be read by humans.
A Microsoft-maintained library for driving Chrome, Firefox, and WebKit headlessly with a unified API.
The Document Object Model, a tree of objects that represents an HTML document in memory and lets JavaScript manipulate it.
A program that systematically follows links between web pages to discover and index content at scale.