What is a headless browser?
A real browser engine running without a visible UI, controlled programmatically through an automation API.
A headless browser is Chrome, Firefox, or WebKit running in a server process with no window. It executes JavaScript, builds the DOM, makes network requests, and fires events the same way a normal browser does, you just drive it with code (Playwright, Puppeteer, Selenium) instead of a mouse.
For scraping, headless browsers solve the JavaScript problem. Single-page apps and most modern sites render their important content client-side, which means a plain HTTP fetch returns near-empty HTML. A headless browser waits for the JS to run, then dumps the populated DOM.
The trade-off is cost. A headless browser session uses 50-200x the memory and CPU of a static fetch. Production scraping infrastructure typically tries the static path first and only escalates to a headless browser when the response is missing the data it needs.
In the wild
- Scraping a Next.js or React app whose content only appears after hydration
- Taking full-page screenshots that match what a real user sees
- Automating end-to-end tests against a deployed app