TL;DR
- A screen scraper extracts content from a rendered web page. Rendering JavaScript is the dividing line between screen scraping and static HTML parsing, which reads the initial page response without running scripts.
- Browser extensions, DIY Python tools, and managed APIs offer different ways to extract rendered content. They differ in how much browser control and infrastructure you manage.
- Choose an extension for occasional work in a logged-in session, Python when you need browser control and can maintain the code, or a managed API such as Context.dev when your AI pipeline needs repeatable retrieval and structured output.
What a screen scraper actually is
A screen scraper extracts information from a page after a browser has rendered it. On a JavaScript-heavy product page, the server might return little more than an empty container and a script. The browser runs the script, fetches the product data, and adds product cards to the page's document structure, or DOM. A scraper can then read those cards.
Parsing static HTML stops earlier. A Python script using requests and BeautifulSoup reads the server's initial response, but BeautifulSoup does not execute JavaScript. If the product cards appear only after a script runs, the parser sees the container rather than the products. A page that includes its content in the initial HTML needs no browser rendering.
Browser automation goes further than reading rendered content. With Playwright, Python can open a browser, wait for content, click controls, and keep a session active across steps. An extension can read the page already open in your browser, including content visible after you log in or interact with it. DIY Python can use either static parsing or browser automation. A managed API such as Context.dev handles retrieval and rendering for extraction requests, but it does not replace a browser your code drives through a multi-step workflow.
Browser extensions vs. DIY Python vs. managed APIs
Choose a browser extension for occasional extraction from a logged-in tab, DIY Python for repeatable work that needs full browser control, and Context.dev for rendered content in an AI pipeline without maintaining browser infrastructure. The main distinction is who controls the browser session. An extension uses the browser you have open, Playwright lets your code drive one, and a managed API handles rendering per request.
| Decision point | Browser extension | DIY Python | Managed API |
|---|---|---|---|
| Setup effort | Install an extension and select page elements | Install libraries, browser binaries, and your extraction code | Get credentials and make API requests |
| JS rendering | Reads the active rendered tab, depending on the extension | Playwright runs page scripts. BeautifulSoup alone does not | Returns rendered content without a browser you operate |
| Repeatability and scheduling | Usually manual unless you add automation | You write and schedule the job | You call the API from a scheduled job |
| Scaling to many pages | Human interaction limits throughput | You manage browser concurrency and retries | The provider manages retrieval infrastructure, subject to API limits |
| Output format | Often CSV or JSON, depending on the extension | Whatever your code produces | Markdown or structured JSON, depending on the request |
| Degree of control | Direct interaction with your logged-in tab | Full control over browser actions and extraction logic | Control over requests, waits, and a few scripted actions, not step-by-step browser control |
| Best-fit scenario | One-off work or a session you already use | Custom, repeatable flows you can maintain | Production retrieval for AI data pipelines |
When a browser extension is the right call
A browser extension fits occasional extraction when you already have the target page open in a logged-in session. You can apply a filter, wait for a JavaScript-loaded table to appear, and select the rows you need. For a small task, inspecting the rendered page and exporting those rows may take less effort than writing and maintaining a script.
A point-and-click workflow becomes a poor fit when an AI pipeline needs fresh data without human intervention. Someone must open each page, run the extraction, and export the results, so the workflow has no unattended schedule or concurrent collection. You must also map those exports into the JSON fields your pipeline expects. If the task requires recurring logged-in interactions, Playwright can automate a persistent browser session while giving you control over clicks, waits, and cookies. If you only need to read pages behind a username-and-password login on a schedule, a Context.dev browser profile signs in once and reuses the saved session.
Building a dynamic-page scraper in Python
Install Playwright and its browser binaries before running the example.
pip install playwright
playwright install chromiumThe script below creates a local product page whose JavaScript adds a card after the page loads. Playwright waits for the rendered card, reads its fields, and prints JSON that a downstream pipeline can consume.
import json
from pathlib import Path
from playwright.sync_api import sync_playwright
html = """
<html>
<body>
<div id="products"></div>
<script>
const products = [{ title: "Wireless Mouse", price: "$24.99" }];
setTimeout(() => {
document.getElementById("products").innerHTML = products
.map((p) => `<div class="product-card"><h2>${p.title}</h2><span class="price">${p.price}</span></div>`)
.join("");
}, 500);
</script>
</body>
</html>
"""
fixture = Path("products.html")
fixture.write_text(html, encoding="utf-8")
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=True)
try:
page = browser.new_page()
page.goto(fixture.resolve().as_uri())
page.wait_for_selector(".product-card")
products = []
cards = page.locator(".product-card")
for index in range(cards.count()):
card = cards.nth(index)
products.append({
"title": card.locator("h2").inner_text().strip(),
"price": card.locator(".price").inner_text().strip(),
})
print(json.dumps(products))
finally:
browser.close()The script prints [{"title": "Wireless Mouse", "price": "$24.99"}]. A plain HTTP fetch would receive the page's initial HTML, which contains an empty #products element. Chromium executes the script, and wait_for_selector keeps extraction from starting before the product card appears. For a real site, replace the local file URL with the product-page URL and adjust the selectors to match its rendered markup.
Before running a browser for every page, check whether the site loads the needed data through a JSON request. If you can use that endpoint, requesting its response directly avoids rendering the page. When rendering is necessary, Playwright gives you control over waits and page interactions, but running concurrent browsers consumes CPU and memory. A managed retrieval API becomes worth considering when browser capacity and maintenance start to dominate the extraction work.
What DIY Python scraping costs you at scale
A Playwright scraper can handle a dynamic page, but running browsers across many sites puts pressure on memory and concurrency. Each headless Chromium instance commonly needs a few hundred megabytes of RAM, and heavy pages push that higher. If you schedule hundreds of pages, you need to limit simultaneous browsers, queue jobs, and decide what happens when a page times out.
You also need to maintain the retrieval layer. Playwright gives you control over page behavior, while BeautifulSoup can parse the HTML you retrieve, but neither runs your schedule or manages retries and proxy rotation for you. Site changes add another recurring task. A selector that worked yesterday may return an empty field after a page redesign, so you need checks that catch missing data before it reaches your pipeline.
Consistent output takes work even when every request succeeds. One site may expose product data in JSON-LD, while another requires a site-specific selector. You can try those sources in order, then validate the extracted values against a common schema before sending them to an LLM. Keeping that code and its browser infrastructure in-house gives you control. The next question is whether you need that control enough to maintain it.
Extracting structured data with a managed API
For a JavaScript-heavy page, Context.dev can retrieve rendered content with one Python call. The Python SDK calls client.web.scrape, which loads the page in a browser and returns the formats you request. You do not need to run a Playwright browser in your application for this request.
pip install context.devimport os
from context.dev import ContextDev
client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])
page = client.web.scrape(
url="https://webscraper.io/test-sites/e-commerce/ajax/computers/laptops",
formats={"html": True, "markdown": True},
shared_params={"main_content_only": True},
)
if not page.markdown.success or page.markdown.data is None:
raise RuntimeError("Markdown extraction failed")
print(page.markdown.data)The URL points to a practice store whose product cards are built by JavaScript. When we ran this on October 26, 2026, the Markdown listed each laptop's price, name, description, and review count. Swap in the dynamic page you tested in Playwright. page.markdown.data holds text you can feed into an LLM retrieval or summarization step, while page.html.data holds the rendered page for your own parser. The main_content_only setting removes navigation and other boilerplate. Each format reports its own success flag, so check the one your pipeline consumes.
For schema-shaped JSON, request the json format and pass a JSON Schema in json_params. An AI model reads the rendered page and returns an object that follows your schema:
page = client.web.scrape(
url="https://webscraper.io/test-sites/e-commerce/ajax/computers/laptops",
formats={"json": True},
json_params={
"schema": {
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {"type": "string"},
"price": {"type": ["string", "null"]},
},
"required": ["title", "price"],
},
},
},
"required": ["products"],
},
},
)
if not page.json_.success or page.json_.data is None:
raise RuntimeError("JSON extraction failed")
print(page.json_.data["products"][:3])Note the trailing underscore: the Python SDK exposes this output as page.json_. Our run returned objects such as {'title': 'Asus VivoBook X4', 'price': '295.99'}, the same shape as the Playwright output, without a selector to maintain. Mark fields nullable when a page might omit them, and validate the values before they reach your pipeline.
Each scrape call is one page visit. On paid plans, browser actions can click, scroll, or wait before capture (up to five steps per request), and browser profiles let later requests reuse a saved login. Your code does not drive the browser between those steps, so keep Playwright or another controllable browser when the next action depends on what the page just showed.
Where managed APIs stop and browser automation starts
Context.dev fits requests that end with extracted page content. Its Scrape API renders JavaScript and returns a result for each request, so you can feed retrieved content into a data pipeline without running your own browser. The actions parameter supports lightweight interaction within a request, and a saved browser profile covers pages behind a simple login, but neither turns the API into a browser session your code controls.
Use Playwright or Puppeteer when extraction depends on logins with MFA or CAPTCHA, long chains of clicks and form entries, or a sequence of pages where each step depends on the last. Those tools let your code hold a browser open and control what happens next. A managed retrieval call is the simpler choice when you need the rendered content rather than control over the visit.
Choosing an approach for your pipeline
Use a browser extension for occasional extraction from a session where you are already logged in. Choose DIY Python tooling when you need repeatable, custom browser interaction and can maintain the code and infrastructure. For a production AI pipeline that collects rendered content across many sites, a managed API removes much of that maintenance work.
We built Context.dev for the managed case. One API handles scraping, crawling, and structured output, so you can deploy quickly without maintaining browsers or proxies and connect retrieved content to an LLM pipeline through our MCP server. Keep a browser tool when your workflow needs step-by-step control of a live browser session.
FAQs
What is a screen scraper compared with a web scraper? A web scraper collects website data. A screen scraper reads content after a browser has rendered the page, including content JavaScript added after the initial HTML response.
Is screen scraping legal? Legality depends on what you access, how you access it, and the laws that apply. Check the site's terms, respect access controls, and review privacy and copyright obligations before collecting data.
Can screen scraping work on a JavaScript site without a browser? A plain HTTP fetch cannot read content that JavaScript adds later. You may be able to request the page's underlying data endpoint instead. Otherwise, you need a browser engine, whether you run one yourself or use a rendering API.
How does screen-scraped data differ from an API response? Scraped data comes from page content built for visitors, so your extractor must identify fields and handle layout changes. An API response usually supplies named fields under a documented format, although APIs can change too.
Can Python alone handle dynamic pages? Python can control a browser through Playwright, then read the rendered page. Python's HTML parsers cannot execute JavaScript on their own.